Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?

[edgar . soldin]

On 29.05.2015 12:44, Remy van Elst wrote:
> 
> 
> On 05/29/2015 12:35 PM, address@hidden wrote:
>> On 29.05.2015 04:41, Remy van Elst wrote:
>>>
>>>
>>> On 05/29/2015 12:25 AM, address@hidden wrote:
>>>> please do not top post if the answer was posted at the bottom..
>>>> i will continue below
>>>
>>>> On 28.05.2015 19:47, Remy van Elst wrote:
>>>>> Here's the first 31000 lines of the log: 
>>>>> https://e75f5515c4f249ffa1122853af811b3e.objectstore.eu/duplicity/d
> up
>>>
>>>>>
> lic
>>>>>
>>>>>
>>> ity.txt
>>>>>
>>>>> After these, it is all lines comparing the files on the
>>>>> machine. That is still running. Is this helpfull?
>>>>>
>>>>> On 05/28/2015 03:54 PM, address@hidden wrote:
>>>>>> On 28.05.2015 15:28, Remy van Elst wrote:
>>>>>>> Any suggestions or other help?
>>>>>>>
>>>>>>> On 05/25/2015 11:04 AM, Remy van Elst wrote:
>>>>>>>> Duplicity seems to loose it's status when trying to do
>>>>>>>> a incremental backup set over 14 days. The backup has
>>>>>>>> been running for a while now but it is using about 1.5
>>>>>>>> TB of backend storage since 2 weeks. It is going to an
>>>>>>>> Openstack swift object store as backend.
>>>>>>>
>>>>>>>> This is the command line:
>>>>>>>
>>>>>>>> duplicity --asynchronous-upload --verbosity 9
>>>>>>>> --log-file /var/log/duplicity.log --volsize 25
>>>>>>>> --tempdir="/tmp" 
>>>>>>>> --file-prefix="vps1.sparklingclouds.nl." 
>>>>>>>> --name="vps1.sparklingclouds.nl."
>>>>>>>> --exclude-device-files 
>>>>>>>> --exclude-globbing-filelist=/etc/duplicity-backup/exclude.conf
>>>>>>>>
>>>>>>>>
>>>
>>>>>>>>
> --full-if-older-than="14D" --no-encryption  /
>>>>>>>> swift://duplicity-backup
>>>>>>>
>>>>>>>> This is what caught my eye in the verbose log file:
>>>>>>>
>>>>>>>> File 
>>>>>>>> vps1.sparklingclouds.nl.duplicity-full.20150430T034939Z.vol166.d
> if
>>>
>>>>>>>>
> ft
>>>>>
>>>>>>>>
>>> ar
>>>>>>>
>>>>>>>>
>>>>> .g
>>>>>>>
>>>>>>>
>>>>>>> z
>>>>>>>> is part of known set File 
>>>>>>>> vps1.sparklingclouds.nl.duplicity-full.20150430T034939Z.vol1660.
> di
>>>
>>>>>>>>
> ff
>>>>>
>>>>>>>>
>>> ta
>>>>>>>
>>>>>>>>
>>>>> r.
>>>>>>>
>>>>>>>
>>>>>>> gz
>>>>>>>> is part of known set Found backup chain [Tue Mar 24 
>>>>>>>> 14:42:44 2015]-[Tue Mar 24 14:42:44 2015] Last full
>>>>>>>> backup is too old, forcing full backup
>>>>>>>
>>>>>>>
>>>>>>>> The last full backup is from yesterday, and it should
>>>>>>>> only do a full backup every 14 days.
>>>>>>>
>>>>>>>> When I do a collection status it seems all the
>>>>>>>> incremental backups are gone:
>>>>>>>
>>>>>>>> Do note that I've got a wrapper script around
>>>>>>>> duplicity which outputs the status, storage used and
>>>>>>>> config settings:
>>>>>>>
>>>>>>>
>>>>>>>> # Hostname: vps1.sparklingclouds.nl # IP: 192.0.2.10 # 
>>>>>>>> Storage used: 1.5T # Full backups to keep: 4 # Create
>>>>>>>> full backup if last full backup is older than: 14D 
>>>>>>>> ----------------------------------------- # Duplicity 
>>>>>>>> collection status: # Local and Remote metadata are 
>>>>>>>> synchronized, no sync needed. # Last full backup date:
>>>>>>>> Mon May 25 10:48:16 2015 # Collection Status # 
>>>>>>>> ----------------- # Connecting with backend:
>>>>>>>> BackendWrapper # Archive dir: 
>>>>>>>> /root/.cache/duplicity/vps1.sparklingclouds.nl. # Found
>>>>>>>> 4 secondary backup chains. # Secondary chain 1 of 4: #
>>>>>>>>  ------------------------- # Chain start time: Tue Mar
>>>>>>>> 24 14:42:44 2015 # Chain end time: Tue Mar 24 14:42:44
>>>>>>>> 2015 # Number of contained backup sets: 1 # Total
>>>>>>>> number of contained volumes: 3009 #  Type of backup
>>>>>>>> set: Time: Num volumes: #                 Full
>>>>>>>> Tue Mar 24 14:42:44 2015 3009 #
>>>>>>>> ------------------------- # Secondary chain 2 of 4: #
>>>>>>>> ------------------------- # Chain start time: Wed Apr
>>>>>>>> 8 05:49:18 2015 # Chain end time: Wed Apr  8 05:49:18
>>>>>>>> 2015 # Number of contained backup sets: 1 # Total 
>>>>>>>> number of contained volumes: 3104 #  Type of backup
>>>>>>>> set: Time:      Num volumes: #                 Full
>>>>>>>> Wed Apr 8 05:49:18 2015 3104 #
>>>>>>>> ------------------------- # Secondary chain 3 of 4: #
>>>>>>>> ------------------------- # Chain start time: Thu Apr
>>>>>>>> 16 05:49:37 2015 # Chain end time: Thu Apr 16 05:49:37
>>>>>>>> 2015 # Number of contained backup sets: 1 # Total
>>>>>>>> number of contained volumes: 3128 #  Type of backup 
>>>>>>>> set: Time:      Num volumes: #                 Full Thu
>>>>>>>> Apr 16 05:49:37 2015 3128 # -------------------------
>>>>>>>> # Secondary chain 4 of 4: # ------------------------- #
>>>>>>>> Chain start time: Thu Apr 30 05:49:39 2015 # Chain end
>>>>>>>> time: Thu Apr 30 05:49:39 2015 # Number of contained
>>>>>>>> backup sets: 1 # Total number of contained volumes: 736
>>>>>>>> # Type of backup set: Time:      Num volumes: #
>>>>>>>> Full Thu Apr 30 05:49:39 2015 736 #
>>>>>>>> ------------------------- # Found primary backup chain
>>>>>>>> with matching signature chain: #
>>>>>>>> ------------------------- # Chain start time: Mon May
>>>>>>>> 25 10:48:16 2015 # Chain end time: Mon May 25 10:48:16
>>>>>>>> 2015 # Number of contained backup sets: 1 # Total
>>>>>>>> number of contained volumes: 0 #  Type of backup set:
>>>>>>>> Time:      Num volumes: # ------------------------- #
>>>>>>>> No orphaned or incomplete backup sets found.
>>>>>>>
>>>>>>>
>>>>>>>> Why does this happen and how can I get this back to 
>>>>>>>> normal?
>>>>>>>
>>>>>
>>>>>> which duplicity version?
>>>>>
>>>>>> please post (zip attach)or pastebin the complete console
>>>>>> output of a max.verbosity '-v9' status run. obfuscate
>>>>>> strings in it you deem private.
>>>>>
>>>>>> ..ede/duply.net
>>>>>
>>>
>>>> the log states
>>>
>>>> --> DEBUG 1 . Found backup chain [Tue Mar 24 14:42:44
>>>> 2015]-[Tue Mar 24 14:42:44 2015]
>>>
>>>> DEBUG 1 . Found backup chain [Wed Apr  8 05:49:18 2015]-[Wed
>>>> Apr  8 05:49:18 2015]
>>>
>>>> DEBUG 1 . Found backup chain [Thu Apr 16 05:49:37 2015]-[Thu
>>>> Apr 16 05:49:37 2015]
>>>
>>>> DEBUG 1 . Found backup chain [Thu Apr 30 05:49:39 2015]-[Thu
>>>> Apr 30 05:49:39 2015]
>>>
>>>> NOTICE 1 . Last full backup date: Thu Apr 30 05:49:39 2015
>>>
>>>> NOTICE 1 . Last full backup is too old, forcing full backup
>>>> <--
>>>
>>>> which sounds reasonable.. there are no incremental backups on
>>>> your backend.
>>>
>>> There should be. There runs a cronjob every night with the same 
>>> command as I ran manually. It runs for about 3 to 5 hours just as
>>> now, and then fails. The backend keeps growing in size.
>>>
>>>
> 
>> ok, please ensure that your manual full went through via 'status'
>> command.
> 
> 
> It is not listed in the output of collection-status. Should I still
> run the backup again?
> 
>> then rerun the same backup cmd line you sent me before and send me
>> the output up until the first 'Getting delta of ...' message.
> 

weird, can you send a list of the files on your backend? looks like not all 
were uploaded successfully.

next step would be to try latest duplicity 0.6.x . maybe some bug crept in the 
0.7 development line. here is a short howto install from tarball to a local 
folder (under INSTALL MULTIPLE VERSIONS).
 http://duply.net/wiki/index.php/Duply-documentation
you might wanna to install to /usr/local/duplicity-0.6.x instead.

..ede/duply.net