Re: [Duplicity-talk] Having to specify incrementals

[edgar . soldin]

On 04.04.2013 18:02, Elvar wrote:
> 
> On 4/4/2013 10:24 AM, address@hidden wrote:
>> On 04.04.2013 17:16, Elvar wrote:
>>> On 3/29/2013 5:29 PM, address@hidden wrote:
>>>> On 29.03.2013 22:24, Elvar wrote:
>>>>> On 3/29/2013 1:59 PM, address@hidden wrote:
>>>>>> On 29.03.2013 19:35, Elvar wrote:
>>>>>>> On 3/29/2013 11:49 AM, address@hidden wrote:
>>>>>>>> On 29.03.2013 17:31, Elvar wrote:
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I recently started using Duplicity to perform offsite backups for a 
>>>>>>>>> linux server of ours. When I performed  a simulated disaster recovery 
>>>>>>>>> scenario I found that the only data I had been able restore was data 
>>>>>>>>> from the initial full backup. It doesn't appear that Duplicity was 
>>>>>>>>> automatically doing incrementals despite the data growing. Below is 
>>>>>>>>> the command I'm using...
>>>>>>>>>
>>>>>>>>> FTP_PASSWORD='somepass' PASSPHRASE='somepass' duplicity /mnt 
>>>>>>>>> ftps://address@hidden
>>>>>>>>>
>>>>>>>>> Shouldn't that automatically assume incrementals if the full had 
>>>>>>>>> already been done?
>>>>>>>>>
>>>>>>>> yes. what's the output of collection-status?
>>>>>>>>
>>>>>>>> ..ede/duply.net
>>>>>>> After having manually ran an incremental or two, here is the current 
>>>>>>> status.
>>>>>>>
>>>>>>> Import of duplicity.backends.giobackend Failed: No module named gio
>>>>>>> Import of duplicity.backends.sshbackend Failed: No module named paramiko
>>>>>>> LFTP version is 4.3.3
>>>>>>> Local and Remote metadata are synchronized, no sync needed.
>>>>>>> Last full backup date: Thu Mar 28 17:03:26 2013
>>>>>>> Collection Status
>>>>>>> -----------------
>>>>>>> Connecting with backend: FTPSBackend
>>>>>>> Archive dir: /root/.cache/duplicity/cb471964ea71f51bdbff729d2a8e763e
>>>>>>>
>>>>>>> Found 0 secondary backup chains.
>>>>>>>
>>>>>>> Found primary backup chain with matching signature chain:
>>>>>>> -------------------------
>>>>>>> Chain start time: Thu Mar 28 17:03:26 2013
>>>>>>> Chain end time: Fri Mar 29 11:23:11 2013
>>>>>>> Number of contained backup sets: 6
>>>>>>> Total number of contained volumes: 61
>>>>>>>     Type of backup set:                            Time:      Num 
>>>>>>> volumes:
>>>>>>>                    Full         Thu Mar 28 17:03:26 2013 50
>>>>>>>             Incremental         Thu Mar 28 19:24:47 2013                
>>>>>>>  1
>>>>>>>             Incremental         Fri Mar 29 09:23:59 2013                
>>>>>>>  1
>>>>>>>             Incremental         Fri Mar 29 09:33:08 2013                
>>>>>>>  1
>>>>>>>             Incremental         Fri Mar 29 10:12:58 2013                
>>>>>>>  1
>>>>>>>             Incremental         Fri Mar 29 11:23:11 2013                
>>>>>>>  7
>>>>>>> -------------------------
>>>>>>> No orphaned or incomplete backup sets found.
>>>>>>>
>>>>>> dunno what your complaining about. it clearly states
>>>>>>
>>>>>> 1 x full
>>>>>> 5 x incrementals
>>>>>>
>>>>>> run again without forced incremental and see if it adds full or incr.
>>>>>>
>>>>>> ..ede
>>>>>>
>>>>>>
>>>>> So I just ran it again and I see it says "NewFiles 0" and "NewFileSize 
>>>>> 0". The content I'm backing up is an email archive that uses Maildir 
>>>>> format for storage. I know for certain several hundred emails or more 
>>>>> have landed in this archive since my last incremental yet this last job 
>>>>> doesn't seem to see it. So, my question is, why isn't the incremental job 
>>>>> grabbing the latest data?
>>>>>
>>>>> Import of duplicity.backends.giobackend Failed: No module named gio
>>>>> Import of duplicity.backends.sshbackend Failed: No module named paramiko
>>>>> LFTP version is 4.3.3
>>>>> Local and Remote metadata are synchronized, no sync needed.
>>>>> Last full backup date: Thu Mar 28 17:03:26 2013
>>>>> --------------[ Backup Statistics ]--------------
>>>>> StartTime 1364591400.73 (Fri Mar 29 16:10:00 2013)
>>>>> EndTime 1364591418.34 (Fri Mar 29 16:10:18 2013)
>>>>> ElapsedTime 17.62 (17.62 seconds)
>>>>> SourceFiles 29069
>>>>> SourceFileSize 2409332518 (2.24 GB)
>>>>> NewFiles 0
>>>>> NewFileSize 0 (0 bytes)
>>>>> DeletedFiles 0
>>>>> ChangedFiles 0
>>>>> ChangedFileSize 0 (0 bytes)
>>>>> ChangedDeltaSize 0 (0 bytes)
>>>>> DeltaEntries 0
>>>>> RawDeltaSize 0 (0 bytes)
>>>>> TotalDestinationSizeChange 103 (103 bytes)
>>>>> Errors 0
>>>>> -------------------------------------------------
>>>>>
>>>> what happened to your first issue? is that solved?
>>>>
>>>> wrt. 0byte change . this can happen when software does not update 
>>>> timestamps correctly. but i doubt that's the case here.
>>>> how about simply restoring the latest archive and binary compare it to the 
>>>> current state? after that run 'verify' and see if that says there is one.
>>>>
>>>> ..ede/duply.net
>>>>
>>>>
>>> Ok, I think I may have figured out the issue. When I was running the 
>>> Duplicity backups initially I was doing it on a read only iscsi mounted 
>>> volume. I noticed if I mounted without read-only I didn't have the same 
>>> issue with it not backing up newer data. Is this expected behavior?
>>>
>> no, please check if timestamps are correctly changing when the iscsi device 
>> is mounted read only.
>>
>> ede/duply.net
>>
>>
> 
> The timestamps of the data I'm backing up? Well, the data is almost entirely 
> archived email being stored in Maildir format. There is constantly new files 
> being created every few seconds. For whatever reason, when mounted read-only, 
> it skips the new files and no data changes. If I run the same exact command 
> mounting rw, it backs up all new files and modified files since the last 
> backup.
> 

duplicity checks if timestamps changed to determine the need for backup. check 
for your maildir files if timestamps change over time when mounted ro. probably 
they don't and that causes the problem?

..ede