Re: [Duplicity-talk] Having to specify incrementals

[edgar . soldin]

On 16.04.2013 17:54, Elvar wrote:
> 
> On 4/4/2013 11:49 AM, address@hidden wrote:
>> On 04.04.2013 18:10, Elvar wrote:
>>> On 4/4/2013 11:05 AM, address@hidden wrote:
>>>> On 04.04.2013 18:02, Elvar wrote:
>>>>> On 4/4/2013 10:24 AM, address@hidden wrote:
>>>>>> On 04.04.2013 17:16, Elvar wrote:
>>>>>>> On 3/29/2013 5:29 PM, address@hidden wrote:
>>>>>>>> On 29.03.2013 22:24, Elvar wrote:
>>>>>>>>> On 3/29/2013 1:59 PM, address@hidden wrote:
>>>>>>>>>> On 29.03.2013 19:35, Elvar wrote:
>>>>>>>>>>> On 3/29/2013 11:49 AM, address@hidden wrote:
>>>>>>>>>>>> On 29.03.2013 17:31, Elvar wrote:
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I recently started using Duplicity to perform offsite backups for 
>>>>>>>>>>>>> a linux server of ours. When I performed  a simulated disaster 
>>>>>>>>>>>>> recovery scenario I found that the only data I had been able 
>>>>>>>>>>>>> restore was data from the initial full backup. It doesn't appear 
>>>>>>>>>>>>> that Duplicity was automatically doing incrementals despite the 
>>>>>>>>>>>>> data growing. Below is the command I'm using...
>>>>>>>>>>>>>
>>>>>>>>>>>>> FTP_PASSWORD='somepass' PASSPHRASE='somepass' duplicity /mnt 
>>>>>>>>>>>>> ftps://address@hidden
>>>>>>>>>>>>>
>>>>>>>>>>>>> Shouldn't that automatically assume incrementals if the full had 
>>>>>>>>>>>>> already been done?
>>>>>>>>>>>>>
>>>>>>>>>>>> yes. what's the output of collection-status?
>>>>>>>>>>>>
>>>>>>>>>>>> ..ede/duply.net
>>>>>>>>>>> After having manually ran an incremental or two, here is the 
>>>>>>>>>>> current status.
>>>>>>>>>>>
>>>>>>>>>>> Import of duplicity.backends.giobackend Failed: No module named gio
>>>>>>>>>>> Import of duplicity.backends.sshbackend Failed: No module named 
>>>>>>>>>>> paramiko
>>>>>>>>>>> LFTP version is 4.3.3
>>>>>>>>>>> Local and Remote metadata are synchronized, no sync needed.
>>>>>>>>>>> Last full backup date: Thu Mar 28 17:03:26 2013
>>>>>>>>>>> Collection Status
>>>>>>>>>>> -----------------
>>>>>>>>>>> Connecting with backend: FTPSBackend
>>>>>>>>>>> Archive dir: /root/.cache/duplicity/cb471964ea71f51bdbff729d2a8e763e
>>>>>>>>>>>
>>>>>>>>>>> Found 0 secondary backup chains.
>>>>>>>>>>>
>>>>>>>>>>> Found primary backup chain with matching signature chain:
>>>>>>>>>>> -------------------------
>>>>>>>>>>> Chain start time: Thu Mar 28 17:03:26 2013
>>>>>>>>>>> Chain end time: Fri Mar 29 11:23:11 2013
>>>>>>>>>>> Number of contained backup sets: 6
>>>>>>>>>>> Total number of contained volumes: 61
>>>>>>>>>>>       Type of backup set:                            Time:      Num 
>>>>>>>>>>> volumes:
>>>>>>>>>>>                      Full         Thu Mar 28 17:03:26 2013 50
>>>>>>>>>>>               Incremental         Thu Mar 28 19:24:47 2013          
>>>>>>>>>>>        1
>>>>>>>>>>>               Incremental         Fri Mar 29 09:23:59 2013          
>>>>>>>>>>>        1
>>>>>>>>>>>               Incremental         Fri Mar 29 09:33:08 2013          
>>>>>>>>>>>        1
>>>>>>>>>>>               Incremental         Fri Mar 29 10:12:58 2013          
>>>>>>>>>>>        1
>>>>>>>>>>>               Incremental         Fri Mar 29 11:23:11 2013          
>>>>>>>>>>>        7
>>>>>>>>>>> -------------------------
>>>>>>>>>>> No orphaned or incomplete backup sets found.
>>>>>>>>>>>
>>>>>>>>>> dunno what your complaining about. it clearly states
>>>>>>>>>>
>>>>>>>>>> 1 x full
>>>>>>>>>> 5 x incrementals
>>>>>>>>>>
>>>>>>>>>> run again without forced incremental and see if it adds full or incr.
>>>>>>>>>>
>>>>>>>>>> ..ede
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> So I just ran it again and I see it says "NewFiles 0" and 
>>>>>>>>> "NewFileSize 0". The content I'm backing up is an email archive that 
>>>>>>>>> uses Maildir format for storage. I know for certain several hundred 
>>>>>>>>> emails or more have landed in this archive since my last incremental 
>>>>>>>>> yet this last job doesn't seem to see it. So, my question is, why 
>>>>>>>>> isn't the incremental job grabbing the latest data?
>>>>>>>>>
>>>>>>>>> Import of duplicity.backends.giobackend Failed: No module named gio
>>>>>>>>> Import of duplicity.backends.sshbackend Failed: No module named 
>>>>>>>>> paramiko
>>>>>>>>> LFTP version is 4.3.3
>>>>>>>>> Local and Remote metadata are synchronized, no sync needed.
>>>>>>>>> Last full backup date: Thu Mar 28 17:03:26 2013
>>>>>>>>> --------------[ Backup Statistics ]--------------
>>>>>>>>> StartTime 1364591400.73 (Fri Mar 29 16:10:00 2013)
>>>>>>>>> EndTime 1364591418.34 (Fri Mar 29 16:10:18 2013)
>>>>>>>>> ElapsedTime 17.62 (17.62 seconds)
>>>>>>>>> SourceFiles 29069
>>>>>>>>> SourceFileSize 2409332518 (2.24 GB)
>>>>>>>>> NewFiles 0
>>>>>>>>> NewFileSize 0 (0 bytes)
>>>>>>>>> DeletedFiles 0
>>>>>>>>> ChangedFiles 0
>>>>>>>>> ChangedFileSize 0 (0 bytes)
>>>>>>>>> ChangedDeltaSize 0 (0 bytes)
>>>>>>>>> DeltaEntries 0
>>>>>>>>> RawDeltaSize 0 (0 bytes)
>>>>>>>>> TotalDestinationSizeChange 103 (103 bytes)
>>>>>>>>> Errors 0
>>>>>>>>> -------------------------------------------------
>>>>>>>>>
>>>>>>>> what happened to your first issue? is that solved?
>>>>>>>>
>>>>>>>> wrt. 0byte change . this can happen when software does not update 
>>>>>>>> timestamps correctly. but i doubt that's the case here.
>>>>>>>> how about simply restoring the latest archive and binary compare it to 
>>>>>>>> the current state? after that run 'verify' and see if that says there 
>>>>>>>> is one.
>>>>>>>>
>>>>>>>> ..ede/duply.net
>>>>>>>>
>>>>>>>>
>>>>>>> Ok, I think I may have figured out the issue. When I was running the 
>>>>>>> Duplicity backups initially I was doing it on a read only iscsi mounted 
>>>>>>> volume. I noticed if I mounted without read-only I didn't have the same 
>>>>>>> issue with it not backing up newer data. Is this expected behavior?
>>>>>>>
>>>>>> no, please check if timestamps are correctly changing when the iscsi 
>>>>>> device is mounted read only.
>>>>>>
>>>>>> ede/duply.net
>>>>>>
>>>>>>
>>>>> The timestamps of the data I'm backing up? Well, the data is almost 
>>>>> entirely archived email being stored in Maildir format. There is 
>>>>> constantly new files being created every few seconds. For whatever 
>>>>> reason, when mounted read-only, it skips the new files and no data 
>>>>> changes. If I run the same exact command mounting rw, it backs up all new 
>>>>> files and modified files since the last backup.
>>>>>
>>>> duplicity checks if timestamps changed to determine the need for backup. 
>>>> check for your maildir files if timestamps change over time when mounted 
>>>> ro. probably they don't and that causes the problem?
>>>>
>>>> ..ede
>>>>
>>> The appliance which is doing the email archiving isn't mounted ro though so 
>>> it should be updating the time & date stamps as it performs writes. 
>>> Wouldn't a separate server mounting the same partition only in ro mode 
>>> still see changes?
>>>
>>>
>>>
>> generally yes, the timestamps should be updated. but i suppose they are not. 
>> please check manually if they are and report back.
>>
>> ..ede
> 
> So I found that whether I mounted it read only or not didn't matter. The only 
> way it was correctly seeing the updated time/date stamps was to remount the 
> iscsi volume just before each backup job. I'm guessing this is because two 
> separate linux hosts are mounting the same ext3 partition and it is not a 
> clustered FS. Either way, by adding umount and mount commands to my script it 
> is working fine now. I do have a different question but I'll start a new 
> topic for that.
> 

thanks for keeping us posted. wrt the iscsi issue. you should contact the 
author or the software's mailing list. either it is a bug or there is something 
to setup. a file system that does not update timestamps is missing half of it's 
purpose ;)

.. ede/duply.net