Re: [Duplicity-talk] Incremental backup of files with changed data but unchanged timestamp

[edgar . soldin]

Ken,

can you as well point me to where the path.py _eq_() method (L325) is called 
during an incremental backup? that would be the place, where i'd put a 
--compare-data switch to enforce data comparison.

but, even if we ignore mtime and _always_ create deltas, wouldn't the current 
design (afaics) assume the file has changed, create a new signature and save 
this to tar - for every file - always?
i didn't see a check for when delta is zero. instead duplicity seems to assume 
that when a delta is created the file has to have changed.

..ede

On 03.08.2014 14:31, Kenneth Loafman wrote:
> We don't need to do the librsync create delta, we just need to ignore the
> timestamp and duplicity will do that for us.
> 
> duplicity.py, dup_time.py, path.py, and tarfile.py are the ones that
> actually reference mtime.  path.py would be the place to look for
> comparison, well before we call librsync.  Take a look in path.py, line 325
> for backup and line 372 for verify.  Replace with 'return 0' and you will
> alway go through the rdiff process.  Very expensive.
> 
> ...Ken
> 
> 
> On Sun, Aug 3, 2014 at 6:39 AM, <address@hidden> wrote:
> 
>> hmm, some more searching didn't reveal no options for the librsync create
>> delta call. it simply seems to create signatures for the whole file only.
>>  http://librsync.sourcefrog.net/doc/librsync.html#processing-whole-files
>>
>> that suggests that the mtime is compared somewhere else, probably in
>>
>> http://bazaar.launchpad.net/~duplicity-team/duplicity/0.7-series/view/head:/duplicity/path.py
>> although i am absolutely clueless as to where in the code path this is
>> supposed to happen.
>>
>> @Ken, Mike: any (more) input?
>>
>> after all this (identical mtime) comes up from time to time on the list
>> e.g.
>>
>> https://lists.nongnu.org/archive/html/duplicity-talk/2013-07/msg00015.html
>> rsync allows to enforce checksum checking '-c' as well, so people probably
>> will expect this from duplicity.
>>
>> ..ede
>>
>>
>> On 03.08.2014 13:05, Kenneth Loafman wrote:
>>> I've seen packages that have the timestamp reflect the version number, so
>>> he's probably right, it would be the packager doing the dirty trick.
>>>
>>> I'm fairly sure you are right that DeltaFile is the first place.  I could
>>> not find anything else.  Mod that and he should be good to go.  It will
>> be
>>> a lot slower, so save the original for the next backup.
>>>
>>> ...Ken
>>>
>>>
>>>
>>> On Sun, Aug 3, 2014 at 5:11 AM, <address@hidden> wrote:
>>>
>>>> On 03.08.2014 02:03, Nate Eldredge wrote:
>>>>> I am using duplicity to make incremental backups of my system.  I have
>>>> some files whose data has changed since the last backup, but whose mtime
>>>> stayed the same.  It looks like `duplicity incremental' ignores files
>> whose
>>>> timestamp has not changed, so it doesn't back up the new data.  Is
>> there a
>>>> way to force duplicity to compare the file with a stored checksum, or
>> even
>>>> to use rdiff unconditionally?  I'd prefer not to have to do a new full
>>>> backup.
>>>>>
>>>>> I'd consider hacking duplicity myself but it would be helpful to know
>>>> where in the code I should look.
>>>>>
>>>>> (Before you accuse me of abusing timestamps: it isn't my fault!  I
>>>> crossgraded this Ubuntu system from 32-bit to 64-bit.  It appears that
>> some
>>>> Ubuntu packages have the same timestamps on corresponding files in the
>>>> 32-bit and 64-bit versions.  Presumably the packages were generated at
>> the
>>>> same time, and coincidentally those files were compiled during the same
>>>> second.  So when I replaced the 32-bit package with the 64-bit package,
>> I
>>>> get a different file with the same timestamp.)
>>>>>
>>>>> I'm using duplicity 0.6.23 (latest from the PPA) on Ubuntu 14.04.
>>>>>
>>>>
>>>> i like "(Before you accuse me of abusing timestamps: it isn't my fault!"
>>>> bit .. hehe as long as the time stamps were old enough you will get off
>>>> scott free i guess..
>>>>
>>>> but seriously - this was obviously not on the horizon of when duplicity
>>>> was developed. i searched a bit but couldn't find anything apart from
>> the
>>>> librsync call 'librsync.DeltaFile(old_sigfp, newfp)' in
>>>>
>>>>
>> http://bazaar.launchpad.net/~duplicity-team/duplicity/0.7-series/view/head:/duplicity/diffdir.py#L136
>>>>
>>>> i cannot seem to find a routine that checks time stamps before that.
>>>>
>>>> @Ken, Mike: can you hint where this magic happens?
>>>>
>>>> ..ede
>>>>
>>>
>>
>