Re: [Duplicity-talk] silent data corruption with checkpoint/restore

[Kenneth Loafman]

Looks like its time for more work on checkpoint/restore!

Thanks to everyone for putting together detailed bug reports.  That helps.

...Ken

address@hidden wrote:
> Thanks, let's wait what Ken has to say about it.
>
> ..ede/duply.net
>
>
> On 04.08.2010 03:18, Liraz Siri wrote:
>> No. A verify run does not detect the data corruption. Also I've
>> simplified the reproduction of this bug to rule out anything related to
>> the unessential specifics of the usage scenario I was testing.
>>
>> First backup::
>>
>>      duplicity --volsize 5 /data file:///tmp/backup/uninterrupted
>>
>> Second backup::
>>
>>      duplicity --volsize 5 /data file:///tmp/backup/interrupted
>>
>> Interrupted repeatedly with CTRL-C.
>>
>> Now restore the interrupted and uninterrupted backups and compare::
>>
>>      duplicity file:///tmp/backup/uninterrupted
>> /tmp/restore/uninterrupted
>>      duplicity file:///tmp/backup/interrupted /tmp/restore/interrupted
>>
>>      cd /tmp/restore/uninterrupted
>>      find -type f | xargs md5sum>  md5sums
>>
>>      cd /tmp/restore/interrupted
>>      find -type f|xargs md5sum>  md5sums
>>
>>      cd /tmp/restore
>>
>>      # one corrupted file for every time we use checkpoint/restore
>>      diff -u {interrupted,uninterrupted}/md5sums
>>
>> Now that we've confirmed data corruption has happened let's try to
>> detect it via a verify run::
>>
>>      duplicity verify file:///tmp/backup/interrupted /data
>>     Verify complete: 1864 files compared, 0 differences found.
>>
>> No dice. Silent and deadly. If you try restoring the full backup YOU
>> WILL suffer data loss.
>>
>> Just in case you're hoping the next incremental backup will smooth
>> things over, no such luck. In fact it gets worse. The next incremental
>> backup will corrupt the backup to the point where it won't restore at
>> all:
>>
>>      duplicity file:///tmp/backup/interrupted /tmp/restore/interrupted
>>      Local and Remote metadata are synchronized, no sync needed.
>>      Last full backup date: Fri Jul 30 04:44:24 2010
>>      Traceback (most recent call last):
>>        File "/usr/bin/duplicity", line 1251, in<module>
>>          with_tempdir(main)
>>        File "/usr/bin/duplicity", line 1244, in with_tempdir
>>          fn()
>>        File "/usr/bin/duplicity", line 1198, in main
>>          restore(col_stats)
>>        File "/usr/bin/duplicity", line 538, in restore
>>          restore_get_patched_rop_iter(col_stats)):
>>        File "/usr/lib/python2.5/site-packages/duplicity/patchdir.py",
>> line 522, in Write_ROPaths
>>          ITR( ropath.index, ropath )
>>        File "/usr/lib/python2.5/site-packages/duplicity/lazy.py", line
>> 320, in __call__
>>          self.process_w_branch(index, self.root_branch, args)
>>        File "/usr/lib/python2.5/site-packages/duplicity/lazy.py", line
>> 293, in process_w_branch
>>          branch.start_process, args)
>>        File "/usr/lib/python2.5/site-packages/duplicity/robust.py", line
>> 37, in check_common_error
>>          return function(*args)
>>        File "/usr/lib/python2.5/site-packages/duplicity/patchdir.py",
>> line 544, in start_process
>>          assert ropath.index == (), ropath.index
>>
>> The good news is that if you run verify now it will at least detect
>> something is terribly wrong:
>>
>>      duplicity verify file:///tmp/backup/interrupted /data
>>      ...
>>      verify complete: 1864 files compared, 781 differences found.
>>
>> Closer inspections shows that it looks like the first incremental backup
>> deleted the contents of all volumes which were corrupted in the first
>> full backup.
>>
>> The second incremental backup restores the deleted contents. This makes
>> the latest backup restorable again. Now there are fewer data corruptions
>> but they still exist. I didn't test what happens if I try to use
>> checkpoint/restore in the incremental backups. I very much doubt it
>> would work.
>>
>> Bug report filed here: https://bugs.launchpad.net/duplicity/+bug/613244
>>
>> Cheers,
>> Liraz
>>
>> address@hidden wrote:
>>> Does a verify run detect them? .. ede
>>>
>>>
>>> On 03.08.2010 04:16, Liraz Siri wrote:
>>>> I've found a much more serious bug with checkpoint/restore that
>>>> leads to
>>>> silent data corruption. I'm using duplicity 0.6.09 installed
>>>> on a 32-bit Ubuntu system, backing up to the local filesystem.
>>>>
>>>> I discovered this by running two full backups with the same
>>>> parameters.
>>>>
>>>> duplicity --archive-dir /var/cache/duplicity --volsize 10
>>>> --include-filelist /tmp/filelist --exclude '**' /
>>>> file:///var/tmp/duplicity
>>>>
>>>> The first backup I allowed to complete without interruption.
>>>>
>>>> The second backup I repeatedly stopped/resumed after the first volume
>>>> had been created.
>>>>
>>>> I then restored the backups to /tmp/restore.broken and /tmp/restore
>>>> and
>>>> compared them as follows:
>>>>
>>>>      cd /tmp/restore
>>>>      find |xargs stat -c "%n %U %G %A %s">   statlist
>>>>
>>>>      cd /tmp/restore.broken
>>>>      find |xargs stat -c "%n %U %G %A %s">   statlist
>>>>
>>>>      cd ../
>>>>      diff -u /tmp/restore/{statlist,statlist.broken}
>>>>
>>>> I discovered what looks like one corrupt file for each time I
>>>> CTRL-C/resumed the backup. I'm pretty sure these are the files
>>>> duplicity
>>>> resumed from.
>>>>
>>>> I would recommend that until this is fixed nobody use the
>>>> checkpoint/restore feature but unfortunately there doesn't seem to be
>>>> any way to disable it (in duplicity 0.6).
>>>>
>>>> If you've already used checkpoint/restore in a version of duplicity
>>>> that
>>>> has this bug it would be wise to recreate your backups immediately.
>>>> Just
>>>> make sure the backup completes.
>>>>
>>>> Cheers,
>>>> Liraz
>>>>
>>>> _______________________________________________
>>>> Duplicity-talk mailing list
>>>> address@hidden
>>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>
>>
>
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>

signature.asc
Description: OpenPGP digital signature