Re: [Duplicity-talk] silent data corruption with checkpoint/restore

[Liraz Siri]

No. A verify run does not detect the data corruption. Also I've
simplified the reproduction of this bug to rule out anything related to
the unessential specifics of the usage scenario I was testing.

First backup::

    duplicity --volsize 5 /data file:///tmp/backup/uninterrupted

Second backup::

    duplicity --volsize 5 /data file:///tmp/backup/interrupted

Interrupted repeatedly with CTRL-C.

Now restore the interrupted and uninterrupted backups and compare::

    duplicity file:///tmp/backup/uninterrupted /tmp/restore/uninterrupted
    duplicity file:///tmp/backup/interrupted /tmp/restore/interrupted

    cd /tmp/restore/uninterrupted
    find -type f | xargs md5sum > md5sums

    cd /tmp/restore/interrupted
    find -type f|xargs md5sum > md5sums

    cd /tmp/restore

    # one corrupted file for every time we use checkpoint/restore
    diff -u {interrupted,uninterrupted}/md5sums

Now that we've confirmed data corruption has happened let's try to
detect it via a verify run::

    duplicity verify file:///tmp/backup/interrupted /data
        Verify complete: 1864 files compared, 0 differences found.

No dice. Silent and deadly. If you try restoring the full backup YOU
WILL suffer data loss.

Just in case you're hoping the next incremental backup will smooth
things over, no such luck. In fact it gets worse. The next incremental
backup will corrupt the backup to the point where it won't restore at
all:

    duplicity file:///tmp/backup/interrupted /tmp/restore/interrupted
    Local and Remote metadata are synchronized, no sync needed.
    Last full backup date: Fri Jul 30 04:44:24 2010
    Traceback (most recent call last):
      File "/usr/bin/duplicity", line 1251, in <module>
        with_tempdir(main)
      File "/usr/bin/duplicity", line 1244, in with_tempdir
        fn()
      File "/usr/bin/duplicity", line 1198, in main
        restore(col_stats)
      File "/usr/bin/duplicity", line 538, in restore
        restore_get_patched_rop_iter(col_stats)):
      File "/usr/lib/python2.5/site-packages/duplicity/patchdir.py",
line 522, in Write_ROPaths
        ITR( ropath.index, ropath )
      File "/usr/lib/python2.5/site-packages/duplicity/lazy.py", line
320, in __call__
        self.process_w_branch(index, self.root_branch, args)
      File "/usr/lib/python2.5/site-packages/duplicity/lazy.py", line
293, in process_w_branch
        branch.start_process, args)
      File "/usr/lib/python2.5/site-packages/duplicity/robust.py", line
37, in check_common_error
        return function(*args)
      File "/usr/lib/python2.5/site-packages/duplicity/patchdir.py",
line 544, in start_process
        assert ropath.index == (), ropath.index

The good news is that if you run verify now it will at least detect
something is terribly wrong:

    duplicity verify file:///tmp/backup/interrupted /data
    ...
    verify complete: 1864 files compared, 781 differences found.

Closer inspections shows that it looks like the first incremental backup
deleted the contents of all volumes which were corrupted in the first
full backup.

The second incremental backup restores the deleted contents. This makes
the latest backup restorable again. Now there are fewer data corruptions
but they still exist. I didn't test what happens if I try to use
checkpoint/restore in the incremental backups. I very much doubt it
would work.

Bug report filed here: https://bugs.launchpad.net/duplicity/+bug/613244

Cheers,
Liraz

address@hidden wrote:
> Does a verify run detect them? .. ede
> 
> 
> On 03.08.2010 04:16, Liraz Siri wrote:
>> I've found a much more serious bug with checkpoint/restore that leads to
>> silent data corruption. I'm using duplicity 0.6.09 installed
>> on a 32-bit Ubuntu system, backing up to the local filesystem.
>>
>> I discovered this by running two full backups with the same parameters.
>>
>> duplicity --archive-dir /var/cache/duplicity --volsize 10
>> --include-filelist /tmp/filelist --exclude '**' /
>> file:///var/tmp/duplicity
>>
>> The first backup I allowed to complete without interruption.
>>
>> The second backup I repeatedly stopped/resumed after the first volume
>> had been created.
>>
>> I then restored the backups to /tmp/restore.broken and /tmp/restore and
>> compared them as follows:
>>
>>     cd /tmp/restore
>>     find |xargs stat -c "%n %U %G %A %s">  statlist
>>
>>     cd /tmp/restore.broken
>>     find |xargs stat -c "%n %U %G %A %s">  statlist
>>
>>     cd ../
>>     diff -u /tmp/restore/{statlist,statlist.broken}
>>
>> I discovered what looks like one corrupt file for each time I
>> CTRL-C/resumed the backup. I'm pretty sure these are the files duplicity
>> resumed from.
>>
>> I would recommend that until this is fixed nobody use the
>> checkpoint/restore feature but unfortunately there doesn't seem to be
>> any way to disable it (in duplicity 0.6).
>>
>> If you've already used checkpoint/restore in a version of duplicity that
>> has this bug it would be wise to recreate your backups immediately. Just
>> make sure the backup completes.
>>
>> Cheers,
>> Liraz
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk


-- 
Liraz Siri
Cell: +972-54-2013512
Twitter: http://twitter.com/lirazsiri