Re: [rdiff-backup-users] rdiff-backup features?

[Andrew Ferguson]

Joe Beda wrote:
> Hi Dave,
> 
> I was going to start with the recovering dropped backups.  My current
> understanding (from reading code and experimentation) is that
> rdiff-backup will go through and modify the destination tree in place.
>  If the backup gets dropped then the dest tree will be left in an
> incomplete state.  This incomplete state can be cleaned up in either
> on the next backup or explicitly.

Correct. However, the incomplete state WILL be cleaned up (not can).
This is so that an increment which is listed as occurring at
"2006-12-11T00:02:50-05:00" truly reflects how the files looked at that
time.

This is, in fact, the whole point of why the clean-up happens, as far as
I understand. If a backup is interrupted midway, the repository isn't
"corrupt" but "inconsistent". Consistency of backups/increments can be
seen as an advantage of rdiff-backup. Therefore, I think the default
behavior should remain.

However, I don't believe there is anything stopping us from introducing
the concept of an "incomplete backup". Let's say the previous backup
failed (for whatever reason). The user could, optionally, "tie off" that
backup as "incomplete" and move forward. The action of tying off a
backup would consist of rolling back any changes that made it across
without their corresponding metadata (due to that file's metadata being
in a buffer on the sending end that wasn't flushed before the link
dropped). Incomplete backups would therefore contain only some files at
that time point, but not all. rdiff-backup would have to learn to go to
the previous best backup for any files that were missing.

Incomplete backups carry the desired advantage of not losing the
previous file transfers since rdiff-backup would perceive them as
properly mirrored from source to destination.

> I've come up with two approaches to making incremental backups work:
> 
> 1) Leave current backup mostly as it is today.  On the next backup
> don't clean up but instead merge metadata, undo any changes not in the
> metadata (or recover from the FS?) and continue forward.  Any changes
> from the incomplete backup would be moved forward and there would be
> no increments at that incomplete backup time.

Metadata can be the reason a file changed. Also, recovering from the FS
should always be the last, last, last thing you do because the source
system can have much richer metadata than the destination.

The other problem with this setup is that it loses all those nice
increments we got for the files that did succeed. (Imagine your backup
always failed on the last file, I think it would be nice if we kept all
the increments for the previous files, no?)

> 2) Change the main flow of the backup.  Instead of modifying the dest
> tree in place, instead write forward looking increments out.  When all
> of the forward increments are written out then patch the mirror tree
> to move it forward.  Care would have to be taken to be able to recover
> if the process was stopped during this forward patching.

Frankly, I'm worried that this isn't enough different from what
currently happens. During backup, are you proposing to generate every
patch and it's opposite (ie, the patch from prev -> current and current
-> prev ?). Then, at completion of generating those patches, apply the
prev -> current and store current -> prev in increments/ ?

Please flesh this idea out, it troubles me.

Andrew

> I like #2 because the mirror tree will most always be a consistent
> snapshot -- even if there is a long backup going on.  However, it is a
> more fundamental change.  Any thoughts or advice before I start
> writing code?


-- 
Andrew Ferguson - address@hidden