Re: [rdiff-backup-users] About backups and increments

[Robert Nichols]

On 08/21/2011 05:27 PM, Yuri D'Elia wrote:
> Is the backup procedure crash/interrupt safe? If I'm backing-up some data and
> for some reason the program gets killed (say, manually), in what state the
> backup will remain? Does the latest snapshot reflect the old state (as if no
> backup was performed), a broken state (cannot reliably recover anything), or a
> mixed state? Is this state ever recorded as an increment so that I can drop it?

Recovery from a failed or interrupted session is reliable, but time consuming.
The next time you try to do any operation on that archive, rdiff-backup will
insist on first performing a regression of the failed session back to the
previous state.  There is also the "--check-destination-dir" action, which
does only the regression, if one is needed.  For a large backup set, the
regression takes a *long* time.

> About space requirements: I assume the space required for the backup is:
>
> - the space of the source files themselves
> - the space of all the increments
> - extra space required to compute the increment?
> * Is this space stored on the source or destination drive?  * This should be
> the size of the file currently computed + it's increments right? So should I
> assume that to backup the *second* increment of some space X (where X can
> possibly be just one huge file) I need at least X * 2 space for the backup -
> just for temporary files?
> * This brings me back to my first question: what happens when the destination
> is full?

I'm not aware of any extra space needed for computing the increment, but the
increment itself, of course, does need to be stored on the destination drive.
If the destination drive runs out of space, the rdiff-backup session will
fail.

> About backup speed. rdiff-backup doesn't seem to support both backupping *and*
> pruning the increments at the same time (yes, I've read the man page). Though
> this sounds like a very sensible thing to do: knowing that you will prune
> several old increments, you can avoid to calculate the reverse diffs. Has this
> been considered?

There's not much point in combining those two, totally independent actions.
Computing the reverse diffs for session N vs. session N-1 is totally
independent of the existence (or lack thereof) of earlier sessions in the
archive.

> A request: increments pruning. --remove-older-than is fine most of the times,
> though I would like an extra knob:
>
> --keep-increments N (where N is the number of most recent increments to keep,
> irregardless of time).

You can already do that.  Though the manpage doesn't mention it, you can also
use a "B" suffix to specify the number of sessions to keep:

       rdiff-backup --remove-older-than 30B /path/to/archive

will retain the most recent 30 sessions.  (Yes, you'll probably need to
include "--force" with that.)

> Let's say I want always to keep at all times at least 2 increments (or 2
> months, if that matters), I have no way to do that directly (I could list the
> increments and calculate the time myself, but that's ugly).

Hey!! Some of us scripting veterans really get off on "ugly"!

    # Each Sunday, delete backups older than the previous Sunday, but
    # always retaining at least 6 backups.
    if [ "$(date +%a)" = "Sun" ]; then
        Cut=$(date -d 'last week' "+increments.%Y-%m-%d")
        CutAt=$(rdiff-backup -l "$BkVolume" | \
            awk -v "Where=$Cut" '
BEGIN {
    stderr = "/dev/stderr"
    Dmax = -1
}
FNR > 1 {
    bdate[10000+FNR] = $1
    if(Dmax < 0 && $1 > Where)  Dmax = FNR
}
END {
    if(Dmax < 0 || Dmax > FNR - 5)  Dmax = FNR - 5
    if(Dmax > 2) {
        if(match(bdate[10000+Dmax], "^increments\\.....-..-..T..:..:..-")) {
            print substr(bdate[10000Dmax], 12, 10)
            exit 0
        }
        else {
            print "Unrecognized increment: \"" bdate[10000+Dmax] "\"" >stderr
        }
    }
    exit 1
}' )
        [ $? = 0 -a -n "$CutAt" ] && rdiff-backup --remove-older-than $CutAt \
            --force "$BkVolume"
    fi


--
Bob Nichols     "NOSPAM" is really part of my email address.
                Do NOT delete it.