Re: [Duplicity-talk] Sliding window backup strategy?

[Colin Ryan]

Sidenote: I believe - please correct me - that duplicity uses the 
signatures and manifest files to  do incrementals from the last backup 
not from the last full, so does this not avoid the incremental 
approaching the size of a full dilemma?  However it does not as you 
point our deal with what I've come to call the "dirty little secret of 
infinite incrementals"  ( I say this as there are so many backup 
services that pitch/do nothing but ongoing incrementals ) which is 
corruption of even one incremental invalidates all the incrementals past 
that point. However now that I make this statement does not duplicity 
essentially handle this as it checks the backup chain, i.e. incomplete 
backup chain found forcing full??? Maybe I'm smoking something on that 
last point.

An open thought on this discussion is the balance of function versus 
simplicity, would not more advanced techniques as you - and I in the 
past - have requested incur a penalty on the download (thought I agree 
there is more bandwidth there). I've the more recursive/parity/whatever 
checking you have to do on the backup chain to accomplish goal "x", the 
more data must also be additionally pulled from the back-end in addition 
to pushed.

Sorry but I digress ;-)

C


Lars Marowsky-Bree wrote:
> On 2008-10-05T11:51:14, Timothee Besset <address@hidden> wrote:
>
>   
> > I am completely on board with the idea that a full backup should only be
> > performed once, and that the full backup should be maintained with
> > little to no extra incrementals to re-apply.
> >
> > As far as implementation, a sliding window as you described is basically
> > doing a full backup but spreading it over time so it hurts "less" ..
> >     
>
> Agreed, see my other mail.
>
>   
> > When I suggested this same basic feature (only one full backup ever) ..
> > some months ago on this list, I was thinking more along the lines of
> > 'reverse delta' .. for instance like subversion does. The basic idea is
> > that you always keep the latest version of the file and you maintain
> > deltas going back in time towards earlier versions. That way you can
> > drop the old stuff as you see fit.
> >
> > I think however that such an implementation requires some server side
> > intelligence, and it would be pretty hard to fit in duplicity's approach
> > of a dumb untrusted remote storage (which was also one of the aspects
> > that initially drew me to it)
> >     
>
> I know how to do it with server-side intelligence. That's easy ;-)
>
> That is essentially what my home-brewn scripts did. On the server side,
> I rotated the last backup out of the way, created a tree of hard-links,
> and rsync'ed into the new tree (rsync knows how to handle hardlinks).
> That provides _trivial_ access to older revisions, while preserving the
> lowest-possible-overhead on backup thanks to rsync.
>
> Whenever I no longer wanted some particular snapshot, I could simply
> delete it. The filesystem took care of everything else.
>
> However, you're perfectly right - this requires server-side
> intelligence, and access to the raw data on the server (trust issues),
> and it's also less space-efficient than storing rdiffs for the most part
> (each modified file would take the full space, _every time_); which is
> exactly what I'm trying to move away from.
>
>
> Thanks,
>     Lars
>
>