Re: Discussion about file format for the future

[Leland Best]

Hi All,

First let me admit up front to being mostly a "lurker" on this list. 
Even now with all the new development I've had far more pressing
matters to deal with and have not been following along in the detail I
should have been.  But this issue about file formats really made me
(figuratively) sit up and think "Wait.  What are we talking about
here?".  But let me give a little background just so everyone knows
where I'm coming from.

The last time I posted here was on 2015-08-21 about Winows ACLs not
being restored correctly (an issue I first raised in a post on 2012-09-
07 and which, as far as I know, was never resolved).  However, I've
been using 'rdiff-backup' to make backups of personal and family
machines since 2009.  And that first backup has files that have been
carried from machine to machine since 1996 or so (including my PhD
dissertation :^o :) ).  So there is a _lot_ of stuff in my backups I
don't want to loose.

Via careful planning (and, I suspect, a fair amount of luck) 'rdiff-
backup' has, over that time, allowed me to recover from countless
disasters often requiring a "bare metal" restore.  So, I am greatly
indebted to 'rdiff-backup' for saving my @ss many times.  I've also
accumulated some 8TBytes of backups.

As time went by and it became increasingly clear (as it seemed then)
that 'rdiff-backup' was basically dead in the water I began to look for
alternative backup solutions.  'borgbackup' and 'restic' seemed like
possibilities.  Both had (in my view) limitations but, if 'rdiff-
backup' ever "went away", I could live with them.  However, the Big
Problem with moving to either of them would be transferring my 8TBytes
of backups to whatever I selected as my new program.  As best I could
tell, the only way to do this would be to "restore" each and every
'rdiff-backup' "increment" somewhere, then back it up with the new
tool.  And even then it was not clear to me that I could force the new
tool to "fake" the date-time of the backup to match the original.

So, when it finally looked like there was really going to be a new
maintainer for 'rdiff-backup' I breathed a (tentative) sigh of relief. 
(There had been false starts in the past.)  The business with having to
switch _everything_ to 'rdiff-backup' v2.x all at once was a little
disconcerting but, given the small number of machines I have to worry
about, it seemed doable (though, for various reasons, I haven't yet).

But then along comes this idea about moving to a new file format for
what I'll call bookkeeping files.  I _think_ I'm 100% behind Patrick
Dufresne when he says:

On Thu, 2020-06-04 at 12:43 -0400, Patrik Dufresne wrote:
> Hi Eric,
> 
> My priority in that regard would be to make sure all of this
> backward 
> compatible with the previous repository. I mean, I have terabytes of
> data 
> and I don't foresee a way to convert all these metadata files to a
> new 
> format.

Indeed!  If one can't continue to use existing repositories for new
backups then it's just as bad (well, almost as bad) as if 'rdiff-
backup' "went away"!  And even if one could continue to _read_ from old
format repositories (but can only write to new format repositories), in
my experience what that really means is that there will come a day when
even _reading_ old repositories will stop working.

>  And I'm probably not alone with this situation.

Obviously, you're not.

> We should probably restructure de code to put the read and write of 
> metadata into a single library.
[...]

I'm certainly in no position to comment on 'rdiff-backup's code
structure, but in terms of functionality, if 'rdiff-backup' really
wants to move to a new file format then, from my point of view, the
only reasonable way forward is to also provide a mechanism for
converting old format repositories to new format repositories.  Whether
this is done by 'rdiff-backup' itself or some external tool/program
doesn't matter.  But there _must_ be a way to make existing
repositories keep up with changes to 'rdiff-backup'.  Otherwise it will
be just like starting over with a new backup program.

Well, basically that's all I wanted to say.  Sadly, I'm one of those
people who catches up on e-mail in batches so this may even have been
addressed already, but the potential ramifications of this subject hit
me hard enough that I wanted to get my $0.02 in _right_now_. :)  My
apologies if I've just been wasting everyone's time.

Cheers
Leland
-- 
-------------------------------------------------------------------------------
Leland C. Best      | Creationists make it sound as though a 'theory' is
lcbpublic@gmail.com | something you dreamt up after being drunk all night.
                    | -- Isaac Asimov
-------------------------------------------------------------------------------