Re: [Duplicity-talk] Version 0.6.0 Released - Checkpoint/Restart

[Kenneth Loafman]

Eric Evans wrote:
> [ Kenneth Loafman ]
>> The thing I realized before I finished this was that there needed to be
>> some way to 'name' a backup set.  Right now, you can use an option like:
>> --archive-dir=~/.duplicity/backup_name, but I'd like to move to the
>> point where each backup had its own directory, located by name, with all
>> the options stored in the directory along with the sig and manifest
>> files, but then I took it another step.
>>
>> It would be handy to use a database for all of this, both the sig and
>> manifest info could be stored in the database, along with options, logs,
>> and history.  An SQL database could be used for this quite easily, but
>> my issue with database access is that now you're into major program
>> requirements that get away from small size.  Plus, to be a true backup,
>> you have to store the same info remotely so nothing is lost if the
>> backup machine dies, which kind of implies multiple formats.  I don't
>> think you'd want to shove a database over the wire for a backup.
> 
> Can I ask what caused you to take that extra step (database, as opposed
> to files stored in a backup-specific directory)? It seems pretty
> straightforward to store all of this to disk.

Yes, most of the info can be stored in single files, however, it gets to
be a lot of files when long chains of incremental signature files are
involved.  Think of coalescing signature files, etc., and you'll see
where the database makes life a lot easier.

>> Bottom line is that I'm strongly thinking of rewriting or at least doing
>> a major re-org on duplicity.  After going through all of this and getting
>> a fairly good understanding of duplicity, I think it has out-grown it's
>> original purpose and needs to either be pared down and simplified, or
>> enhanced and functionally completed into something that will handle
>> major backup tasks.
> 
> If you decide to go down the latter path, it's probably worth looking at
> what you have planned and comparing that to existing backup solutions to
> see if you're bringing anything new to the table.

No plans yet, just ideas tossed out to be discussed.

> FWIW, I think there is a niche for backup software that falls somewhere 
> between using cp/scp/rsync/tar/etc directly, and systems like bacula and
> amanda (does anyone still use this?); duplicity fills this niche pretty
> nicely IMO.
> 
>> Some thoughts come to mind:
>> - Named backups, i.e. named directory or keyed database entries.
>> - Plugin style backends, i.e. we supply basic backends, others are able
>>   to be plugged in by just putting them in a plugin directory.
>> - Get away from tar for backend data structure.
>> - Allow incrementals to be folded forward into a new complete set.
> 
> This would be pretty neat. Is there any reason this couldn't be
> implemented as a separate utility right now, something that could be
> run independently on a remote backup host?

Actually, the named backups would be nice to have now that the
archive-dir has to be around between runs.

The plugin idea is just a small matter of programming, not really needed
on desktops, but needed on things like tablets that have limited resources.

Getting away from tar may be more difficult.  The real change I would
make prior to that would be to encrypt the individual files rather then
the entire volume.  That way, lacking par or similar, only a single file
would be lost to data corruption, not the entire volume.  Tradeoff there
would be that the filenames themselves would be exposed, so some info
leaking there.

Incremental folding would be handy for those that do not want to make
regular full backup.  That's a practice I would like to discourage, but
some need to experience failure before learning from it.

...Ken

signature.asc
Description: OpenPGP digital signature