Re: [Duplicity-talk] (Option to) Cache retrieved volumes between (fetching/restoring) runs locally?!

[Daniel Hahler]

Hello,

>>> It would be much easier to simply mirror you repository to a
>>> local path and restore from there.
>> 
>> It would be ~30GB, which would take a while to mirror, apart from
>> that you might not have the space available.

> if you are lacking the space, you probably have no space for the
> cache either.

While you make a good point here of course, not the whole backup gets
used when restoring only certain files from it (according to duplicity's
design and the log).
Otherwise it would have been much slower anyway.

>>> Can't you just fetch/restore all files/folders in one go?
>> 
>> Does "duply fetch" / duplicity support fetching multiple files? It
>> does not look so from the man page.
> 
> Indeed it doesn't. I always expected duplicity to work with
> exclude/include also on restoring. It doesn't. Currently you can
> obviously only restore all or one file/folder. For the latter. It is
> the folder only without it's contents ;).

While I have not verified it (and don't know if you're joking), but it
seemed like the contents of a folder gets restored completely - at least
the files contained therein.

> You could of course restore all to a temp folder and move what you
> need to wherever needed. That's what I did whenever I had to
> restore.

Yes, might have worked better after all - given that you cannot provide
"includes"/pattern yet.

>> If you're referring to get the meat of it only, that would have
>> been the root directory of all virtual containers (which is >90% of
>> the whole backup).
> when it is 90% of the backup, the last 10% download really do not
> weigh out the speed improvement once you restore from your local
> 'cache'.

Not 90% were being downloaded now, but only certain files out of the 90%
(below /vz/private, where my containers are in).

> Eventually I think it might be more important to make restore respect
> in/exclude, as a user would expect it to, instead of working around
> it by looping/caching over the backup.

> Would you create a feature request on launchpad for that?

Here you go: https://bugs.launchpad.net/duplicity/+bug/677177


Thanks,
Daniel

> 
> regards ..ede/duply.net
> 
>> 
>> 
>> Thanks, Daniel
>> 
>>> On 09.11.2010 21:53, Daniel Hahler wrote:
>>>> Hello,
>>>> 
>>>> I would like to be able to cache retrieved files from the
>>>> backend locally between multiple runs of duplicity, e.g. via
>>>> some config or command line option.
>>>> 
>>>> Use case: having accidentally overwritten a lot of my
>>>> (virtual) containers files, I've used the following to restore
>>>> the previous state: for i in $=VCIDS; do b=path/to/files 
>>>> /bin/rm -rf /$b* duply profile fetch $b /$b 1D duply profile
>>>> fetch ${b}.d /${b}.d 1D done
>>>> 
>>>> This makes up 60+ runs of duplicity (2 runs of duplicity per
>>>> 30+ containers, one for a single file, the other for a
>>>> directory), and when looking at it with "--verbosity 9" it
>>>> looks like a lot of the same volumes (with a size of 50M in my
>>>> case) are downloaded every time.
>>>> 
>>>> I think it would speed this (particular) use case dramatically
>>>> up, if these files would get cached locally.
>>>> 
>>>> I could imagine to configure something like "keep files for X
>>>> hours", and when duplicity gets run and files are older than
>>>> this, they get cleaned on shutdown. When being accessed, they
>>>> would get touched to reset the timer.
>>>> 
>>>> However, there should also be a maximum number of files to
>>>> cache, since this might easily fill your local volume
>>>> otherwise.
>>>> 
>>>> I am thinking about caching the files encrypted (just as on the
>>>> remote site), but maybe caching decrypted files would make
>>>> sense, too?
>>>> 
>>>> Obviously, this should take into account if this is a remote
>>>> backup (maybe by looking at the transfer rate of the files?!),
>>>> and not pollute the cache, if the backend is as fast as local
>>>> transfers might be.
>>>> 
>>>> What do you think?
>>>> 
>>>> 
>>>> Cheers, Daniel
>>> 
>>> _______________________________________________ Duplicity-talk
>>> mailing list address@hidden 
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>> 
>>> 
>>> _______________________________________________ Duplicity-talk
>>> mailing list address@hidden 
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>> 
>> 
>> _______________________________________________ Duplicity-talk
>> mailing list address@hidden 
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
> 
> _______________________________________________ Duplicity-talk
> mailing list address@hidden 
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk