Re: [Duplicity-talk] 0.4.3-rc11 failure on FTP

[Kenneth Loafman]

Nicolas Aspert wrote:
> Hello
> 
> Kenneth Loafman wrote:
>> As to the slowdown... I've tested on my own systems, mostly Linux to
>> Linux, but one Linux to Windows and have seen a small slowdown of maybe
>> 5-10%.  That would not be a problem considering the wide range of ftp
>> servers that ncftp will talk to.  I'd like to know what your source and
>> target systems are and if you have any insight into what may be the
>> problem and why one would work better than the other.
> 
> My source is a poweredge server running centos 5 (= rhel5) and my
> destination is a terastation (linux also). I am backuping the home dirs
> of users i.e. ~140GB of data for a full backup.
> Both systems are on a gigabit lan (with GB network cards too).
> Volume size is 10MB

That's a fairly large backup, and what should be a good setup for it.
Let me see if there's anything I can find out.

> Correct me if I am wrong but I recently run duplicity for an incremental
> with verbosity=8 and its behavior looks sequential: duplicity generates
> files to be uploaded in /tmp and when it has enough of them, uploads
> them, then continues with incremental backup.

Yes, it is sequential: process, upload, repeat.  This saves space on the
local system since only one chunk needs to be active at a time.  On my
systems, it's roughly half and half, processing to uploading.  Assuming
complete parallelization, we could almost halve the processing time for
my case.  For yours, the limit would be processing time.

> I am them wondering if this is possible to improve this a bit by using
> the spooling capacities of ncftp to avoid pausing in volume generation
> (cf. ncftpspooler man page)

It would be.  I need to look into how we could track the progress of the
uploads so we would know when to delete the chunks as they completed.
The current duplicity would try to delete right after spooling since it
would think the process was complete.

This mechanism could then be generalized for all backends so that both
puts and gets could be parallelized with a user settable limit on the
amount of local storage to use.

One solution, if you have the space, is to backup locally using the file
backend, then rsync to the backup server.  rsync has much better restart
capabilities than anything I've seen so far.

...Ken

signature.asc
Description: OpenPGP digital signature