Re: [Duplicity-talk] Duplicity Backup capability

[Kenneth Loafman]

Edward wrote:
> Hi Guys,
> 
> I found duplicity is a very cool idea to do backup, so i tried to use it
> for one of my servers, to a locally attached USB drive.
> However, I encountered a few problems.
> 
> 1. Speed. I backed up a 90G folder which took me 3 days. And about 200G
> of data goes infinite. It has been running for 2 weeks but the backup
> data grows super super slow about 19G.
> 2. It failed with a simple error message: MemoryError, (below is the
> stack trace)
> 
> No signatures found, switching to full backup.
> Traceback (most recent call last):
>   File "/usr/bin/duplicity", line 401, in ?
>     if __name__ == "__main__": main()
>   File "/usr/bin/duplicity", line 394, in main
>     if not sig_chain: full_backup(col_stats)
>   File "/usr/bin/duplicity", line 146, in full_backup
>     bytes_written = write_multivol("full", tarblock_iter, globals.backend)
>   File "/usr/bin/duplicity", line 83, in write_multivol
>     else: at_end = gpg.GzipWriteFile(tarblock_iter,
> tdp.name,globals.volsize)
>   File "/usr/lib/python2.3/site-packages/duplicity/gpg.py", line 241, in
> GzipWriteFile
>     try: new_block = block_iter.next(bytes_to_go)
>   File "/usr/lib/python2.3/site-packages/duplicity/diffdir.py", line
> 405, in next
>     if self.process_waiting: result = self.process_continued(size)
>   File "/usr/lib/python2.3/site-packages/duplicity/diffdir.py", line
> 520, in process_continued
>     data, last_block = self.get_data_block(self.process_fp, size-512)
>   File "/usr/lib/python2.3/site-packages/duplicity/diffdir.py", line
> 508, in get_data_block
>     buf = fp.read(read_size)
>   File "/usr/lib/python2.3/site-packages/duplicity/diffdir.py", line
> 339, in read
>     self.sig_gen.update(buf)
>   File "/usr/lib/python2.3/site-packages/duplicity/librsync.py", line
> 179, in update
>     if self.process_buffer():
>   File "/usr/lib/python2.3/site-packages/duplicity/librsync.py", line
> 187, in process_buffer
>     self.sig_string += cycle_out
> MemoryError
> 
> Does anyone encounter the same problem?

Very few try to run a single application that long.  The best bet is to
break up the work into manageable chunks, by directory for example, then
backup each on its own schedule.

That said, you have run across a known (to me) problem in Python 2.3
where garbage collection alone is not adequate to solve the problem.
The system slows down and the memory and CPU usage go up.  The only
solution to this for now is to make smaller runs.

...Ken

signature.asc
Description: OpenPGP digital signature