Re: [Duplicity-talk] Version 0.6.0 Released - Checkpoint/Restart

[Kenneth Loafman]

Peter Schuller wrote:
>> The long delayed and much awaited Checkpoint/Restart capability has been
>> added to duplicity and its ready now.  Please read the notes below, try
>> it out, and let me know what you think.
> 
> dup_temp.py now does something crazy :)
> 
>         elif pr.encrypted:
>             gpg.GPGWriteFile(src_iter, tgt.name,
>                              globals.gpg_profile, size = sys.maxint / 2)
> 
> Woooh! This in turn is passed to SrcIter.next() which promptly tries
> to self.fp.read() up to that amount. The result is likely dependent on
> the Python version and platform, but on FreeBSD this has the
> not-so-unexpected effect of failing due to an out-of-memory condition,
> presumably because a buffer of that size is in fact allocated for the
> purpose of the read operation. In my case the resulting value is
> 4611686018427320319. Memory is cheap, but not *that* cheap :)
> 
> Comments leads me to believe there is an attempt to use a very large
> number to ensure we have a chance of filling up the compressed
> block. However even on platforms where the implementation is
> different, and memory is actually allocated on-demand, it seems
> produent that the maximum size should still be reasonable. I can't see
> wanting to use more than a few megs tops for buffering, or duplicity
> turns into a memory hog.
> 
> I do not really have time to really understand all the new code right
> now; is there a reason why we cannot just iterate over this doing the
> I/O at suitable block sizes? I changed the defaults to a more sensible
> 5 MB in gpg.py (though I think even less than that would be prudcent,
> like 128kb), but I'm not sure if this is going to break when actually
> trying to write larger volumes.
> 
> Kenneth, can you shed some light on what the intent is here?

I was trying to make it so GPGWriteFile did not stop at a max file size
for this case because we're not set up for multi-volume sig or manifest
files at this point.  That's coming soon.

As a temp fix, the size= parm on GPGwriteFile should be left as is, but
the arg to SrcIter.next should be much smaller.  This way we'll only get
one complete file, but read in smaller blocks.  I'll see about getting a
patch out soon.

...Ken

signature.asc
Description: PGP signature

signature.asc
Description: OpenPGP digital signature