[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upl
From: |
Kris |
Subject: |
Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http:// |
Date: |
Wed, 2 Nov 2011 22:10:08 -0400 |
OK,
This should take care of it:
--- /usr/lib64/python2.6/site-packages/duplicity/backends/botobackend.py.orig
2011-11-02 21:35:38.000000000 -0400
+++ /usr/lib64/python2.6/site-packages/duplicity/backends/botobackend.py
2011-11-02 22:06:37.000000000 -0400
@@ -333,10 +333,13 @@
# Decide in how many chunks to upload
bytes = os.path.getsize(filename)
+ last_chunk = 0
if bytes < chunk_size:
chunks = 1
+ last_chunk = bytes
else:
chunks = bytes / chunk_size
if (bytes % chunk_size):
chunks += 1
+ last_chunk = bytes % chunk_size
log.Debug("Uploading %d bytes in %d chunks" % (bytes, chunks))
@@ -346,4 +349,9 @@
pool = multiprocessing.Pool(processes=chunks)
for n in range(chunks):
+ if last_chunk > 0 and n == chunks-1:
+ chunk_bytes = last_chunk
+ else:
+ chunk_bytes = chunk_size
+
params = {
'scheme': self.scheme,
@@ -353,5 +361,5 @@
'filename': filename,
'offset': n,
- 'bytes': chunk_size,
+ 'bytes': chunk_bytes,
'num_retries': globals.num_retries,
}
Kris
On 2011-11-02, at 9:46 PM, Kris wrote:
> Sorry, I'm asleep, my last post was nonsense. I'm just going to write up a
> quick tested patch if it helps, unless you're already tackling this.
>
> Kris
>
>
>
> On 2011-11-02, at 9:30 PM, Kris wrote:
>
>> Thanks for your help Michael! It's hugely appreciated. I can confirm that I
>> was looking at the same code and saw that the last chunk of a file not
>> perfectly divisible will not have an incorrect number of bytes passed in to
>> param for the last chunk.
>>
>> Just to confirm, I believe the value passed to the 'bytes' key should be:
>>
>> bytes - (chunk_size*(chunk_size-1))
>>
>> Kris
>>
>>
>>
>> On 2011-11-02, at 9:18 PM, Michael Terry wrote:
>>
>>> On 2 November 2011 19:34, Kris <address@hidden> wrote:
>>>> I'm taking a look at it again right now. Just a FYI before I dig in, I
>>>> removed the volsize cmd line and it did the same thing. The default
>>>> volsize is 25MB in duplicity and that's what the tmp file is. The remote
>>>> file however is 50 megs. The PoolWorkers split the chunk in to two 25 meg
>>>> chunks and uploaded those, which isn't proper.
>>>
>>> Good debugging work, Kris. This isn't bug 498933 after all, but is
>>> instead a bug in 0.6.16 specifically. In that version, the S3 backend
>>> got new 'multichunk' support.
>>>
>>> But it seems to have a bug that when the size of the volume isn't
>>> perfectly divisible by the multichunk size, it will upload too much
>>> data. I've filed bug https://bugs.launchpad.net/duplicity/+bug/885513
>>> about it.
>>>
>>> For now, I believe you can workaround it with
>>> --s3-multipart-chunk-size=1000 (causing duplicity to basically ignore
>>> the multipart code).
>>>
>>> I feel like suggesting that maybe the multipart branch should be
>>> backed out until it can be stress tested a bit more? (so far we've
>>> noticed this bug and bug 878220 (the one where botobackend.py added a
>>> new, uncommon, and unconditional import due to the patch)
>>>
>>> -mt
>>
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden
>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>
>
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, (continued)
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Michael Terry, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, edgar . soldin, 2011/11/02
- Message not available
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, edgar . soldin, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Kris, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, edgar . soldin, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Kris, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Michael Terry, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Kris, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Kris, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Michael Terry, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://,
Kris <=
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, edgar . soldin, 2011/11/03
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Kris, 2011/11/03
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Kris, 2011/11/03