|
From: | Tashrif |
Subject: | Re: [Duplicity-talk] How to backup stuff>5GB |
Date: | Fri, 25 Feb 2022 17:19:26 -0500 |
Thank you for the reference, Ken. In the meantime, I want to hack it. Which line in duplicity performs put request? Can I put an if condition for sigtar before that line so it is not attempted to put? And then I upload that sigtar using `aws s3 cp` command?Best,TashrifOn Fri, Feb 25, 2022 at 12:40 PM Kenneth Loafman <kenneth@loafman.com> wrote:No, it's due to this bug: https://bugs.launchpad.net/duplicity/+bug/385495I am working on the next major revision to duplicity, 0.9.x, which will fix this and some others. It's going slowly....Thanks,...KenOn Fri, Feb 25, 2022 at 10:49 AM Tashrif <tashrifbillah@gmail.com> wrote:Hi Kenneth,No, aws did not split the file in the bucket. I have been doing quite a bit of research on it. I see the following code segment in s3_boto3_backend only:duplicity/backends/s3_boto3_backend.py:141: transfer_config = TransferConfig(multipart_chunksize=config.s3_multipart_chunk_size,
duplicity/backends/s3_boto3_backend.py:142: multipart_threshold=config.s3_multipart_chunk_size)But I have used TARGET="s3+http://my_bucket" which should be the old boto. Do you think the latter has anything to do with this error?Best,
TashrifOn Fri, Feb 25, 2022 at 11:43 AM Kenneth Loafman <kenneth@loafman.com> wrote:Hi Tashrif,The sigtar size problem has been around forever. For now I suggest splitting the backup into smaller portions.I am surprised the aws command completes properly. Did it split the file in the bucket?...KenOn Thu, Feb 24, 2022 at 10:57 PM Tashrif via Duplicity-talk <duplicity-talk@nongnu.org> wrote:During a backup task, duplicity created a 7.5 GB file at the very end: duplicity-full-signatures.20220222T150726Z.sigtar.gz. However, its upload fails with the following traceback:_______________________________________________```File "min3-duply/lib/python3.9/site-packages/boto/s3/key.py", line 760, in send_file
self._send_file_internal(fp, headers=headers, cb=cb, num_cb=num_cb,
File "min3-duply/lib/python3.9/site-packages/boto/s3/key.py", line 957, in _send_file_internal
resp = self.bucket.connection.make_request(
File "min3-duply/lib/python3.9/site-packages/boto/s3/connection.py", line 667, in make_request
return super(S3Connection, self).make_request(
File "min3-duply/lib/python3.9/site-packages/boto/connection.py", line 1077, in make_request
return self._mexe(http_request, sender, override_num_retries,
File "min3-duply/lib/python3.9/site-packages/boto/connection.py", line 946, in _mexe
response = sender(connection, request.method, request.path,
File "min3-duply/lib/python3.9/site-packages/boto/s3/key.py", line 895, in sender
raise provider.storage_response_error(
boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>EntityTooLarge</Code><Message>Your proposed upload exceeds the maximum allowed size</Message><ProposedSize>7422574715</ProposedSize><MaxSizeAllowed>5368709120</MaxSizeAllowed><RequestId>HJD8DQ49S18RBFWQ</RequestId><HostId>7t7enU1YX/HY7ho7qA74knGEIzerBk/hDogp=</HostId></Error>
Attempt of move Nr. 1 failed. S3ResponseError: Bad Request```Meanwhile, `aws s3 cp duplicity-full-signatures.20220222T150726Z.sigtar.gz s3://my_bucket/` succeeds gracefully. That said, how do I enable duply/duplicity to upload files larger than 5GB?Thank you,Tashrif
Duplicity-talk mailing list
Duplicity-talk@nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
[Prev in Thread] | Current Thread | [Next in Thread] |