|
| From: | Tashrif |
| Subject: | Re: [Duplicity-talk] How to backup stuff>5GB |
| Date: | Fri, 25 Feb 2022 18:24:34 -0500 |
or stick with boto2 and use the parameters mentioned.
On 26.02.2022 00:11, Tashrif wrote:
> did you try the prefix 'boto3+s3://' which according to man page section 'A NOTE ON AMAZON S3' does it automatically and is the recent
>
>
> Yes, that is what I reported in my last email. Again, you are right--I found the idea from A NOTE ON AMAZON S3. Right now, I am trying to confirm that boto3+s3 backend performs multi-part upload automatically.
>
or stick with boto2 and use the parameters mentioned. i seem to recall that they were added after users ran into the file size per PUT operation limit. but yes, generally try to use current code.
Ken: shouldn't we make boto3+s3:// default by now? alias it to s3:// ? rename the others boto2+...:// ?
..ede
> -Tashrif
>
> On Fri, Feb 25, 2022 at 5:55 PM edgar.soldin--- via Duplicity-talk <duplicity-talk@nongnu.org <mailto:duplicity-talk@nongnu.org>> wrote:
>
> hey Tashrif,
>
> reads here, that the limitiation to 5GB can be avoided by using multi-chunk uploads
> https://aws.amazon.com/s3/faqs/#:~:text=Individual%20Amazon%20S3%20objects%20can,using%20the%20Multipart%20Upload%20capability <https://aws.amazon.com/s3/faqs/#:~:text=Individual%20Amazon%20S3%20objects%20can,using%20the%20Multipart%20Upload%20capability>.
>
> did you try '--s3-use-multiprocessing', '--s3-multipart-chunk-size' as documented in the man page
> https://duplicity.gitlab.io/duplicity-web/vers8/duplicity.1.html <https://duplicity.gitlab.io/duplicity-web/vers8/duplicity.1.html>
>
> did you try the prefix 'boto3+s3://' which according to man page section 'A NOTE ON AMAZON S3' does it automatically and is the recent, still maintained library to access s3?
>
> so many questions :).. ede/duply.net <http://duply.net>
>
> On 25.02.2022 21:14, Tashrif via Duplicity-talk wrote:
> > Thank you for the reference, Ken. In the meantime, I want to hack it. Which line in duplicity performs put request? Can I put an if condition for sigtar before that line so it is not attempted to put? And then I upload that sigtar using `aws s3 cp` command?
> >
> > Best,
> > Tashrif
> >
> > On Fri, Feb 25, 2022 at 12:40 PM Kenneth Loafman <kenneth@loafman.com <mailto:kenneth@loafman.com> <mailto:kenneth@loafman.com <mailto:kenneth@loafman.com>>> wrote:
> >
> > No, it's due to this bug: https://bugs.launchpad.net/duplicity/+bug/385495 <https://bugs.launchpad.net/duplicity/+bug/385495> <https://bugs.launchpad.net/duplicity/+bug/385495 <https://bugs.launchpad.net/duplicity/+bug/385495>>
> >
> > I am working on the next major revision to duplicity, 0.9.x, which will fix this and some others. It's going slowly.
> >
> > ...Thanks,
> > ...Ken
> >
> >
> > On Fri, Feb 25, 2022 at 10:49 AM Tashrif <tashrifbillah@gmail.com <mailto:tashrifbillah@gmail.com> <mailto:tashrifbillah@gmail.com <mailto:tashrifbillah@gmail.com>>> wrote:
> >
> > Hi Kenneth,
> >
> > No, aws did not split the file in the bucket. I have been doing quite a bit of research on it. I see the following code segment in s3_boto3_backend only:
> >
> > duplicity/backends/s3_boto3_backend.py:141: transfer_config = TransferConfig(multipart_chunksize=config.s3_multipart_chunk_size,
> > duplicity/backends/s3_boto3_backend.py:142: multipart_threshold=config.s3_multipart_chunk_size)
> >
> > But I have used TARGET="s3+http://my_bucket <http://my_bucket> <http://my_bucket <http://my_bucket>>" which should be the old boto. Do you think the latter has anything to do with this error?
> >
> > Best,
> > Tashrif
> >
> > On Fri, Feb 25, 2022 at 11:43 AM Kenneth Loafman <kenneth@loafman.com <mailto:kenneth@loafman.com> <mailto:kenneth@loafman.com <mailto:kenneth@loafman.com>>> wrote:
> >
> > Hi Tashrif,
> >
> > The sigtar size problem has been around forever. For now I suggest splitting the backup into smaller portions.
> >
> > I am surprised the aws command completes properly. Did it split the file in the bucket?
> >
> > ...Ken
> >
> >
> > On Thu, Feb 24, 2022 at 10:57 PM Tashrif via Duplicity-talk <duplicity-talk@nongnu.org <mailto:duplicity-talk@nongnu.org> <mailto:duplicity-talk@nongnu.org <mailto:duplicity-talk@nongnu.org>>> wrote:
> >
> > During a backup task, duplicity created a 7.5 GB file at the very end: duplicity-full-signatures.20220222T150726Z.sigtar.gz. However, its upload fails with the following traceback:
> >
> > ```
> > File "min3-duply/lib/python3.9/site-packages/boto/s3/key.py", line 760, in send_file
> > self._send_file_internal(fp, headers=headers, cb=cb, num_cb=num_cb,
> > File "min3-duply/lib/python3.9/site-packages/boto/s3/key.py", line 957, in _send_file_internal
> > resp = self.bucket.connection.make_request(
> > File "min3-duply/lib/python3.9/site-packages/boto/s3/connection.py", line 667, in make_request
> > return super(S3Connection, self).make_request(
> > File "min3-duply/lib/python3.9/site-packages/boto/connection.py", line 1077, in make_request
> > return self._mexe(http_request, sender, override_num_retries,
> > File "min3-duply/lib/python3.9/site-packages/boto/connection.py", line 946, in _mexe
> > response = sender(connection, request.method, request.path,
> > File "min3-duply/lib/python3.9/site-packages/boto/s3/key.py", line 895, in sender
> > raise provider.storage_response_error(
> > boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
> > <?xml version="1.0" encoding="UTF-8"?>
> > <Error><Code>EntityTooLarge</Code><Message>Your proposed upload exceeds the maximum allowed size</Message><ProposedSize>7422574715</ProposedSize><MaxSizeAllowed>5368709120</MaxSizeAllowed><RequestId>HJD8DQ49S18RBFWQ</RequestId><HostId>7t7enU1YX/HY7ho7qA74knGEIzerBk/hDogp=</HostId></Error>
> >
> > Attempt of move Nr. 1 failed. S3ResponseError: Bad Request
> > ```
> >
> > Meanwhile, `aws s3 cp duplicity-full-signatures.20220222T150726Z.sigtar.gz s3://my_bucket/` succeeds gracefully. That said, how do I enable duply/duplicity to upload files larger than 5GB?
> >
> > Thank you,
> > Tashrif
> > _______________________________________________
> > Duplicity-talk mailing list
> > Duplicity-talk@nongnu.org <mailto:Duplicity-talk@nongnu.org> <mailto:Duplicity-talk@nongnu.org <mailto:Duplicity-talk@nongnu.org>>
> > https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk> <https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>>
> >
> >
> > _______________________________________________
> > Duplicity-talk mailing list
> > Duplicity-talk@nongnu.org <mailto:Duplicity-talk@nongnu.org>
> > https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>
>
>
> _______________________________________________
> Duplicity-talk mailing list
> Duplicity-talk@nongnu.org <mailto:Duplicity-talk@nongnu.org>
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>
>
| [Prev in Thread] | Current Thread | [Next in Thread] |