Re: [Duplicity-talk] Performance issues w/ S3

[edgar . soldin]

On 12.07.2024 05:06, tink via Duplicity-talk wrote:
> On Fri, 12 Jul 2024 at 10:14, edgar.soldin--- via Duplicity-talk 
> <duplicity-talk@nongnu.org <mailto:duplicity-talk@nongnu.org>> wrote:
>
>     On 11.07.2024 23:56, tink via Duplicity-talk wrote:
>      > Hi,
>
> Hi Edgar,
>
>      > I've been banging my head against this for a while. (duplicity 2.2.3)
>      > I'm trying to backup a sizeable chunk of data to Wasabi S3 storage, and 
> using boto3+s3 with
>
>     boto+s3:// was removed a while ago. you can simply use s3:// now
>
> Tried that ... no change.

wasn't supposed to. just stated that the boto3 backend is used anyway.

>      >     --volsize 200 \
>      >     --s3-multipart-chunk-size 20 \
>      >     --s3-multipart-max-procs 6 \
>
>     does removing --s3-multipart-chunk-size, --s3-multipart-max-procs change 
> anything?
>
> As with the above change - no difference.
>
>
>      > am seeing horribly slow upload speeds (shuffling 400GB in 24 h, so 
> roughly 5 MB/s).
>      >
>      > Using aws cli tools on the same server to the same bucket I get 80MB/s.
>      > Looking at a tcpdump during both attempts the latter looks fine and 
> swift, the duplicity one is seeing the far end throttling me badly, every other 
> second it tells me to back off for 1.5 seconds (setting a win-size of 0 for that 
> period of time).
>
>     maybe some boto3 issue? do you have the latest version? 
> https://pypi.orgo/project/boto3/ <https://pypi.org/project/boto3/>
>
> Using the boto3 that comes packaged w/ the distro, 1.34.108, so somewhat behind 
> but not drastically ...

how do you know it's not a bug they fixed in between? worth a try.

sunny regards ..ede