Re: [Duplicity-talk] sigtar file: AccessDenied error using S3 Glacier after upgrade to 0.8.23

[edgar . soldin]

hey Bernhard,

can't try it now.

the man page though states
https://duplicity.us/vers8/duplicity.1.html
"
--s3-use-glacier
Store volumes using Glacier Flexible Storage when uploading to Amazon S3. This 
storage class has a lower cost of storage but a higher per-request cost along 
with delays of up to 12 hours from the time of retrieval request. This storage 
cost is calculated against a 90-day storage minimum. According to Amazon this 
storage is ideal for data archiving and long-term backup offering 99.999999999% 
durability. To restore a backup you will have to manually migrate all data 
stored on AWS Glacier back to Standard S3 and wait for AWS to complete the 
migration.

NOTE: Duplicity will store the manifest.gpg files from full and incremental 
backups on AWS S3 standard storage to allow quick retrieval for later 
incremental backups, all other data is stored in S3 Glacier.
"
the statement suggests that this should work. a quick check in 
https://gitlab.com/duplicity/duplicity/-/blob/main/duplicity/backends/_boto_single.py#L244
 shows that the old boto backend did the same (only exclude manifests).

did you try to use an empty backup target, create a full and some incrementals? 
if it does indeed not excluding signatures surely is an option, but needs to be 
documented.

sunny regards ..ede/duply.net

On 18.07.2022 09:05, Bernhard Patsch via Duplicity-talk wrote:
> Hi Ede,
>
> Are you saying only incremental backups fail on Glacier?
>
> At least in my setup, duplicity only works if Glacier is disabled. My 
> configuration is based on the man page and various tutorials.
> If there are any parameters I can tweak to make it working again, pls let me 
> know.
>
> The biggest benefit of Glacier is cost. The downside is that data is not 
> instantly available.
>
> I understand that restoring from Glacier requires manual steps but in many 
> cases the lower price outweighs the extra cost.
> I would argue IF excluding the signature file from Glacier would allow 
> (incremental?) backups, it would extend the use-cases and improve the current 
> implementation.
> In my setup with incremental backup, I have to remove the --s3-use-glacier 
> option and therefore have to pay the extra cost for standard class for 
> metadata+payload.
>
> Thoughts?
>
> Kind regards,
> Bernhard
>
> Am Fr., 15. Juli 2022 um 11:16 Uhr schrieb edgar.soldin--- via Duplicity-talk 
> <duplicity-talk@nongnu.org <mailto:duplicity-talk@nongnu.org>>:
>
>     hey Bernhard,
>
>     On 15.07.2022 07:06, Bernhard Patsch via Duplicity-talk wrote:
>      > Hello,
>      >
>      > I use duplicity with an S3 Glacier bucket from Scaleway. After upgrading 
> to 0.8.23 and enabling boto3, the backup fails with the following error:
>      >
>      > % /usr/bin/duplicity -v3 --full-if-older-than 1M --s3-use-new-style --s3-use-glacier 
> --s3-endpoint-url https://s3.nl-ams.scw.cloud <https://s3.nl-ams.scw.cloud> 
> <https://s3.nl-ams.scw.cloud <https://s3.nl-ams.scw.cloud>> --s3-region-name nl-ams 
> --asynchronous-upload --s3-multipart-chunk-size 10 --metadata-sync-mode partial --exclude-device-files 
> --include=/etc --include=/data/Pictures '--exclude=**' / boto3+s3://<bucket-name>/duplicity
>      >
>      > Synchronizing remote metadata to local cache...
>      >  >>> Copying duplicity-full-signatures.20220611T132657Z.sigtar.gpg to 
> local cache.
>      >  >>> Attempt of get Nr. 1 failed. ClientError: An error occurred (AccessDenied) 
> when calling the >>> GetObject operation: Access Denied.
>      >
>      > I understand that the current boto3 implementation does not support 
> restore from Glacier storage.
>
>     changing the target url, changes '--archive-dir' path, so a new one needs 
> to be downloaded. that's halfway to a restore on a different machine.
>
>      > According to the man page, the manifest file is always stored in 
> standard class.
>
>     yes, because only them are needed for a successful incremental if 
> archive-dir is intact.
>
>      > However, the sigtar file is stored in Glacier class when the option 
> s3_use_clacier is set.
>      > The error is gone when the sigtar file is manually restored to standard 
> class.
>
>     that's indeed needed. check the comment in the backend source
>     
> https://gitlab.com/duplicity/duplicity/-/blob/main/duplicity/backends/s3_boto3_backend.py#L31-43
>  
> <https://gitlab.com/duplicity/duplicity/-/blob/main/duplicity/backends/s3_boto3_backend.py#L31-43>
>
>      > I assume that the storage class should be set explicitely for the sigtar 
> file, similar to the handling for the manifest in s3_boto3_backend.py:123++?
>      > If this assumption is correct, I can raise a bug ticket for that problem.
>
>     i'd say it is a bit but not fully. a described above the sig is not 
> strictly needed and might become quite big over time as it is not split 
> currently.
>
>     sunny regards ..ede/duply.net <http://duply.net>
>
>     _______________________________________________
>     Duplicity-talk mailing list
>     Duplicity-talk@nongnu.org <mailto:Duplicity-talk@nongnu.org>
>     https://lists.nongnu.org/mailman/listinfo/duplicity-talk 
> <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>
>
>
> _______________________________________________
> Duplicity-talk mailing list
> Duplicity-talk@nongnu.org
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk