Thanks for your valuable input, Ede.
I'll give ot a try on the weekend.
hey Bernhard,
can't try it now.
the man page though states
https://duplicity.us/vers8/duplicity.1.html
"
--s3-use-glacier
Store volumes using Glacier Flexible Storage when uploading to Amazon S3. This storage class has a lower cost of storage but a higher per-request cost along with delays of up to 12 hours from the time of retrieval request. This storage cost is calculated against a 90-day storage minimum. According to Amazon this storage is ideal for data archiving and long-term backup offering 99.999999999% durability. To restore a backup you will have to manually migrate all data stored on AWS Glacier back to Standard S3 and wait for AWS to complete the migration.
NOTE: Duplicity will store the manifest.gpg files from full and incremental backups on AWS S3 standard storage to allow quick retrieval for later incremental backups, all other data is stored in S3 Glacier.
"
the statement suggests that this should work. a quick check in https://gitlab.com/duplicity/duplicity/-/blob/main/duplicity/backends/_boto_single.py#L244 shows that the old boto backend did the same (only exclude manifests).
did you try to use an empty backup target, create a full and some incrementals? if it does indeed not excluding signatures surely is an option, but needs to be documented.
sunny regards ..ede/duply.net
On 18.07.2022 09:05, Bernhard Patsch via Duplicity-talk wrote:
> Hi Ede,
>
> Are you saying only incremental backups fail on Glacier?
>
> At least in my setup, duplicity only works if Glacier is disabled. My configuration is based on the man page and various tutorials.
> If there are any parameters I can tweak to make it working again, pls let me know.
>
> The biggest benefit of Glacier is cost. The downside is that data is not instantly available.
>
> I understand that restoring from Glacier requires manual steps but in many cases the lower price outweighs the extra cost.
> I would argue IF excluding the signature file from Glacier would allow (incremental?) backups, it would extend the use-cases and improve the current implementation.
> In my setup with incremental backup, I have to remove the --s3-use-glacier option and therefore have to pay the extra cost for standard class for metadata+payload.
>
> Thoughts?
>
> Kind regards,
> Bernhard
>
> Am Fr., 15. Juli 2022 um 11:16 Uhr schrieb edgar.soldin--- via Duplicity-talk <duplicity-talk@nongnu.org <mailto:duplicity-talk@nongnu.org>>:
>
> hey Bernhard,
>
> On 15.07.2022 07:06, Bernhard Patsch via Duplicity-talk wrote:
> > Hello,
> >
> > I use duplicity with an S3 Glacier bucket from Scaleway. After upgrading to 0.8.23 and enabling boto3, the backup fails with the following error:
> >
> > % /usr/bin/duplicity -v3 --full-if-older-than 1M --s3-use-new-style --s3-use-glacier --s3-endpoint-url https://s3.nl-ams.scw.cloud <https://s3.nl-ams.scw.cloud> <https://s3.nl-ams.scw.cloud <https://s3.nl-ams.scw.cloud>> --s3-region-name nl-ams --asynchronous-upload --s3-multipart-chunk-size 10 --metadata-sync-mode partial --exclude-device-files --include=/etc --include=/data/Pictures '--exclude=**' / boto3+s3://<bucket-name>/duplicity
> >
> > Synchronizing remote metadata to local cache...
> > >>> Copying duplicity-full-signatures.20220611T132657Z.sigtar.gpg to local cache.
> > >>> Attempt of get Nr. 1 failed. ClientError: An error occurred (AccessDenied) when calling the >>> GetObject operation: Access Denied.
> >
> > I understand that the current boto3 implementation does not support restore from Glacier storage.
>
> changing the target url, changes '--archive-dir' path, so a new one needs to be downloaded. that's halfway to a restore on a different machine.
>
> > According to the man page, the manifest file is always stored in standard class.
>
> yes, because only them are needed for a successful incremental if archive-dir is intact.
>
> > However, the sigtar file is stored in Glacier class when the option s3_use_clacier is set.
> > The error is gone when the sigtar file is manually restored to standard class.
>
> that's indeed needed. check the comment in the backend source
> https://gitlab.com/duplicity/duplicity/-/blob/main/duplicity/backends/s3_boto3_backend.py#L31-43 <https://gitlab.com/duplicity/duplicity/-/blob/main/duplicity/backends/s3_boto3_backend.py#L31-43>
>
> > I assume that the storage class should be set explicitely for the sigtar file, similar to the handling for the manifest in s3_boto3_backend.py:123++?
> > If this assumption is correct, I can raise a bug ticket for that problem.
>
> i'd say it is a bit but not fully. a described above the sig is not strictly needed and might become quite big over time as it is not split currently.
>
> sunny regards ..ede/duply.net <http://duply.net>
>
> _______________________________________________
> Duplicity-talk mailing list
> Duplicity-talk@nongnu.org <mailto:Duplicity-talk@nongnu.org>
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>
>
>
> _______________________________________________
> Duplicity-talk mailing list
> Duplicity-talk@nongnu.org
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
_______________________________________________
Duplicity-talk mailing list
Duplicity-talk@nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk