Re: [Duplicity-talk] Files left behind when deleting full backups sets

[Elvis Stansvik]

Replying to self...

I can't believe I missed that this was a known and fixed bug (since
this Wednesday) [1].

Sorry for the noise!

So this bug is really fixed now? Or was the "fix" just making sure
cleanup --extra-clean is more thorough? When is 0.6.20 due btw?

Regards,
Elvis

[1] https://bugs.launchpad.net/duplicity/+bug/519948

2012/10/21 Elvis Stansvik <address@hidden>:
> Good evening,
>
> I have some strange behavior I can't explain. Let me show with a very
> simple test case:
>
> address@hidden ~/dup_test]$ rm -rf ~/.cache
> address@hidden ~/dup_test]$ mkdir source target
> address@hidden ~/dup_test]$ touch source/testfile
>
> So we have our source and target. Let's create the first backup set:
>
> address@hidden ~/dup_test]$ duplicity --no-encryption
> --no-print-statistics source file://target
> Local and Remote metadata are synchronized, no sync needed.
> Last full backup date: none
> No signatures found, switching to full backup.
>
> Since this was the first time, Duplicity made a full set. Let's make
> an incremental as well:
>
> address@hidden ~/dup_test]$ duplicity --no-encryption
> --no-print-statistics source file://target
> Local and Remote metadata are synchronized, no sync needed.
> Last full backup date: Sun Oct 21 18:37:17 2012
>
> Let's do the same thing again, and create another full backup, with an
> incremental on top:
>
> address@hidden ~/dup_test]$ duplicity full --no-encryption
> --no-print-statistics source file://target
> Local and Remote metadata are synchronized, no sync needed.
> Last full backup date: Sun Oct 21 18:37:17 2012
> address@hidden ~/dup_test]$ duplicity --no-encryption
> --no-print-statistics source file://target
> Local and Remote metadata are synchronized, no sync needed.
> Last full backup date: Sun Oct 21 18:37:41 2012
>
> Now let's see what it looks like:
>
> address@hidden ~/dup_test]$ duplicity --no-encryption collection-status
> file://target
> Local and Remote metadata are synchronized, no sync needed.
> Last full backup date: Sun Oct 21 18:37:41 2012
> Collection Status
> -----------------
> Connecting with backend: LocalBackend
> Archive dir: /home/dup_test/.cache/duplicity/2334326747ca7273e1b6291b0ce4d508
>
> Found 1 secondary backup chain.
> Secondary chain 1 of 1:
> -------------------------
> Chain start time: Sun Oct 21 18:37:17 2012
> Chain end time: Sun Oct 21 18:37:25 2012
> Number of contained backup sets: 2
> Total number of contained volumes: 2
>  Type of backup set:                            Time:      Num volumes:
>                 Full         Sun Oct 21 18:37:17 2012                 1
>          Incremental         Sun Oct 21 18:37:25 2012                 1
> -------------------------
>
>
> Found primary backup chain with matching signature chain:
> -------------------------
> Chain start time: Sun Oct 21 18:37:41 2012
> Chain end time: Sun Oct 21 18:37:52 2012
> Number of contained backup sets: 2
> Total number of contained volumes: 2
>  Type of backup set:                            Time:      Num volumes:
>                 Full         Sun Oct 21 18:37:41 2012                 1
>          Incremental         Sun Oct 21 18:37:52 2012                 1
> -------------------------
> No orphaned or incomplete backup sets found.
>
> Looks very good. We have two chains, each with one full backup set and
> one incremental set.
>
> Let's have a look at the backup files:
>
> address@hidden ~/dup_test]$ ls -1 target/
> duplicity-full-signatures.20121021T163717Z.sigtar.gz
> duplicity-full-signatures.20121021T163741Z.sigtar.gz
> duplicity-full.20121021T163717Z.manifest
> duplicity-full.20121021T163717Z.vol1.difftar.gz
> duplicity-full.20121021T163741Z.manifest
> duplicity-full.20121021T163741Z.vol1.difftar.gz
> duplicity-inc.20121021T163717Z.to.20121021T163725Z.manifest
> duplicity-inc.20121021T163717Z.to.20121021T163725Z.vol1.difftar.gz
> duplicity-inc.20121021T163741Z.to.20121021T163752Z.manifest
> duplicity-inc.20121021T163741Z.to.20121021T163752Z.vol1.difftar.gz
> duplicity-new-signatures.20121021T163717Z.to.20121021T163725Z.sigtar.gz
> duplicity-new-signatures.20121021T163741Z.to.20121021T163752Z.sigtar.gz
>
> This looks fine right? First comes the sigtars, difftars and manifests
> for the two full backups. Then comes the the same for the two
> incrementals.
>
> Now let's try to remove the older of the two full backup sets. Here's
> where the strange behavior begins!
>
> address@hidden ~/dup_test]$ duplicity remove-all-but-n-full 1 --force
> --no-encryption file://target
> Local and Remote metadata are synchronized, no sync needed.
> Last full backup date: Sun Oct 21 18:37:41 2012
> Deleting backup sets at times:
> Sun Oct 21 18:37:17 2012
> Sun Oct 21 18:37:25 2012
> Deleting set inc Sun Oct 21 18:37:25 2012
> Deleting set full Sun Oct 21 18:37:17 2012
> Warning, found the following local orphaned signature file:
> duplicity-new-signatures.20121021T163717Z.to.20121021T163725Z.sigtar.gz
>
> It looks good to begin with. It correctly identified the full backup
> set from 18:37:17 and its associated incremental backups set from
> 18:37:25, and it says it deleted them.
>
> (What about that warning though? Why can't it simply remove the
> orphaned signature from the cache?)
>
> But what is even stranger is when we look at the backup files again:
>
> address@hidden ~/dup_test]$ ls -1 target/
> duplicity-full-signatures.20121021T163717Z.sigtar.gz     <-- ??
> duplicity-full-signatures.20121021T163741Z.sigtar.gz
> duplicity-full.20121021T163741Z.manifest
> duplicity-full.20121021T163741Z.vol1.difftar.gz
> duplicity-inc.20121021T163741Z.to.20121021T163752Z.manifest
> duplicity-inc.20121021T163741Z.to.20121021T163752Z.vol1.difftar.gz
> duplicity-new-signatures.20121021T163717Z.to.20121021T163725Z.sigtar.gz
>     <-- ??
> duplicity-new-signatures.20121021T163741Z.to.20121021T163752Z.sigtar.gz
>
> It didn't delete the sets fully! There's still the sigtars of the full
> and incremental set left (indicated above).
>
> Looking at the cache directory, we see the same problem, but here only
> the sigtar of the (deleted) incremental set is still there:
>
> address@hidden ~/dup_test]$ ls -1
> ~/.cache/duplicity/2334326747ca7273e1b6291b0ce4d508/
> duplicity-full-signatures.20121021T163741Z.sigtar.gz
> duplicity-full.20121021T163741Z.manifest
> duplicity-inc.20121021T163741Z.to.20121021T163752Z.manifest
> duplicity-new-signatures.20121021T163717Z.to.20121021T163725Z.sigtar.gz
>    <-- ??
> duplicity-new-signatures.20121021T163741Z.to.20121021T163752Z.sigtar.gz
>
> Why is it that Duplicity leaves these files behind?
>
> For my purposes, I want a backup routine like:
>
> Mon - Full backup + Remove everything but the last 2 full backups
> Tue - Incremental
> Wed - Incremental
> Thu - Incremental
> Fri - Incremental
> Sat - Incremental
> Sun - Incremental
>
> Which means I'd always have the option of restoring from any day in
> the last 1-2 weeks.
>
> But if Duplicity leaves files behind like this when deleting sets,
> it's just going to pile up, and I really don't want that. I want all
> files associated with those sets to be deleted, both from the backup
> location and the cache.
>
> So, is this a bug or some intended behavior of some kind?
>
> Best regards,
> Elvis Stansvik