Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?

[Remy van Elst]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256



On 05/31/2015 02:22 PM, address@hidden wrote:
> [SNIP]
>> 
>>> weird, can you send a list of the files on your backend? looks 
>>> like not all were uploaded successfully.
>> 
>> 
>> Here's a list of all the files in the backend:
>> 
>> https://3335ae642c4a42549b7a4489adf98d7c.objectstore.eu/duplicity/dup
lic
>>
>>
>> 
ity-3.txt
>> 
> 
> ok, comparing this with your first duplicity.txt output (line 325) 
> shows that duplicity only lists ~9900 files while there are ~74000 
> online.
> 
> this partial "blindness" let's it find only some backups,
> seemingly the oldest ones.
> 
> this reminds me of 
> https://answers.launchpad.net/duplicity/+question/241395 where a 
> paramiko bug shortened the list of files on the backend.
> 
> another possibility would be a bug within python, i seem to
> loosely remember that this was the case at one time in past on
> MacOS though.
> 
> 
> your leads from here (please report your progress) 1. update to 
> latest swift-client. doublecheck that the python based swift cmd 
> line client does indeed list _all_ files. 2. update python to the 
> latest stable 3. research problems with python, swift and
> truncated file listings
> 
> ..ede/duply.net
> 
> _______________________________________________ Duplicity-talk 
> mailing list address@hidden 
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 


I might have located a bug.

The duplicity swift module for listing a backend uses the following code
:

def _list(self):
        headers, objs = self.conn.get_container(self.container)
        return [o['name'] for o in objs]


The Swift client documentation states the following for this function:

http://docs.openstack.org/developer/python-swiftclient/swiftclient.html#
swiftclient.client.get_container

 swiftclient.client.get_container(url, token, container, marker=None,
limit=None, prefix=None, delimiter=None, end_marker=None, path=None,
http_conn=None, full_listing=False)

full_listing – if True, return a full listing, else returns a max of
10000 listings

This small addition to the swiftbackend seems to fix this issue. I now
get the full contents of the container, here's the new collection-status
:




    # Hostname: vps1.example.org
    # Storage used: 1.9T
    # Full backups to keep: 4
    # Create full backup if last full backup is older than: 14D
    -----------------------------------------
    # Duplicity collection status:
    # Local and Remote metadata are synchronized, no sync needed.
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150427T0
34927Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150418T0
34914Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150421T0
34917Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150424T0
34932Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150425T0
34922Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150419T0
35000Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150420T0
34952Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150417T0
34949Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150428T0
34902Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150423T0
34926Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150422T0
34928Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150426T0
34916Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150502T0
34959Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150509T0
35017Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150501T0
34946Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150512T0
34953Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150514T0
34927Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150507T0
35120Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150513T0
34941Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150506T0
34921Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150504T0
35029Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150508T0
35145Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150503T0
34948Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150505T0
34933Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150511T0
35016Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150418T0
34914Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150419T0
35000Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150420T0
34952Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150421T0
34917Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150422T0
34928Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150423T0
34926Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150424T0
34932Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150425T0
34922Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150426T0
34916Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150427T0
34927Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150428T0
34902Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150429T0
34952Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150502T0
34959Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150503T0
34948Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150504T0
35029Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150505T0
34933Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150506T0
34921Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150507T0
35120Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150508T0
35145Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150509T0
35017Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150510T0
35016Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150511T0
35016Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150512T0
34953Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150513T0
34941Z.sigtar.gz
    #
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150514T0
34927Z.sigtar.gz
    # Last full backup date: Fri May 29 13:49:11 2015
    # Collection Status
    # -----------------
    # Connecting with backend: BackendWrapper
    # Archive dir: /root/.cache/duplicity/vps1.example.org.
    # Found 17 secondary backup chains.
    # Secondary chain 1 of 17:
    # -------------------------
    # Chain start time: Tue Mar 24 14:42:44 2015
    # Chain end time: Tue Apr  7 05:49:27 2015
    # Number of contained backup sets: 14
    # Total number of contained volumes: 5532
    #  Type of backup set:                            Time:      Num
volumes:
    #                 Full         Tue Mar 24 14:42:44 2015
   3009
    #          Incremental         Wed Mar 25 05:49:32 2015
    170
    #          Incremental         Fri Mar 27 05:49:24 2015
    198
    #          Incremental         Sat Mar 28 05:49:28 2015
    199
    #          Incremental         Sun Mar 29 05:49:12 2015
    188
    #          Incremental         Mon Mar 30 05:49:27 2015
    194
    #          Incremental         Tue Mar 31 05:49:37 2015
    194
    #          Incremental         Wed Apr  1 05:49:27 2015
    195
    #          Incremental         Thu Apr  2 05:51:05 2015
    198
    #          Incremental         Fri Apr  3 05:50:07 2015
    196
    #          Incremental         Sat Apr  4 05:50:10 2015
    218
    #          Incremental         Sun Apr  5 05:49:59 2015
    187
    #          Incremental         Mon Apr  6 05:49:42 2015
    193
    #          Incremental         Tue Apr  7 05:49:27 2015
    193
    # -------------------------
    # Secondary chain 2 of 17:
    # -------------------------
    # Chain start time: Wed Apr  8 05:49:18 2015
    # Chain end time: Wed Apr 15 05:52:25 2015
    # Number of contained backup sets: 8
    # Total number of contained volumes: 4514
    #  Type of backup set:                            Time:      Num
volumes:
    #                 Full         Wed Apr  8 05:49:18 2015
   3104
    #          Incremental         Thu Apr  9 05:49:18 2015
    241
    #          Incremental         Fri Apr 10 05:49:23 2015
    195
    #          Incremental         Sat Apr 11 05:49:03 2015
    194
    #          Incremental         Sun Apr 12 05:49:03 2015
    194
    #          Incremental         Mon Apr 13 05:49:08 2015
    193
    #          Incremental         Tue Apr 14 05:49:09 2015
    196
    #          Incremental         Wed Apr 15 05:52:25 2015
    197
    # -------------------------
    # Secondary chain 3 of 17:
    # -------------------------
    # Chain start time: Thu Apr 16 05:49:37 2015
    # Chain end time: Wed Apr 29 05:49:52 2015
    # Number of contained backup sets: 2
    # Total number of contained volumes: 3401
    #  Type of backup set:                            Time:      Num
volumes:
    #                 Full         Thu Apr 16 05:49:37 2015
   3128
    #          Incremental         Wed Apr 29 05:49:52 2015
    273
    # -------------------------
    # Secondary chain 4 of 17:
    # -------------------------
    # Chain start time: Thu Apr 30 05:49:39 2015
    # Chain end time: Thu May 14 05:49:27 2015
    # Number of contained backup sets: 2
    # Total number of contained volumes: 3467
    #  Type of backup set:                            Time:      Num
volumes:
    #                 Full         Thu Apr 30 05:49:39 2015
   3206
    #          Incremental         Thu May 14 05:49:27 2015
    261
    # -------------------------
    # Secondary chain 5 of 17:
    # -------------------------
    # Chain start time: Fri May 15 05:50:13 2015
    # Chain end time: Fri May 15 05:50:13 2015
    # Number of contained backup sets: 1
    # Total number of contained volumes: 3270
    #  Type of backup set:                            Time:      Num
volumes:
    #                 Full         Fri May 15 05:50:13 2015
   3270
    # -------------------------
    # Secondary chain 6 of 17:
    # -------------------------
    # Chain start time: Sat May 16 05:51:23 2015
    # Chain end time: Sat May 16 05:51:23 2015
    # Number of contained backup sets: 1
    # Total number of contained volumes: 3324
    #  Type of backup set:                            Time:      Num
volumes:
    #                 Full         Sat May 16 05:51:23 2015
   3324
    # -------------------------
    # Secondary chain 7 of 17:
    # -------------------------
    # Chain start time: Sun May 17 05:52:07 2015
    # Chain end time: Sun May 17 05:52:07 2015
    # Number of contained backup sets: 1
    # Total number of contained volumes: 3378
    #  Type of backup set:                            Time:      Num
volumes:
    #                 Full         Sun May 17 05:52:07 2015
   3378
    # -------------------------
    # Secondary chain 8 of 17:
    # -------------------------
    # Chain start time: Mon May 18 05:52:13 2015
    # Chain end time: Mon May 18 05:52:13 2015
    # Number of contained backup sets: 1
    # Total number of contained volumes: 3432
    #  Type of backup set:                            Time:      Num
volumes:
    #                 Full         Mon May 18 05:52:13 2015
   3432
    # -------------------------
    # Secondary chain 9 of 17:
    # -------------------------
    # Chain start time: Tue May 19 05:50:27 2015
    # Chain end time: Tue May 19 05:50:27 2015
    # Number of contained backup sets: 1
    # Total number of contained volumes: 3488
    #  Type of backup set:                            Time:      Num
volumes:
    #                 Full         Tue May 19 05:50:27 2015
   3488
    # -------------------------
    # Secondary chain 10 of 17:
    # -------------------------
    # Chain start time: Wed May 20 05:50:34 2015
    # Chain end time: Wed May 20 05:50:34 2015
    # Number of contained backup sets: 1
    # Total number of contained volumes: 3544
    #  Type of backup set:                            Time:      Num
volumes:
    #                 Full         Wed May 20 05:50:34 2015
   3544
    # -------------------------
    # Secondary chain 11 of 17:
    # -------------------------
    # Chain start time: Thu May 21 05:49:33 2015
    # Chain end time: Thu May 21 05:49:33 2015
    # Number of contained backup sets: 1
    # Total number of contained volumes: 3607
    #  Type of backup set:                            Time:      Num
volumes:
    #                 Full         Thu May 21 05:49:33 2015
   3607
    # -------------------------
    # Secondary chain 12 of 17:
    # -------------------------
    # Chain start time: Fri May 22 05:51:50 2015
    # Chain end time: Fri May 22 05:51:50 2015
    # Number of contained backup sets: 1
    # Total number of contained volumes: 3664
    #  Type of backup set:                            Time:      Num
volumes:
    #                 Full         Fri May 22 05:51:50 2015
   3664
    # -------------------------
    # Secondary chain 13 of 17:
    # -------------------------
    # Chain start time: Sat May 23 05:50:52 2015
    # Chain end time: Sat May 23 05:50:52 2015
    # Number of contained backup sets: 1
    # Total number of contained volumes: 3720
    #  Type of backup set:                            Time:      Num
volumes:
    #                 Full         Sat May 23 05:50:52 2015
   3720
    # -------------------------
    # Secondary chain 14 of 17:
    # -------------------------
    # Chain start time: Sun May 24 05:49:21 2015
    # Chain end time: Sun May 24 05:49:21 2015
    # Number of contained backup sets: 1
    # Total number of contained volumes: 3776
    #  Type of backup set:                            Time:      Num
volumes:
    #                 Full         Sun May 24 05:49:21 2015
   3776
    # -------------------------
    # Secondary chain 15 of 17:
    # -------------------------
    # Chain start time: Mon May 25 05:49:42 2015
    # Chain end time: Mon May 25 05:49:42 2015
    # Number of contained backup sets: 1
    # Total number of contained volumes: 3833
    #  Type of backup set:                            Time:      Num
volumes:
    #                 Full         Mon May 25 05:49:42 2015
   3833
    # -------------------------
    # Secondary chain 16 of 17:
    # -------------------------
    # Chain start time: Tue May 26 05:49:16 2015
    # Chain end time: Tue May 26 05:49:16 2015
    # Number of contained backup sets: 1
    # Total number of contained volumes: 3890
    #  Type of backup set:                            Time:      Num
volumes:
    #                 Full         Tue May 26 05:49:16 2015
   3890
    # -------------------------
    # Secondary chain 17 of 17:
    # -------------------------
    # Chain start time: Thu May 28 15:59:52 2015
    # Chain end time: Thu May 28 15:59:52 2015
    # Number of contained backup sets: 1
    # Total number of contained volumes: 3953
    #  Type of backup set:                            Time:      Num
volumes:
    #                 Full         Thu May 28 15:59:52 2015
   3953
    # -------------------------
    # Found primary backup chain with matching signature chain:
    # -------------------------
    # Chain start time: Fri May 29 13:49:11 2015
    # Chain end time: Fri May 29 13:49:11 2015
    # Number of contained backup sets: 1
    # Total number of contained volumes: 4009
    #  Type of backup set:                            Time:      Num
volumes:
    #                 Full         Fri May 29 13:49:11 2015
   4009
    # -------------------------
    # Also found 0 backup sets not part of any chain,
    # and 3 incomplete backup sets.
    # These may be deleted by running duplicity with the "cleanup"
command.



Restoring also seems to work just fine.

Pull/merge request:
https://code.launchpad.net/~raymii/duplicity/fix-swiftbackend-max-10000-
files-in-list/+merge/260660
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCAAGBQJVay4DAAoJECtnVb0bf4jchVAH/R8RdgOyaztk3VpDnYMvR1Cx
klhYIoTscrYPWyYiFyi8acDlFRzkRgv6wYxOaRZNk5JOFZ3/HhKMmSl0THDAK7iR
EZkyFIOmEQZwOE7RuN47QvmjiAA04BPAecHEsl5h0P7RvcZenne8RRDz/14lR3/e
y0gmYwthbqAxlmz3uMPD1/a/KMRJjAJ93xlgTABZp3aKoWM+TUQ5dIcIbBjRzVSI
y+sxZVLEJJAlG6/dRwlnEgIOO+MKHYC5eRpHdZLGbMkzVbB5jx9PyY2kWPqqAzqf
LJSmKCW4AzqRmdKoKQxlSqyshsspAAtzTxtkqx7rW1cJ3gctH7/aRL5tNKs7mo8=
=x1sF
-----END PGP SIGNATURE-----

0x1B7F88DC.asc
Description: application/pgp-keys

0x1B7F88DC.asc.sig
Description: PGP signature