Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?

[edgar . soldin]

On 31.05.2015 17:51, Remy van Elst wrote:
> 
> 
> On 05/31/2015 02:22 PM, address@hidden wrote:
>> [SNIP]
>>>
>>>> weird, can you send a list of the files on your backend? looks 
>>>> like not all were uploaded successfully.
>>>
>>>
>>> Here's a list of all the files in the backend:
>>>
>>> https://3335ae642c4a42549b7a4489adf98d7c.objectstore.eu/duplicity/dup
> lic
>>>
>>>
>>>
> ity-3.txt
>>>
> 
>> ok, comparing this with your first duplicity.txt output (line 325) 
>> shows that duplicity only lists ~9900 files while there are ~74000 
>> online.
> 
>> this partial "blindness" let's it find only some backups,
>> seemingly the oldest ones.
> 
>> this reminds me of 
>> https://answers.launchpad.net/duplicity/+question/241395 where a 
>> paramiko bug shortened the list of files on the backend.
> 
>> another possibility would be a bug within python, i seem to
>> loosely remember that this was the case at one time in past on
>> MacOS though.
> 
> 
>> your leads from here (please report your progress) 1. update to 
>> latest swift-client. doublecheck that the python based swift cmd 
>> line client does indeed list _all_ files. 2. update python to the 
>> latest stable 3. research problems with python, swift and
>> truncated file listings
> 
>> ..ede/duply.net
> 
>> _______________________________________________ Duplicity-talk 
>> mailing list address@hidden 
>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
> 
> 
> I might have located a bug.
> 
> The duplicity swift module for listing a backend uses the following code
> :
> 
> def _list(self):
>         headers, objs = self.conn.get_container(self.container)
>         return [o['name'] for o in objs]
> 
> 
> The Swift client documentation states the following for this function:
> 
> http://docs.openstack.org/developer/python-swiftclient/swiftclient.html#
> swiftclient.client.get_container
> 
>  swiftclient.client.get_container(url, token, container, marker=None,
> limit=None, prefix=None, delimiter=None, end_marker=None, path=None,
> http_conn=None, full_listing=False)
> 
> full_listing – if True, return a full listing, else returns a max of
> 10000 listings
> 
> This small addition to the swiftbackend seems to fix this issue. I now
> get the full contents of the container, here's the new collection-status
> :
> 
> 
> 
> 
>     # Hostname: vps1.example.org
>     # Storage used: 1.9T
>     # Full backups to keep: 4
>     # Create full backup if last full backup is older than: 14D
>     -----------------------------------------
>     # Duplicity collection status:
>     # Local and Remote metadata are synchronized, no sync needed.
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150427T0
> 34927Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150418T0
> 34914Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150421T0
> 34917Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150424T0
> 34932Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150425T0
> 34922Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150419T0
> 35000Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150420T0
> 34952Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150417T0
> 34949Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150428T0
> 34902Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150423T0
> 34926Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150422T0
> 34928Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150426T0
> 34916Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150502T0
> 34959Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150509T0
> 35017Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150501T0
> 34946Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150512T0
> 34953Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150514T0
> 34927Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150507T0
> 35120Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150513T0
> 34941Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150506T0
> 34921Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150504T0
> 35029Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150508T0
> 35145Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150503T0
> 34948Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150505T0
> 34933Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150511T0
> 35016Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150418T0
> 34914Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150419T0
> 35000Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150420T0
> 34952Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150421T0
> 34917Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150422T0
> 34928Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150423T0
> 34926Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150424T0
> 34932Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150425T0
> 34922Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150426T0
> 34916Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150427T0
> 34927Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150428T0
> 34902Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150429T0
> 34952Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150502T0
> 34959Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150503T0
> 34948Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150504T0
> 35029Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150505T0
> 34933Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150506T0
> 34921Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150507T0
> 35120Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150508T0
> 35145Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150509T0
> 35017Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150510T0
> 35016Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150511T0
> 35016Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150512T0
> 34953Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150513T0
> 34941Z.sigtar.gz
>     #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150514T0
> 34927Z.sigtar.gz
>     # Last full backup date: Fri May 29 13:49:11 2015
>     # Collection Status
>     # -----------------
>     # Connecting with backend: BackendWrapper
>     # Archive dir: /root/.cache/duplicity/vps1.example.org.
>     # Found 17 secondary backup chains.
>     # Secondary chain 1 of 17:
>     # -------------------------
>     # Chain start time: Tue Mar 24 14:42:44 2015
>     # Chain end time: Tue Apr  7 05:49:27 2015
>     # Number of contained backup sets: 14
>     # Total number of contained volumes: 5532
>     #  Type of backup set:                            Time:      Num
> volumes:
>     #                 Full         Tue Mar 24 14:42:44 2015
>    3009
>     #          Incremental         Wed Mar 25 05:49:32 2015
>     170
>     #          Incremental         Fri Mar 27 05:49:24 2015
>     198
>     #          Incremental         Sat Mar 28 05:49:28 2015
>     199
>     #          Incremental         Sun Mar 29 05:49:12 2015
>     188
>     #          Incremental         Mon Mar 30 05:49:27 2015
>     194
>     #          Incremental         Tue Mar 31 05:49:37 2015
>     194
>     #          Incremental         Wed Apr  1 05:49:27 2015
>     195
>     #          Incremental         Thu Apr  2 05:51:05 2015
>     198
>     #          Incremental         Fri Apr  3 05:50:07 2015
>     196
>     #          Incremental         Sat Apr  4 05:50:10 2015
>     218
>     #          Incremental         Sun Apr  5 05:49:59 2015
>     187
>     #          Incremental         Mon Apr  6 05:49:42 2015
>     193
>     #          Incremental         Tue Apr  7 05:49:27 2015
>     193
>     # -------------------------
>     # Secondary chain 2 of 17:
>     # -------------------------
>     # Chain start time: Wed Apr  8 05:49:18 2015
>     # Chain end time: Wed Apr 15 05:52:25 2015
>     # Number of contained backup sets: 8
>     # Total number of contained volumes: 4514
>     #  Type of backup set:                            Time:      Num
> volumes:
>     #                 Full         Wed Apr  8 05:49:18 2015
>    3104
>     #          Incremental         Thu Apr  9 05:49:18 2015
>     241
>     #          Incremental         Fri Apr 10 05:49:23 2015
>     195
>     #          Incremental         Sat Apr 11 05:49:03 2015
>     194
>     #          Incremental         Sun Apr 12 05:49:03 2015
>     194
>     #          Incremental         Mon Apr 13 05:49:08 2015
>     193
>     #          Incremental         Tue Apr 14 05:49:09 2015
>     196
>     #          Incremental         Wed Apr 15 05:52:25 2015
>     197
>     # -------------------------
>     # Secondary chain 3 of 17:
>     # -------------------------
>     # Chain start time: Thu Apr 16 05:49:37 2015
>     # Chain end time: Wed Apr 29 05:49:52 2015
>     # Number of contained backup sets: 2
>     # Total number of contained volumes: 3401
>     #  Type of backup set:                            Time:      Num
> volumes:
>     #                 Full         Thu Apr 16 05:49:37 2015
>    3128
>     #          Incremental         Wed Apr 29 05:49:52 2015
>     273
>     # -------------------------
>     # Secondary chain 4 of 17:
>     # -------------------------
>     # Chain start time: Thu Apr 30 05:49:39 2015
>     # Chain end time: Thu May 14 05:49:27 2015
>     # Number of contained backup sets: 2
>     # Total number of contained volumes: 3467
>     #  Type of backup set:                            Time:      Num
> volumes:
>     #                 Full         Thu Apr 30 05:49:39 2015
>    3206
>     #          Incremental         Thu May 14 05:49:27 2015
>     261
>     # -------------------------
>     # Secondary chain 5 of 17:
>     # -------------------------
>     # Chain start time: Fri May 15 05:50:13 2015
>     # Chain end time: Fri May 15 05:50:13 2015
>     # Number of contained backup sets: 1
>     # Total number of contained volumes: 3270
>     #  Type of backup set:                            Time:      Num
> volumes:
>     #                 Full         Fri May 15 05:50:13 2015
>    3270
>     # -------------------------
>     # Secondary chain 6 of 17:
>     # -------------------------
>     # Chain start time: Sat May 16 05:51:23 2015
>     # Chain end time: Sat May 16 05:51:23 2015
>     # Number of contained backup sets: 1
>     # Total number of contained volumes: 3324
>     #  Type of backup set:                            Time:      Num
> volumes:
>     #                 Full         Sat May 16 05:51:23 2015
>    3324
>     # -------------------------
>     # Secondary chain 7 of 17:
>     # -------------------------
>     # Chain start time: Sun May 17 05:52:07 2015
>     # Chain end time: Sun May 17 05:52:07 2015
>     # Number of contained backup sets: 1
>     # Total number of contained volumes: 3378
>     #  Type of backup set:                            Time:      Num
> volumes:
>     #                 Full         Sun May 17 05:52:07 2015
>    3378
>     # -------------------------
>     # Secondary chain 8 of 17:
>     # -------------------------
>     # Chain start time: Mon May 18 05:52:13 2015
>     # Chain end time: Mon May 18 05:52:13 2015
>     # Number of contained backup sets: 1
>     # Total number of contained volumes: 3432
>     #  Type of backup set:                            Time:      Num
> volumes:
>     #                 Full         Mon May 18 05:52:13 2015
>    3432
>     # -------------------------
>     # Secondary chain 9 of 17:
>     # -------------------------
>     # Chain start time: Tue May 19 05:50:27 2015
>     # Chain end time: Tue May 19 05:50:27 2015
>     # Number of contained backup sets: 1
>     # Total number of contained volumes: 3488
>     #  Type of backup set:                            Time:      Num
> volumes:
>     #                 Full         Tue May 19 05:50:27 2015
>    3488
>     # -------------------------
>     # Secondary chain 10 of 17:
>     # -------------------------
>     # Chain start time: Wed May 20 05:50:34 2015
>     # Chain end time: Wed May 20 05:50:34 2015
>     # Number of contained backup sets: 1
>     # Total number of contained volumes: 3544
>     #  Type of backup set:                            Time:      Num
> volumes:
>     #                 Full         Wed May 20 05:50:34 2015
>    3544
>     # -------------------------
>     # Secondary chain 11 of 17:
>     # -------------------------
>     # Chain start time: Thu May 21 05:49:33 2015
>     # Chain end time: Thu May 21 05:49:33 2015
>     # Number of contained backup sets: 1
>     # Total number of contained volumes: 3607
>     #  Type of backup set:                            Time:      Num
> volumes:
>     #                 Full         Thu May 21 05:49:33 2015
>    3607
>     # -------------------------
>     # Secondary chain 12 of 17:
>     # -------------------------
>     # Chain start time: Fri May 22 05:51:50 2015
>     # Chain end time: Fri May 22 05:51:50 2015
>     # Number of contained backup sets: 1
>     # Total number of contained volumes: 3664
>     #  Type of backup set:                            Time:      Num
> volumes:
>     #                 Full         Fri May 22 05:51:50 2015
>    3664
>     # -------------------------
>     # Secondary chain 13 of 17:
>     # -------------------------
>     # Chain start time: Sat May 23 05:50:52 2015
>     # Chain end time: Sat May 23 05:50:52 2015
>     # Number of contained backup sets: 1
>     # Total number of contained volumes: 3720
>     #  Type of backup set:                            Time:      Num
> volumes:
>     #                 Full         Sat May 23 05:50:52 2015
>    3720
>     # -------------------------
>     # Secondary chain 14 of 17:
>     # -------------------------
>     # Chain start time: Sun May 24 05:49:21 2015
>     # Chain end time: Sun May 24 05:49:21 2015
>     # Number of contained backup sets: 1
>     # Total number of contained volumes: 3776
>     #  Type of backup set:                            Time:      Num
> volumes:
>     #                 Full         Sun May 24 05:49:21 2015
>    3776
>     # -------------------------
>     # Secondary chain 15 of 17:
>     # -------------------------
>     # Chain start time: Mon May 25 05:49:42 2015
>     # Chain end time: Mon May 25 05:49:42 2015
>     # Number of contained backup sets: 1
>     # Total number of contained volumes: 3833
>     #  Type of backup set:                            Time:      Num
> volumes:
>     #                 Full         Mon May 25 05:49:42 2015
>    3833
>     # -------------------------
>     # Secondary chain 16 of 17:
>     # -------------------------
>     # Chain start time: Tue May 26 05:49:16 2015
>     # Chain end time: Tue May 26 05:49:16 2015
>     # Number of contained backup sets: 1
>     # Total number of contained volumes: 3890
>     #  Type of backup set:                            Time:      Num
> volumes:
>     #                 Full         Tue May 26 05:49:16 2015
>    3890
>     # -------------------------
>     # Secondary chain 17 of 17:
>     # -------------------------
>     # Chain start time: Thu May 28 15:59:52 2015
>     # Chain end time: Thu May 28 15:59:52 2015
>     # Number of contained backup sets: 1
>     # Total number of contained volumes: 3953
>     #  Type of backup set:                            Time:      Num
> volumes:
>     #                 Full         Thu May 28 15:59:52 2015
>    3953
>     # -------------------------
>     # Found primary backup chain with matching signature chain:
>     # -------------------------
>     # Chain start time: Fri May 29 13:49:11 2015
>     # Chain end time: Fri May 29 13:49:11 2015
>     # Number of contained backup sets: 1
>     # Total number of contained volumes: 4009
>     #  Type of backup set:                            Time:      Num
> volumes:
>     #                 Full         Fri May 29 13:49:11 2015
>    4009
>     # -------------------------
>     # Also found 0 backup sets not part of any chain,
>     # and 3 incomplete backup sets.
>     # These may be deleted by running duplicity with the "cleanup"
> command.
> 
> 
> 
> Restoring also seems to work just fine.
> 
> Pull/merge request:
> https://code.launchpad.net/~raymii/duplicity/fix-swiftbackend-max-10000-
> files-in-list/+merge/260660
> 
> 

btw. 74k files is quite a lot.. you might want to consider raising your volume 
size.

..ede/duply.net