Re: [Duplicity-talk] OSError: [Errno 24] Too many open files

[edgar . soldin]

On 21.07.2020 10:46, Diagon via Duplicity-talk wrote:
> Date: Mon, 20 Jul 2020 11:48:04 +0200
>
>> On 20.07.2020 07:16, Diagon via Duplicity-talk wrote:
>>> Ubuntu 16.04, and yes I'm still running 0.7.19. Everything was fine until 
>>> my cron scheduled backups stopped without warning on June 6th, as I just 
>>> discovered.
>>>
>>> Command line that looks something like:
>>>
>>> PASSPHRASE="xxx" duplicity --log-file /home/me/duplicity.log 
>>> --backend-retry-delay 60 --asynchronous-upload --name Remote --volsize 50 
>>> --full-if-older-than 6M --exclude '**.lock' /home/dev/mydir 
>>> sftp://me@server.com/Backup
>>>
>>> Error below. Any way around this until I get a new OS installed?
>>>
>>> Thanks -
>>>
>>> /usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141: 
>>> FutureWarning: CTR mode needs counter parameter, not IV
> SNIP
>>> . File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 
>>> 374, in run
>>> . create_fhs, attach_fhs)
>>> . File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 
>>> 402, in _attach_fork_exec
>>> . pipe = os.pipe()
>>> . OSError: [Errno 24] Too many open files
>
>> for now you can work around the issue by raising the open files limit via 
>> ulimit
>> https://linuxhandbook.com/ulimit-command/
>
> Is that on the server or on the client that I have to raise that limit?

on the box duplicity is running on.

>> would you mind posting a 'duplicity collection-status ...' of your backup. i 
>> would guess you some really very very long chain in there. maybe it is time 
>> to do a new full?
>
> I'm backing up this directory every 20 minutes, but almost all of the time 
> it's only one file changing.  So it looks like the following:
>
> $ duplicity collection-status sftp://me@server.com/Backup
> /usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141: 
> FutureWarning: CTR mode needs counter parameter, not IV
>   self._cipher = factory.new(key, *args, **kwargs)
>   Last full backup date: Wed May 13 22:20:03 2020
>   Collection Status
>   -----------------
>   Connecting with backend: BackendWrapper
>   Archive dir: /home/me/.cache/duplicity/xxxxxxxxxxxxxxxxxxxxxx
>
> Found 0 secondary backup chains.
>
> Found primary backup chain with matching signature chain:
> -------------------------
> Chain start time: Wed May 13 22:20:03 2020
> Chain end time: Mon Jun 15 17:40:06 2020
> Number of contained backup sets: 1007
> Total number of contained volumes: 1052
> Type of backup set:                            Time:      Num volumes:
> Full                Wed May 13 22:20:03 2020                46
> Incremental         Thu May 14 03:40:04 2020                 1
>
> <That last line, with different date/times and Num volumes = 1, repeats 1006 
> times>

how do you manage to do 1000+ incrementals between 14 May and today (21 July)?

1000+ incrementals is a long chain. consider doing a new full. and generally 
doing them monthly or so.

..ede/duply.net