Re: [Duplicity-talk] OSError: [Errno 24] Too many open files

[Edgar Soldin]

true they close every time .. the only thing i see is that the amount of
gpg processes grows with the amount of incrementals ...

@yang: how many incrementals does your  backup hold?

the other thing is, maybe gpg itself is buggy  ..

@yang: could you try another gpg version?

and the third thing

@ken: is it really impossible for 0.5.17 to use an oldGnuPGInterface
version, that might be laying around on the system? How could yang look
for it?

.. 2cents again :) ..ede
--
> I tried this with tiny volume sizes of 1M over a 2GB backup (2K volumes)
> and did a count of the gpg processes (live, defunct) every 2 seconds and
> never had more than 5 (3 live, 2 defunct).  This was on a very fast link
> between local VMs.  Tried both backup and restore so it would test both
> directions.  By the end, all were closed.
>
> It's possible to have some defunct while the task scheduler is updating,
> but Yang is showing hundreds that are holding files open.  That's the
> confusing part.
>
> ...Ken
>
> Edgar Soldin wrote:
>   
>> i really have to be fast (a window of 10s let's say) and it helps to
>> have a slow backup connection with lots of incrementals... but I may be
>> seeing ghosts here. Can it that it is perfectly normal for duplicity to
>> gpg decrypt something per each incremental but close all processes just
>> a bit later? .. just a guess
>> ..ede
>>     
>>> I cannot reproduce this on a 32bit machine at all.  I'll be able to test
>>> on a 64bit machine sometime this week and, hopefully, find the problem.
>>>
>>> ...Ken
>>>
>>> address@hidden wrote:
>>>   
>>>       
>>>> i seem to get closer to it ...
>>>> the more incrementals (regardless if file: or ftp: backend) the more
>>>> defunct gpg processes appear .. but dissappear again while the backup is
>>>> still running
>>>>
>>>> I did
>>>> initial backup of 3GB
>>>> touched a new test file to have a change & did a 2nd backup
>>>> touched a new test2 file to have a change & did a 3nd backup
>>>> ...
>>>>
>>>> and since the 3rd backup the defunct gpg processes keep appearing
>>>>
>>>> essentially I say: Take any more than 3 incrementals in size backup and
>>>> do a new incremental and you will see those zombie gpg processes ..
>>>>
>>>> while I am not sure that this is what happens on the ulimit problem I
>>>> thought I should report these observations ... ede
>>>>
>>>> PS:
>>>>
>>>> here a 'ps xf -u user' shortly after starting the sixth run .. the
>>>> number of gpg zombies seems to be number of sets (full + incr's) minus 2 ..
>>>>
>>>> 9122 pts/0    S+     0:00          \_ /bin/bash
>>>> /srv/www/user//release/ftplicity_1.4.2/ftplicity test backup
>>>> 19128 pts/0    S+     0:00              \_ /bin/bash
>>>> /srv/www/user//release/ftplicity_1.4.2/ftplicity backup
>>>> 19193 pts/0    S+     0:00                  \_ /bin/bash
>>>> /srv/www/user//release/ftplicity_1.4.2/ftplicity backup
>>>> 19194 pts/0    R+     0:03                      \_ /usr/bin/python
>>>> /srv/www/user/_apps/duplicity-0.5.10/bin/duplicity --verbosity 4
>>>> --encrypt-key 00000000--sign-key 00000000--gpg-options=--always-trust
>>>> --volsize 1 --exclude-globbing-filelist /srv/www/user//.ftplicity/test/excl
>>>> 19196 pts/0    SL+    0:00                          \_ gpg --logger-fd 4
>>>> --passphrase-fd 8 --batch --no-tty --default-key 00000000--recipient
>>>> 00000000--no-secmem-warning --always-trust --encrypt --sign
>>>> 19197 pts/0    SL+    0:00                          \_ gpg --status-fd 6
>>>> --passphrase-fd 11 --logger-fd 5 --batch --no-tty --default-key
>>>> 00000000--no-secmem-warning --always-trust --decrypt
>>>> 19199 pts/0    Z+     0:00                          \_ [gpg] <defunct>
>>>> 19200 pts/0    Z+     0:00                          \_ [gpg] <defunct>
>>>> 19201 pts/0    Z+     0:00                          \_ [gpg] <defunct>
>>>> 19202 pts/0    Z+     0:00                          \_ [gpg] <defunct>
>>>> 19203 pts/0    SL+    0:00                          \_ gpg --logger-fd
>>>> 23 --passphrase-fd 28 --batch --no-tty --default-key 00000000--recipient
>>>> 00000000--no-secmem-warning --always-trust --encrypt --sign
>>>>
>>>>
>>>>     
>>>>         
>>>>> hmm .. you are right these processes are zombie (they appear in top as
>>>>> such) .. later on the processes disappear, while duplicity is still
>>>>> running ..
>>>>> but I don't feel this is normal.. or is it?
>>>>> just to be more sure, I'll try a 1MB chunk full backup of my 3GB home
>>>>> dir .. let's see with how many zombies I'll end up ...
>>>>>
>>>>> interestingly my ulimit = unlimited ... seems to be default in the old
>>>>> SUSE 10.2, btw also on my debian 5.0 box both 32bit
>>>>> can't remember to ever have touched this setting
>>>>>
>>>>> ..ede
>>>>> -- 
>>>>>       
>>>>>           
>>>>>> These are not open processes, they are zombie processes, so no real
>>>>>> resources taken.  That simplifies it a bit.  What normally causes this
>>>>>> is the failure of a parent process to properly retrieve its exit status,
>>>>>> but this is not the case or more people would have this problem.
>>>>>>
>>>>>> As far as I can tell, the only systems this is happening on are 64bit,
>>>>>> and not even all of those.  Some of my systems are 64bit and they don't
>>>>>> show this, so I'm wondering if this could be limited to a particular
>>>>>> version of GnuPG, or what.
>>>>>>
>>>>>> Edgar, from your note I could not tell, is this happening to you too?
>>>>>>
>>>>>> ...Ken
>>>>>>
>>>>>> address@hidden wrote:
>>>>>>  
>>>>>>         
>>>>>>             
>>>>>>> just my quick observation .. a simple incr backup opens and leaves over
>>>>>>> 20 gpg processes open. I don't think this is healthy and also can
>>>>>>> imagine that this multiplies if the volumes get smaller (mine are
>>>>>>> 50MB).
>>>>>>>
>>>>>>> regards ede
>>>>>>>
>>>>>>>
>>>>>>> address@hidden:~> ps -u user xf
>>>>>>>  PID TTY      STAT   TIME COMMAND
>>>>>>> 32731 ?        S      0:00 sshd: address@hidden/0
>>>>>>> 32732 pts/0    Ss     0:00  \_ -bash
>>>>>>> 1167 pts/0    S      0:00      \_ /bin/bash /usr/local/bin/ftplicity
>>>>>>> profile backup
>>>>>>> 1168 pts/0    S      0:00      |   \_ /bin/bash
>>>>>>> /srv/www//release/ftplicity_1.4.2/ftplicity profile backup
>>>>>>> 1174 pts/0    S      0:00      |       \_ /bin/bash
>>>>>>> /srv/www//release/ftplicity_1.4.2/ftplicity backup
>>>>>>> 1239 pts/0    S      0:00      |           \_ /bin/bash
>>>>>>> /srv/www//release/ftplicity_1.4.2/ftplicity backup
>>>>>>> 1240 pts/0    S      0:05      |               \_ /usr/bin/python
>>>>>>> /srv/www/user/_apps/duplicity-0.5.10/bin/duplicity --verbosity 4
>>>>>>> --encrypt-key XXXXXXX --sign-key B59ECD99 --gpg-options=--always-trust
>>>>>>> --full-if-older-than 1M --volsize 50 --exclude-globbing-filelist
>>>>>>> /srv/www/...
>>>>>>> 1249 pts/0    SL     0:00      |                   \_ gpg --logger-fd 4
>>>>>>> --passphrase-fd 8 --batch --no-tty --default-key XXXXXXXX --recipient
>>>>>>> XXXXXXXX --no-secmem-warning --always-trust --encrypt --sign
>>>>>>> 1265 pts/0    SL     0:00      |                   \_ gpg --status-fd 6
>>>>>>> --passphrase-fd 11 --logger-fd 5 --batch --no-tty --default-key
>>>>>>> XXXXXXXX
>>>>>>> --no-secmem-warning --always-trust --decrypt
>>>>>>> 1267 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>>> 1272 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>>> 1275 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>>> 1278 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>>> 1282 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>>> 1286 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>>> 1289 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>>> 1292 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>>> 1296 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>>> 1299 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>>> 1302 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>>> 1305 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>>> 1308 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>>> 1311 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>>> 1314 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>>> 1317 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>>> 1320 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>>> 1324 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>>> 1327 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>>> 1330 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>>> 1333 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>>> 1334 pts/1    Ss+    0:00      |                   \_ /usr/bin/ncftp -u
>>>>>>> ******* backup.server.de
>>>>>>> 1335 pts/0    R+     0:00      \_ ps -u user xf
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Duplicity-talk mailing list
>>>>>>> address@hidden
>>>>>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>>>>>
>>>>>>>     
>>>>>>>           
>>>>>>>               
>>>>>>  
>>>>>> ------------------------------------------------------------------------
>>>>>>
>>>>>> _______________________________________________
>>>>>> Duplicity-talk mailing list
>>>>>> address@hidden
>>>>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>>>>   
>>>>>>         
>>>>>>             
>>>>> _______________________________________________
>>>>> Duplicity-talk mailing list
>>>>> address@hidden
>>>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>>>       
>>>>>           
>>>> _______________________________________________
>>>> Duplicity-talk mailing list
>>>> address@hidden
>>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>>
>>>>     
>>>>         
>>>   
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>   
>>>       
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>
>>     
>
>
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>