Re: [Duplicity-talk] OSError: [Errno 24] Too many open files

[Kenneth Loafman]

I tried this with tiny volume sizes of 1M over a 2GB backup (2K volumes)
and did a count of the gpg processes (live, defunct) every 2 seconds and
never had more than 5 (3 live, 2 defunct).  This was on a very fast link
between local VMs.  Tried both backup and restore so it would test both
directions.  By the end, all were closed.

It's possible to have some defunct while the task scheduler is updating,
but Yang is showing hundreds that are holding files open.  That's the
confusing part.

...Ken

Edgar Soldin wrote:
> i really have to be fast (a window of 10s let's say) and it helps to
> have a slow backup connection with lots of incrementals... but I may be
> seeing ghosts here. Can it that it is perfectly normal for duplicity to
> gpg decrypt something per each incremental but close all processes just
> a bit later? .. just a guess
> ..ede
>> I cannot reproduce this on a 32bit machine at all.  I'll be able to test
>> on a 64bit machine sometime this week and, hopefully, find the problem.
>>
>> ...Ken
>>
>> address@hidden wrote:
>>   
>>> i seem to get closer to it ...
>>> the more incrementals (regardless if file: or ftp: backend) the more
>>> defunct gpg processes appear .. but dissappear again while the backup is
>>> still running
>>>
>>> I did
>>> initial backup of 3GB
>>> touched a new test file to have a change & did a 2nd backup
>>> touched a new test2 file to have a change & did a 3nd backup
>>> ...
>>>
>>> and since the 3rd backup the defunct gpg processes keep appearing
>>>
>>> essentially I say: Take any more than 3 incrementals in size backup and
>>> do a new incremental and you will see those zombie gpg processes ..
>>>
>>> while I am not sure that this is what happens on the ulimit problem I
>>> thought I should report these observations ... ede
>>>
>>> PS:
>>>
>>> here a 'ps xf -u user' shortly after starting the sixth run .. the
>>> number of gpg zombies seems to be number of sets (full + incr's) minus 2 ..
>>>
>>> 9122 pts/0    S+     0:00          \_ /bin/bash
>>> /srv/www/user//release/ftplicity_1.4.2/ftplicity test backup
>>> 19128 pts/0    S+     0:00              \_ /bin/bash
>>> /srv/www/user//release/ftplicity_1.4.2/ftplicity backup
>>> 19193 pts/0    S+     0:00                  \_ /bin/bash
>>> /srv/www/user//release/ftplicity_1.4.2/ftplicity backup
>>> 19194 pts/0    R+     0:03                      \_ /usr/bin/python
>>> /srv/www/user/_apps/duplicity-0.5.10/bin/duplicity --verbosity 4
>>> --encrypt-key 00000000--sign-key 00000000--gpg-options=--always-trust
>>> --volsize 1 --exclude-globbing-filelist /srv/www/user//.ftplicity/test/excl
>>> 19196 pts/0    SL+    0:00                          \_ gpg --logger-fd 4
>>> --passphrase-fd 8 --batch --no-tty --default-key 00000000--recipient
>>> 00000000--no-secmem-warning --always-trust --encrypt --sign
>>> 19197 pts/0    SL+    0:00                          \_ gpg --status-fd 6
>>> --passphrase-fd 11 --logger-fd 5 --batch --no-tty --default-key
>>> 00000000--no-secmem-warning --always-trust --decrypt
>>> 19199 pts/0    Z+     0:00                          \_ [gpg] <defunct>
>>> 19200 pts/0    Z+     0:00                          \_ [gpg] <defunct>
>>> 19201 pts/0    Z+     0:00                          \_ [gpg] <defunct>
>>> 19202 pts/0    Z+     0:00                          \_ [gpg] <defunct>
>>> 19203 pts/0    SL+    0:00                          \_ gpg --logger-fd
>>> 23 --passphrase-fd 28 --batch --no-tty --default-key 00000000--recipient
>>> 00000000--no-secmem-warning --always-trust --encrypt --sign
>>>
>>>
>>>     
>>>> hmm .. you are right these processes are zombie (they appear in top as
>>>> such) .. later on the processes disappear, while duplicity is still
>>>> running ..
>>>> but I don't feel this is normal.. or is it?
>>>> just to be more sure, I'll try a 1MB chunk full backup of my 3GB home
>>>> dir .. let's see with how many zombies I'll end up ...
>>>>
>>>> interestingly my ulimit = unlimited ... seems to be default in the old
>>>> SUSE 10.2, btw also on my debian 5.0 box both 32bit
>>>> can't remember to ever have touched this setting
>>>>
>>>> ..ede
>>>> -- 
>>>>       
>>>>> These are not open processes, they are zombie processes, so no real
>>>>> resources taken.  That simplifies it a bit.  What normally causes this
>>>>> is the failure of a parent process to properly retrieve its exit status,
>>>>> but this is not the case or more people would have this problem.
>>>>>
>>>>> As far as I can tell, the only systems this is happening on are 64bit,
>>>>> and not even all of those.  Some of my systems are 64bit and they don't
>>>>> show this, so I'm wondering if this could be limited to a particular
>>>>> version of GnuPG, or what.
>>>>>
>>>>> Edgar, from your note I could not tell, is this happening to you too?
>>>>>
>>>>> ...Ken
>>>>>
>>>>> address@hidden wrote:
>>>>>  
>>>>>         
>>>>>> just my quick observation .. a simple incr backup opens and leaves over
>>>>>> 20 gpg processes open. I don't think this is healthy and also can
>>>>>> imagine that this multiplies if the volumes get smaller (mine are
>>>>>> 50MB).
>>>>>>
>>>>>> regards ede
>>>>>>
>>>>>>
>>>>>> address@hidden:~> ps -u user xf
>>>>>>  PID TTY      STAT   TIME COMMAND
>>>>>> 32731 ?        S      0:00 sshd: address@hidden/0
>>>>>> 32732 pts/0    Ss     0:00  \_ -bash
>>>>>> 1167 pts/0    S      0:00      \_ /bin/bash /usr/local/bin/ftplicity
>>>>>> profile backup
>>>>>> 1168 pts/0    S      0:00      |   \_ /bin/bash
>>>>>> /srv/www//release/ftplicity_1.4.2/ftplicity profile backup
>>>>>> 1174 pts/0    S      0:00      |       \_ /bin/bash
>>>>>> /srv/www//release/ftplicity_1.4.2/ftplicity backup
>>>>>> 1239 pts/0    S      0:00      |           \_ /bin/bash
>>>>>> /srv/www//release/ftplicity_1.4.2/ftplicity backup
>>>>>> 1240 pts/0    S      0:05      |               \_ /usr/bin/python
>>>>>> /srv/www/user/_apps/duplicity-0.5.10/bin/duplicity --verbosity 4
>>>>>> --encrypt-key XXXXXXX --sign-key B59ECD99 --gpg-options=--always-trust
>>>>>> --full-if-older-than 1M --volsize 50 --exclude-globbing-filelist
>>>>>> /srv/www/...
>>>>>> 1249 pts/0    SL     0:00      |                   \_ gpg --logger-fd 4
>>>>>> --passphrase-fd 8 --batch --no-tty --default-key XXXXXXXX --recipient
>>>>>> XXXXXXXX --no-secmem-warning --always-trust --encrypt --sign
>>>>>> 1265 pts/0    SL     0:00      |                   \_ gpg --status-fd 6
>>>>>> --passphrase-fd 11 --logger-fd 5 --batch --no-tty --default-key
>>>>>> XXXXXXXX
>>>>>> --no-secmem-warning --always-trust --decrypt
>>>>>> 1267 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>> 1272 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>> 1275 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>> 1278 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>> 1282 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>> 1286 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>> 1289 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>> 1292 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>> 1296 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>> 1299 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>> 1302 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>> 1305 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>> 1308 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>> 1311 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>> 1314 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>> 1317 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>> 1320 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>> 1324 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>> 1327 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>> 1330 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>> 1333 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>>> 1334 pts/1    Ss+    0:00      |                   \_ /usr/bin/ncftp -u
>>>>>> ******* backup.server.de
>>>>>> 1335 pts/0    R+     0:00      \_ ps -u user xf
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Duplicity-talk mailing list
>>>>>> address@hidden
>>>>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>>>>
>>>>>>     
>>>>>>           
>>>>>  
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>> _______________________________________________
>>>>> Duplicity-talk mailing list
>>>>> address@hidden
>>>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>>>   
>>>>>         
>>>> _______________________________________________
>>>> Duplicity-talk mailing list
>>>> address@hidden
>>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>>       
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>>     
>>
>>   
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>   
> 
> 
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>

signature.asc
Description: OpenPGP digital signature