Re: [Duplicity-talk] OSError: [Errno 24] Too many open files

[Edgar Soldin]

i really have to be fast (a window of 10s let's say) and it helps to
have a slow backup connection with lots of incrementals... but I may be
seeing ghosts here. Can it that it is perfectly normal for duplicity to
gpg decrypt something per each incremental but close all processes just
a bit later? .. just a guess
..ede
> I cannot reproduce this on a 32bit machine at all.  I'll be able to test
> on a 64bit machine sometime this week and, hopefully, find the problem.
>
> ...Ken
>
> address@hidden wrote:
>   
>> i seem to get closer to it ...
>> the more incrementals (regardless if file: or ftp: backend) the more
>> defunct gpg processes appear .. but dissappear again while the backup is
>> still running
>>
>> I did
>> initial backup of 3GB
>> touched a new test file to have a change & did a 2nd backup
>> touched a new test2 file to have a change & did a 3nd backup
>> ...
>>
>> and since the 3rd backup the defunct gpg processes keep appearing
>>
>> essentially I say: Take any more than 3 incrementals in size backup and
>> do a new incremental and you will see those zombie gpg processes ..
>>
>> while I am not sure that this is what happens on the ulimit problem I
>> thought I should report these observations ... ede
>>
>> PS:
>>
>> here a 'ps xf -u user' shortly after starting the sixth run .. the
>> number of gpg zombies seems to be number of sets (full + incr's) minus 2 ..
>>
>> 9122 pts/0    S+     0:00          \_ /bin/bash
>> /srv/www/user//release/ftplicity_1.4.2/ftplicity test backup
>> 19128 pts/0    S+     0:00              \_ /bin/bash
>> /srv/www/user//release/ftplicity_1.4.2/ftplicity backup
>> 19193 pts/0    S+     0:00                  \_ /bin/bash
>> /srv/www/user//release/ftplicity_1.4.2/ftplicity backup
>> 19194 pts/0    R+     0:03                      \_ /usr/bin/python
>> /srv/www/user/_apps/duplicity-0.5.10/bin/duplicity --verbosity 4
>> --encrypt-key 00000000--sign-key 00000000--gpg-options=--always-trust
>> --volsize 1 --exclude-globbing-filelist /srv/www/user//.ftplicity/test/excl
>> 19196 pts/0    SL+    0:00                          \_ gpg --logger-fd 4
>> --passphrase-fd 8 --batch --no-tty --default-key 00000000--recipient
>> 00000000--no-secmem-warning --always-trust --encrypt --sign
>> 19197 pts/0    SL+    0:00                          \_ gpg --status-fd 6
>> --passphrase-fd 11 --logger-fd 5 --batch --no-tty --default-key
>> 00000000--no-secmem-warning --always-trust --decrypt
>> 19199 pts/0    Z+     0:00                          \_ [gpg] <defunct>
>> 19200 pts/0    Z+     0:00                          \_ [gpg] <defunct>
>> 19201 pts/0    Z+     0:00                          \_ [gpg] <defunct>
>> 19202 pts/0    Z+     0:00                          \_ [gpg] <defunct>
>> 19203 pts/0    SL+    0:00                          \_ gpg --logger-fd
>> 23 --passphrase-fd 28 --batch --no-tty --default-key 00000000--recipient
>> 00000000--no-secmem-warning --always-trust --encrypt --sign
>>
>>
>>     
>>> hmm .. you are right these processes are zombie (they appear in top as
>>> such) .. later on the processes disappear, while duplicity is still
>>> running ..
>>> but I don't feel this is normal.. or is it?
>>> just to be more sure, I'll try a 1MB chunk full backup of my 3GB home
>>> dir .. let's see with how many zombies I'll end up ...
>>>
>>> interestingly my ulimit = unlimited ... seems to be default in the old
>>> SUSE 10.2, btw also on my debian 5.0 box both 32bit
>>> can't remember to ever have touched this setting
>>>
>>> ..ede
>>> -- 
>>>       
>>>> These are not open processes, they are zombie processes, so no real
>>>> resources taken.  That simplifies it a bit.  What normally causes this
>>>> is the failure of a parent process to properly retrieve its exit status,
>>>> but this is not the case or more people would have this problem.
>>>>
>>>> As far as I can tell, the only systems this is happening on are 64bit,
>>>> and not even all of those.  Some of my systems are 64bit and they don't
>>>> show this, so I'm wondering if this could be limited to a particular
>>>> version of GnuPG, or what.
>>>>
>>>> Edgar, from your note I could not tell, is this happening to you too?
>>>>
>>>> ...Ken
>>>>
>>>> address@hidden wrote:
>>>>  
>>>>         
>>>>> just my quick observation .. a simple incr backup opens and leaves over
>>>>> 20 gpg processes open. I don't think this is healthy and also can
>>>>> imagine that this multiplies if the volumes get smaller (mine are
>>>>> 50MB).
>>>>>
>>>>> regards ede
>>>>>
>>>>>
>>>>> address@hidden:~> ps -u user xf
>>>>>  PID TTY      STAT   TIME COMMAND
>>>>> 32731 ?        S      0:00 sshd: address@hidden/0
>>>>> 32732 pts/0    Ss     0:00  \_ -bash
>>>>> 1167 pts/0    S      0:00      \_ /bin/bash /usr/local/bin/ftplicity
>>>>> profile backup
>>>>> 1168 pts/0    S      0:00      |   \_ /bin/bash
>>>>> /srv/www//release/ftplicity_1.4.2/ftplicity profile backup
>>>>> 1174 pts/0    S      0:00      |       \_ /bin/bash
>>>>> /srv/www//release/ftplicity_1.4.2/ftplicity backup
>>>>> 1239 pts/0    S      0:00      |           \_ /bin/bash
>>>>> /srv/www//release/ftplicity_1.4.2/ftplicity backup
>>>>> 1240 pts/0    S      0:05      |               \_ /usr/bin/python
>>>>> /srv/www/user/_apps/duplicity-0.5.10/bin/duplicity --verbosity 4
>>>>> --encrypt-key XXXXXXX --sign-key B59ECD99 --gpg-options=--always-trust
>>>>> --full-if-older-than 1M --volsize 50 --exclude-globbing-filelist
>>>>> /srv/www/...
>>>>> 1249 pts/0    SL     0:00      |                   \_ gpg --logger-fd 4
>>>>> --passphrase-fd 8 --batch --no-tty --default-key XXXXXXXX --recipient
>>>>> XXXXXXXX --no-secmem-warning --always-trust --encrypt --sign
>>>>> 1265 pts/0    SL     0:00      |                   \_ gpg --status-fd 6
>>>>> --passphrase-fd 11 --logger-fd 5 --batch --no-tty --default-key
>>>>> XXXXXXXX
>>>>> --no-secmem-warning --always-trust --decrypt
>>>>> 1267 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>> 1272 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>> 1275 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>> 1278 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>> 1282 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>> 1286 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>> 1289 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>> 1292 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>> 1296 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>> 1299 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>> 1302 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>> 1305 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>> 1308 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>> 1311 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>> 1314 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>> 1317 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>> 1320 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>> 1324 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>> 1327 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>> 1330 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>> 1333 pts/0    Z      0:00      |                   \_ [gpg] <defunct>
>>>>> 1334 pts/1    Ss+    0:00      |                   \_ /usr/bin/ncftp -u
>>>>> ******* backup.server.de
>>>>> 1335 pts/0    R+     0:00      \_ ps -u user xf
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Duplicity-talk mailing list
>>>>> address@hidden
>>>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>>>
>>>>>     
>>>>>           
>>>>  
>>>> ------------------------------------------------------------------------
>>>>
>>>> _______________________________________________
>>>> Duplicity-talk mailing list
>>>> address@hidden
>>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>>   
>>>>         
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>       
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>
>>     
>
>
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>