Re: [Duplicity-talk] OSError: [Errno 24] Too many open files

[Edgar Soldin]

just curious, could you detect a difference between 32bit & 64bit systems?
how many incrementals did you need to hit the 1024 ulimit?

@yang: how many incrementals does your backup hold? just to doublecheck
kens findings.

..ede
> I have found the problem and its going to take me a bit to get it fixed.
>
> As it works duplicity creates iterators that traverse each of the sig
> files.  The iterators do not close until the process is complete and the
> files they are using are being held open.  The gpg process is held as
> well since we don't wait for it until we close the file.  The zombie
> processes were just a symptom of the real problem, too many open
> resources at one time.
>
> This hits directly the issue of too long an incremental chain.  Each
> increments sig file is going to have file descriptors open until the
> backup/incremental/restore completes.  The only workaround at this point
> is to increase the 'ulimit -n' parameter for any long increment chains
> and perform a full backup on a more frequent basis.
>
> I have some ideas about how to clean this up, but need to examine the
> code a bit closer before committing to a solution.
>
> ...Ken
>
> Kenneth Loafman wrote:
>   
>> As far as I can tell, its just lots of incrementals.  duplicity has to
>> open and process each signature file and, in some cases, the gpg process
>> is not being closed completely causing a zombie process.
>>
>> This "seems" to be worse on a 64bit system, but I have been able to
>> partially reproduce it on a 32bit.  The most zombie processes I've been
>> able to see are 5-7 and that's with about 60 incrementals.  Even then,
>> all of them go away quickly.
>>
>> ...Ken
>>
>> Jeff Singer wrote:
>>     
>>> I have a 64bit Ubuntu 9.04 system here that I'd be happy to attempt to
>>> reproduce it over the weekend. Is it basically a backup that involves
>>> lots of incrementals with many volumes that's causing this behavior?
>>>
>>> On Tue, May 12, 2009 at 1:51 PM, Ian Barton <address@hidden
>>> <mailto:address@hidden>> wrote:
>>>
>>>     I have got a couple of 64 bit Ubuntu 9.1 (Jaunty) computers. I can
>>>     try running a test on my system if you tell me what you would like
>>>     me to do.
>>>
>>>     FWIW my main computer for running is Debian Lenny 64bit and I have
>>>     never seen the problem on it.
>>>
>>>     Ian.
>>>
>>>
>>>         Thanks for the help!
>>>
>>>         I'm not sure at this point.  I built a 64bit Ubuntu 9.04 system
>>>         and it
>>>         seems to be working fine.  Debian may be different, but not much.
>>>
>>>         As to what's broke, it could even be at the system level, but my
>>>         bet is
>>>         on gpg.
>>>
>>>         ...Ken
>>>
>>>         address@hidden <mailto:address@hidden> wrote:
>>>
>>>             I am gonna install a 64bit debian in a virtual machine and gonna
>>>             doublecheck that ... but what do you think is buggy? python,
>>>             gpg ?
>>>
>>>             ..ede
>>>
>>>                 The only real change to GnuPGInterface is a shift right
>>>                 instead of a
>>>                 shift left when printing the error code after an
>>>                 exception.  He's not
>>>                 getting any exceptions, so he's not hitting the change.
>>>                  Other than
>>>                 that, its the same code from years ago.
>>>
>>>                 My best guess is still that its a 32-bit versus 64-bit
>>>                 issue.
>>>
>>>                 ...Ken
>>>
>>>                 Edgar Soldin wrote:
>>>                  
>>>
>>>                     true they close every time .. the only thing i see
>>>                     is that the amount of
>>>                     gpg processes grows with the amount of incrementals ...
>>>
>>>                     @yang: how many incrementals does your  backup hold?
>>>
>>>                     the other thing is, maybe gpg itself is buggy  ..
>>>
>>>                     @yang: could you try another gpg version?
>>>
>>>                     and the third thing
>>>
>>>                     @ken: is it really impossible for 0.5.17 to use an
>>>                     oldGnuPGInterface
>>>                     version, that might be laying around on the system?
>>>                     How could yang look
>>>                     for it?
>>>
>>>                     .. 2cents again :) ..ede
>>>                     -- 
>>>                      
>>>
>>>                         I tried this with tiny volume sizes of 1M over a
>>>                         2GB backup (2K
>>>                         volumes)
>>>                         and did a count of the gpg processes (live,
>>>                         defunct) every 2 seconds
>>>                         and
>>>                         never had more than 5 (3 live, 2 defunct).  This
>>>                         was on a very fast
>>>                         link
>>>                         between local VMs.  Tried both backup and
>>>                         restore so it would test both
>>>                         directions.  By the end, all were closed.
>>>
>>>                         It's possible to have some defunct while the
>>>                         task scheduler is
>>>                         updating,
>>>                         but Yang is showing hundreds that are holding
>>>                         files open.  That's the
>>>                         confusing part.
>>>
>>>                         ...Ken
>>>
>>>                         Edgar Soldin wrote:
>>>                              
>>>
>>>                             i really have to be fast (a window of 10s
>>>                             let's say) and it helps to
>>>                             have a slow backup connection with lots of
>>>                             incrementals... but I
>>>                             may be
>>>                             seeing ghosts here. Can it that it is
>>>                             perfectly normal for
>>>                             duplicity to
>>>                             gpg decrypt something per each incremental
>>>                             but close all processes
>>>                             just
>>>                             a bit later? .. just a guess
>>>                             ..ede
>>>                                      
>>>
>>>                                 I cannot reproduce this on a 32bit
>>>                                 machine at all.  I'll be able
>>>                                 to test
>>>                                 on a 64bit machine sometime this week
>>>                                 and, hopefully, find the
>>>                                 problem.
>>>
>>>                                 ...Ken
>>>
>>>                                 address@hidden
>>>                                 <mailto:address@hidden> wrote:
>>>                                                
>>>
>>>                                     i seem to get closer to it ...
>>>                                     the more incrementals (regardless if
>>>                                     file: or ftp: backend) the more
>>>                                     defunct gpg processes appear .. but
>>>                                     dissappear again while the
>>>                                     backup is
>>>                                     still running
>>>
>>>                                     I did
>>>                                     initial backup of 3GB
>>>                                     touched a new test file to have a
>>>                                     change & did a 2nd backup
>>>                                     touched a new test2 file to have a
>>>                                     change & did a 3nd backup
>>>                                     ...
>>>
>>>                                     and since the 3rd backup the defunct
>>>                                     gpg processes keep appearing
>>>
>>>                                     essentially I say: Take any more
>>>                                     than 3 incrementals in size
>>>                                     backup and
>>>                                     do a new incremental and you will
>>>                                     see those zombie gpg processes ..
>>>
>>>                                     while I am not sure that this is
>>>                                     what happens on the ulimit
>>>                                     problem I
>>>                                     thought I should report these
>>>                                     observations ... ede
>>>
>>>                                     PS:
>>>
>>>                                     here a 'ps xf -u user' shortly after
>>>                                     starting the sixth run .. the
>>>                                     number of gpg zombies seems to be
>>>                                     number of sets (full + incr's)
>>>                                     minus 2 ..
>>>
>>>                                     9122 pts/0    S+     0:00        
>>>                                      \_ /bin/bash
>>>                                     
>>> /srv/www/user//release/ftplicity_1.4.2/ftplicity
>>>                                     test backup
>>>                                     19128 pts/0    S+     0:00          
>>>                                        \_ /bin/bash
>>>                                     
>>> /srv/www/user//release/ftplicity_1.4.2/ftplicity
>>>                                     backup
>>>                                     19193 pts/0    S+     0:00          
>>>                                            \_ /bin/bash
>>>                                     
>>> /srv/www/user//release/ftplicity_1.4.2/ftplicity
>>>                                     backup
>>>                                     19194 pts/0    R+     0:03          
>>>                                                \_ /usr/bin/python
>>>                                     
>>> /srv/www/user/_apps/duplicity-0.5.10/bin/duplicity
>>>                                     --verbosity 4
>>>                                     --encrypt-key 00000000--sign-key
>>>                                     00000000--gpg-options=--always-trust
>>>                                     --volsize 1 --exclude-globbing-filelist
>>>                                     /srv/www/user//.ftplicity/test/excl
>>>                                     19196 pts/0    SL+    0:00          
>>>                                                    \_ gpg
>>>                                     --logger-fd 4
>>>                                     --passphrase-fd 8 --batch --no-tty
>>>                                     --default-key 00000000--recipient
>>>                                     00000000--no-secmem-warning
>>>                                     --always-trust --encrypt --sign
>>>                                     19197 pts/0    SL+    0:00          
>>>                                                    \_ gpg
>>>                                     --status-fd 6
>>>                                     --passphrase-fd 11 --logger-fd 5
>>>                                     --batch --no-tty --default-key
>>>                                     00000000--no-secmem-warning
>>>                                     --always-trust --decrypt
>>>                                     19199 pts/0    Z+     0:00          
>>>                                                    \_ [gpg]
>>>                                     <defunct>
>>>                                     19200 pts/0    Z+     0:00          
>>>                                                    \_ [gpg]
>>>                                     <defunct>
>>>                                     19201 pts/0    Z+     0:00          
>>>                                                    \_ [gpg]
>>>                                     <defunct>
>>>                                     19202 pts/0    Z+     0:00          
>>>                                                    \_ [gpg]
>>>                                     <defunct>
>>>                                     19203 pts/0    SL+    0:00          
>>>                                                    \_ gpg
>>>                                     --logger-fd
>>>                                     23 --passphrase-fd 28 --batch
>>>                                     --no-tty --default-key
>>>                                     00000000--recipient
>>>                                     00000000--no-secmem-warning
>>>                                     --always-trust --encrypt --sign
>>>
>>>
>>>                                                          
>>>
>>>                                         hmm .. you are right these
>>>                                         processes are zombie (they appear in
>>>                                         top as
>>>                                         such) .. later on the processes
>>>                                         disappear, while duplicity is still
>>>                                         running ..
>>>                                         but I don't feel this is
>>>                                         normal.. or is it?
>>>                                         just to be more sure, I'll try a
>>>                                         1MB chunk full backup of my 3GB
>>>                                         home
>>>                                         dir .. let's see with how many
>>>                                         zombies I'll end up ...
>>>
>>>                                         interestingly my ulimit =
>>>                                         unlimited ... seems to be default in
>>>                                         the old
>>>                                         SUSE 10.2, btw also on my debian
>>>                                         5.0 box both 32bit
>>>                                         can't remember to ever have
>>>                                         touched this setting
>>>
>>>                                         ..ede
>>>                                         -- 
>>>                                                                    
>>>
>>>                                             These are not open
>>>                                             processes, they are zombie
>>>                                             processes, so no
>>>                                             real
>>>                                             resources taken.  That
>>>                                             simplifies it a bit.  What
>>>                                             normally
>>>                                             causes this
>>>                                             is the failure of a parent
>>>                                             process to properly retrieve its
>>>                                             exit status,
>>>                                             but this is not the case or
>>>                                             more people would have this
>>>                                             problem.
>>>
>>>                                             As far as I can tell, the
>>>                                             only systems this is
>>>                                             happening on are
>>>                                             64bit,
>>>                                             and not even all of those.
>>>                                              Some of my systems are
>>>                                             64bit and
>>>                                             they don't
>>>                                             show this, so I'm wondering
>>>                                             if this could be limited to a
>>>                                             particular
>>>                                             version of GnuPG, or what.
>>>
>>>                                             Edgar, from your note I
>>>                                             could not tell, is this
>>>                                             happening to
>>>                                             you too?
>>>
>>>                                             ...Ken
>>>
>>>                                             address@hidden
>>>                                             <mailto:address@hidden>
>>>                                             wrote:
>>>                                                                        
>>>                                                    
>>>
>>>                                                 just my quick
>>>                                                 observation .. a simple
>>>                                                 incr backup opens and
>>>                                                 leaves over
>>>                                                 20 gpg processes open. I
>>>                                                 don't think this is
>>>                                                 healthy and also can
>>>                                                 imagine that this
>>>                                                 multiplies if the
>>>                                                 volumes get smaller
>>>                                                 (mine are
>>>                                                 50MB).
>>>
>>>                                                 regards ede
>>>
>>>
>>>                                                 address@hidden:~> ps -u 
>>> user xf
>>>                                                  PID TTY      STAT  
>>>                                                 TIME COMMAND
>>>                                                 32731 ?        S    
>>>                                                  0:00 sshd: address@hidden/0
>>>                                                 32732 pts/0    Ss    
>>>                                                 0:00  \_ -bash
>>>                                                 1167 pts/0    S    
>>>                                                  0:00      \_ /bin/bash
>>>                                                 /usr/local/bin/ftplicity
>>>                                                 profile backup
>>>                                                 1168 pts/0    S    
>>>                                                  0:00      |   \_ /bin/bash
>>>                                                 
>>> /srv/www//release/ftplicity_1.4.2/ftplicity
>>>                                                 profile backup
>>>                                                 1174 pts/0    S    
>>>                                                  0:00      |       \_
>>>                                                 /bin/bash
>>>                                                 
>>> /srv/www//release/ftplicity_1.4.2/ftplicity
>>>                                                 backup
>>>                                                 1239 pts/0    S    
>>>                                                  0:00      |          
>>>                                                 \_ /bin/bash
>>>                                                 
>>> /srv/www//release/ftplicity_1.4.2/ftplicity
>>>                                                 backup
>>>                                                 1240 pts/0    S    
>>>                                                  0:05      |            
>>>                                                   \_ /usr/bin/python
>>>                                                 
>>> /srv/www/user/_apps/duplicity-0.5.10/bin/duplicity
>>>                                                 --verbosity 4
>>>                                                 --encrypt-key XXXXXXX
>>>                                                 --sign-key B59ECD99
>>>                                                 --gpg-options=--always-trust
>>>                                                 --full-if-older-than 1M
>>>                                                 --volsize 50
>>>                                                 --exclude-globbing-filelist
>>>                                                 /srv/www/...
>>>                                                 1249 pts/0    SL    
>>>                                                 0:00      |            
>>>                                                       \_ gpg
>>>                                                 --logger-fd 4
>>>                                                 --passphrase-fd 8
>>>                                                 --batch --no-tty
>>>                                                 --default-key XXXXXXXX
>>>                                                 --recipient
>>>                                                 XXXXXXXX
>>>                                                 --no-secmem-warning
>>>                                                 --always-trust --encrypt
>>>                                                 --sign
>>>                                                 1265 pts/0    SL    
>>>                                                 0:00      |            
>>>                                                       \_ gpg
>>>                                                 --status-fd 6
>>>                                                 --passphrase-fd 11
>>>                                                 --logger-fd 5 --batch
>>>                                                 --no-tty --default-key
>>>                                                 XXXXXXXX
>>>                                                 --no-secmem-warning
>>>                                                 --always-trust --decrypt
>>>                                                 1267 pts/0    Z    
>>>                                                  0:00      |            
>>>                                                       \_ [gpg]
>>>                                                 <defunct>
>>>                                                 1272 pts/0    Z    
>>>                                                  0:00      |            
>>>                                                       \_ [gpg]
>>>                                                 <defunct>
>>>                                                 1275 pts/0    Z    
>>>                                                  0:00      |            
>>>                                                       \_ [gpg]
>>>                                                 <defunct>
>>>                                                 1278 pts/0    Z    
>>>                                                  0:00      |            
>>>                                                       \_ [gpg]
>>>                                                 <defunct>
>>>                                                 1282 pts/0    Z    
>>>                                                  0:00      |            
>>>                                                       \_ [gpg]
>>>                                                 <defunct>
>>>                                                 1286 pts/0    Z    
>>>                                                  0:00      |            
>>>                                                       \_ [gpg]
>>>                                                 <defunct>
>>>                                                 1289 pts/0    Z    
>>>                                                  0:00      |            
>>>                                                       \_ [gpg]
>>>                                                 <defunct>
>>>                                                 1292 pts/0    Z    
>>>                                                  0:00      |            
>>>                                                       \_ [gpg]
>>>                                                 <defunct>
>>>                                                 1296 pts/0    Z    
>>>                                                  0:00      |            
>>>                                                       \_ [gpg]
>>>                                                 <defunct>
>>>                                                 1299 pts/0    Z    
>>>                                                  0:00      |            
>>>                                                       \_ [gpg]
>>>                                                 <defunct>
>>>                                                 1302 pts/0    Z    
>>>                                                  0:00      |            
>>>                                                       \_ [gpg]
>>>                                                 <defunct>
>>>                                                 1305 pts/0    Z    
>>>                                                  0:00      |            
>>>                                                       \_ [gpg]
>>>                                                 <defunct>
>>>                                                 1308 pts/0    Z    
>>>                                                  0:00      |            
>>>                                                       \_ [gpg]
>>>                                                 <defunct>
>>>                                                 1311 pts/0    Z    
>>>                                                  0:00      |            
>>>                                                       \_ [gpg]
>>>                                                 <defunct>
>>>                                                 1314 pts/0    Z    
>>>                                                  0:00      |            
>>>                                                       \_ [gpg]
>>>                                                 <defunct>
>>>                                                 1317 pts/0    Z    
>>>                                                  0:00      |            
>>>                                                       \_ [gpg]
>>>                                                 <defunct>
>>>                                                 1320 pts/0    Z    
>>>                                                  0:00      |            
>>>                                                       \_ [gpg]
>>>                                                 <defunct>
>>>                                                 1324 pts/0    Z    
>>>                                                  0:00      |            
>>>                                                       \_ [gpg]
>>>                                                 <defunct>
>>>                                                 1327 pts/0    Z    
>>>                                                  0:00      |            
>>>                                                       \_ [gpg]
>>>                                                 <defunct>
>>>                                                 1330 pts/0    Z    
>>>                                                  0:00      |            
>>>                                                       \_ [gpg]
>>>                                                 <defunct>
>>>                                                 1333 pts/0    Z    
>>>                                                  0:00      |            
>>>                                                       \_ [gpg]
>>>                                                 <defunct>
>>>                                                 1334 pts/1    Ss+  
>>>                                                  0:00      |            
>>>                                                       \_
>>>                                                 /usr/bin/ncftp -u
>>>                                                 ******* backup.server.de
>>>                                                 <http://backup.server.de>
>>>                                                 1335 pts/0    R+    
>>>                                                 0:00      \_ ps -u user xf
>>>
>>>
>>>
>>>
>>>                                                 
>>> _______________________________________________
>>>                                                 Duplicity-talk mailing list
>>>                                                 address@hidden
>>>                                                 <mailto:address@hidden>
>>>                                                 
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>>                                                                        
>>>                                                                      
>>>
>>>                                              
>>> ------------------------------------------------------------------------
>>>
>>>
>>>                                             
>>> _______________________________________________
>>>                                             Duplicity-talk mailing list
>>>                                             address@hidden
>>>                                             <mailto:address@hidden>
>>>                                             
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>                                                                        
>>>                                                      
>>>
>>>                                         
>>> _______________________________________________
>>>                                         Duplicity-talk mailing list
>>>                                         address@hidden
>>>                                         <mailto:address@hidden>
>>>                                         
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>                                                                      
>>>
>>>                                     
>>> _______________________________________________
>>>                                     Duplicity-talk mailing list
>>>                                     address@hidden
>>>                                     <mailto:address@hidden>
>>>                                     
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>>                                                            
>>>
>>>                                  
>>> ------------------------------------------------------------------------
>>>
>>>
>>>                                 
>>> _______________________________________________
>>>                                 Duplicity-talk mailing list
>>>                                 address@hidden
>>>                                 <mailto:address@hidden>
>>>                                 
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>                                                  
>>>
>>>                             _______________________________________________
>>>                             Duplicity-talk mailing list
>>>                             address@hidden
>>>                             <mailto:address@hidden>
>>>                             
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>>                                        
>>>
>>>                          
>>> ------------------------------------------------------------------------
>>>
>>>
>>>                         _______________________________________________
>>>                         Duplicity-talk mailing list
>>>                         address@hidden
>>>                         <mailto:address@hidden>
>>>                         
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>                                
>>>
>>>                     _______________________________________________
>>>                     Duplicity-talk mailing list
>>>                     address@hidden
>>>                     <mailto:address@hidden>
>>>                     http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>>                        
>>>
>>>
>>>                  
>>> ------------------------------------------------------------------------
>>>
>>>                 _______________________________________________
>>>                 Duplicity-talk mailing list
>>>                 address@hidden <mailto:address@hidden>
>>>                 http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>                  
>>>
>>>
>>>
>>>             _______________________________________________
>>>             Duplicity-talk mailing list
>>>             address@hidden <mailto:address@hidden>
>>>             http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>>
>>>
>>>
>>>         
>>> ------------------------------------------------------------------------
>>>
>>>
>>>         _______________________________________________
>>>         Duplicity-talk mailing list
>>>         address@hidden <mailto:address@hidden>
>>>         http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>>
>>>
>>>
>>>     _______________________________________________
>>>     Duplicity-talk mailing list
>>>     address@hidden <mailto:address@hidden>
>>>     http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>       
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>     
>
>
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>