Re: [Duplicity-talk] OSError: [Errno 24] Too many open files

[Kenneth Loafman]

As far as I can tell, its just lots of incrementals.  duplicity has to
open and process each signature file and, in some cases, the gpg process
is not being closed completely causing a zombie process.

This "seems" to be worse on a 64bit system, but I have been able to
partially reproduce it on a 32bit.  The most zombie processes I've been
able to see are 5-7 and that's with about 60 incrementals.  Even then,
all of them go away quickly.

...Ken

Jeff Singer wrote:
> I have a 64bit Ubuntu 9.04 system here that I'd be happy to attempt to
> reproduce it over the weekend. Is it basically a backup that involves
> lots of incrementals with many volumes that's causing this behavior?
> 
> On Tue, May 12, 2009 at 1:51 PM, Ian Barton <address@hidden
> <mailto:address@hidden>> wrote:
> 
>     I have got a couple of 64 bit Ubuntu 9.1 (Jaunty) computers. I can
>     try running a test on my system if you tell me what you would like
>     me to do.
> 
>     FWIW my main computer for running is Debian Lenny 64bit and I have
>     never seen the problem on it.
> 
>     Ian.
> 
> 
>         Thanks for the help!
> 
>         I'm not sure at this point.  I built a 64bit Ubuntu 9.04 system
>         and it
>         seems to be working fine.  Debian may be different, but not much.
> 
>         As to what's broke, it could even be at the system level, but my
>         bet is
>         on gpg.
> 
>         ...Ken
> 
>         address@hidden <mailto:address@hidden> wrote:
> 
>             I am gonna install a 64bit debian in a virtual machine and gonna
>             doublecheck that ... but what do you think is buggy? python,
>             gpg ?
> 
>             ..ede
> 
>                 The only real change to GnuPGInterface is a shift right
>                 instead of a
>                 shift left when printing the error code after an
>                 exception.  He's not
>                 getting any exceptions, so he's not hitting the change.
>                  Other than
>                 that, its the same code from years ago.
> 
>                 My best guess is still that its a 32-bit versus 64-bit
>                 issue.
> 
>                 ...Ken
> 
>                 Edgar Soldin wrote:
>                  
> 
>                     true they close every time .. the only thing i see
>                     is that the amount of
>                     gpg processes grows with the amount of incrementals ...
> 
>                     @yang: how many incrementals does your  backup hold?
> 
>                     the other thing is, maybe gpg itself is buggy  ..
> 
>                     @yang: could you try another gpg version?
> 
>                     and the third thing
> 
>                     @ken: is it really impossible for 0.5.17 to use an
>                     oldGnuPGInterface
>                     version, that might be laying around on the system?
>                     How could yang look
>                     for it?
> 
>                     .. 2cents again :) ..ede
>                     -- 
>                      
> 
>                         I tried this with tiny volume sizes of 1M over a
>                         2GB backup (2K
>                         volumes)
>                         and did a count of the gpg processes (live,
>                         defunct) every 2 seconds
>                         and
>                         never had more than 5 (3 live, 2 defunct).  This
>                         was on a very fast
>                         link
>                         between local VMs.  Tried both backup and
>                         restore so it would test both
>                         directions.  By the end, all were closed.
> 
>                         It's possible to have some defunct while the
>                         task scheduler is
>                         updating,
>                         but Yang is showing hundreds that are holding
>                         files open.  That's the
>                         confusing part.
> 
>                         ...Ken
> 
>                         Edgar Soldin wrote:
>                              
> 
>                             i really have to be fast (a window of 10s
>                             let's say) and it helps to
>                             have a slow backup connection with lots of
>                             incrementals... but I
>                             may be
>                             seeing ghosts here. Can it that it is
>                             perfectly normal for
>                             duplicity to
>                             gpg decrypt something per each incremental
>                             but close all processes
>                             just
>                             a bit later? .. just a guess
>                             ..ede
>                                      
> 
>                                 I cannot reproduce this on a 32bit
>                                 machine at all.  I'll be able
>                                 to test
>                                 on a 64bit machine sometime this week
>                                 and, hopefully, find the
>                                 problem.
> 
>                                 ...Ken
> 
>                                 address@hidden
>                                 <mailto:address@hidden> wrote:
>                                                
> 
>                                     i seem to get closer to it ...
>                                     the more incrementals (regardless if
>                                     file: or ftp: backend) the more
>                                     defunct gpg processes appear .. but
>                                     dissappear again while the
>                                     backup is
>                                     still running
> 
>                                     I did
>                                     initial backup of 3GB
>                                     touched a new test file to have a
>                                     change & did a 2nd backup
>                                     touched a new test2 file to have a
>                                     change & did a 3nd backup
>                                     ...
> 
>                                     and since the 3rd backup the defunct
>                                     gpg processes keep appearing
> 
>                                     essentially I say: Take any more
>                                     than 3 incrementals in size
>                                     backup and
>                                     do a new incremental and you will
>                                     see those zombie gpg processes ..
> 
>                                     while I am not sure that this is
>                                     what happens on the ulimit
>                                     problem I
>                                     thought I should report these
>                                     observations ... ede
> 
>                                     PS:
> 
>                                     here a 'ps xf -u user' shortly after
>                                     starting the sixth run .. the
>                                     number of gpg zombies seems to be
>                                     number of sets (full + incr's)
>                                     minus 2 ..
> 
>                                     9122 pts/0    S+     0:00        
>                                      \_ /bin/bash
>                                     
> /srv/www/user//release/ftplicity_1.4.2/ftplicity
>                                     test backup
>                                     19128 pts/0    S+     0:00          
>                                        \_ /bin/bash
>                                     
> /srv/www/user//release/ftplicity_1.4.2/ftplicity
>                                     backup
>                                     19193 pts/0    S+     0:00          
>                                            \_ /bin/bash
>                                     
> /srv/www/user//release/ftplicity_1.4.2/ftplicity
>                                     backup
>                                     19194 pts/0    R+     0:03          
>                                                \_ /usr/bin/python
>                                     
> /srv/www/user/_apps/duplicity-0.5.10/bin/duplicity
>                                     --verbosity 4
>                                     --encrypt-key 00000000--sign-key
>                                     00000000--gpg-options=--always-trust
>                                     --volsize 1 --exclude-globbing-filelist
>                                     /srv/www/user//.ftplicity/test/excl
>                                     19196 pts/0    SL+    0:00          
>                                                    \_ gpg
>                                     --logger-fd 4
>                                     --passphrase-fd 8 --batch --no-tty
>                                     --default-key 00000000--recipient
>                                     00000000--no-secmem-warning
>                                     --always-trust --encrypt --sign
>                                     19197 pts/0    SL+    0:00          
>                                                    \_ gpg
>                                     --status-fd 6
>                                     --passphrase-fd 11 --logger-fd 5
>                                     --batch --no-tty --default-key
>                                     00000000--no-secmem-warning
>                                     --always-trust --decrypt
>                                     19199 pts/0    Z+     0:00          
>                                                    \_ [gpg]
>                                     <defunct>
>                                     19200 pts/0    Z+     0:00          
>                                                    \_ [gpg]
>                                     <defunct>
>                                     19201 pts/0    Z+     0:00          
>                                                    \_ [gpg]
>                                     <defunct>
>                                     19202 pts/0    Z+     0:00          
>                                                    \_ [gpg]
>                                     <defunct>
>                                     19203 pts/0    SL+    0:00          
>                                                    \_ gpg
>                                     --logger-fd
>                                     23 --passphrase-fd 28 --batch
>                                     --no-tty --default-key
>                                     00000000--recipient
>                                     00000000--no-secmem-warning
>                                     --always-trust --encrypt --sign
> 
> 
>                                                          
> 
>                                         hmm .. you are right these
>                                         processes are zombie (they appear in
>                                         top as
>                                         such) .. later on the processes
>                                         disappear, while duplicity is still
>                                         running ..
>                                         but I don't feel this is
>                                         normal.. or is it?
>                                         just to be more sure, I'll try a
>                                         1MB chunk full backup of my 3GB
>                                         home
>                                         dir .. let's see with how many
>                                         zombies I'll end up ...
> 
>                                         interestingly my ulimit =
>                                         unlimited ... seems to be default in
>                                         the old
>                                         SUSE 10.2, btw also on my debian
>                                         5.0 box both 32bit
>                                         can't remember to ever have
>                                         touched this setting
> 
>                                         ..ede
>                                         -- 
>                                                                    
> 
>                                             These are not open
>                                             processes, they are zombie
>                                             processes, so no
>                                             real
>                                             resources taken.  That
>                                             simplifies it a bit.  What
>                                             normally
>                                             causes this
>                                             is the failure of a parent
>                                             process to properly retrieve its
>                                             exit status,
>                                             but this is not the case or
>                                             more people would have this
>                                             problem.
> 
>                                             As far as I can tell, the
>                                             only systems this is
>                                             happening on are
>                                             64bit,
>                                             and not even all of those.
>                                              Some of my systems are
>                                             64bit and
>                                             they don't
>                                             show this, so I'm wondering
>                                             if this could be limited to a
>                                             particular
>                                             version of GnuPG, or what.
> 
>                                             Edgar, from your note I
>                                             could not tell, is this
>                                             happening to
>                                             you too?
> 
>                                             ...Ken
> 
>                                             address@hidden
>                                             <mailto:address@hidden>
>                                             wrote:
>                                                                        
>                                                    
> 
>                                                 just my quick
>                                                 observation .. a simple
>                                                 incr backup opens and
>                                                 leaves over
>                                                 20 gpg processes open. I
>                                                 don't think this is
>                                                 healthy and also can
>                                                 imagine that this
>                                                 multiplies if the
>                                                 volumes get smaller
>                                                 (mine are
>                                                 50MB).
> 
>                                                 regards ede
> 
> 
>                                                 address@hidden:~> ps -u user 
> xf
>                                                  PID TTY      STAT  
>                                                 TIME COMMAND
>                                                 32731 ?        S    
>                                                  0:00 sshd: address@hidden/0
>                                                 32732 pts/0    Ss    
>                                                 0:00  \_ -bash
>                                                 1167 pts/0    S    
>                                                  0:00      \_ /bin/bash
>                                                 /usr/local/bin/ftplicity
>                                                 profile backup
>                                                 1168 pts/0    S    
>                                                  0:00      |   \_ /bin/bash
>                                                 
> /srv/www//release/ftplicity_1.4.2/ftplicity
>                                                 profile backup
>                                                 1174 pts/0    S    
>                                                  0:00      |       \_
>                                                 /bin/bash
>                                                 
> /srv/www//release/ftplicity_1.4.2/ftplicity
>                                                 backup
>                                                 1239 pts/0    S    
>                                                  0:00      |          
>                                                 \_ /bin/bash
>                                                 
> /srv/www//release/ftplicity_1.4.2/ftplicity
>                                                 backup
>                                                 1240 pts/0    S    
>                                                  0:05      |            
>                                                   \_ /usr/bin/python
>                                                 
> /srv/www/user/_apps/duplicity-0.5.10/bin/duplicity
>                                                 --verbosity 4
>                                                 --encrypt-key XXXXXXX
>                                                 --sign-key B59ECD99
>                                                 --gpg-options=--always-trust
>                                                 --full-if-older-than 1M
>                                                 --volsize 50
>                                                 --exclude-globbing-filelist
>                                                 /srv/www/...
>                                                 1249 pts/0    SL    
>                                                 0:00      |            
>                                                       \_ gpg
>                                                 --logger-fd 4
>                                                 --passphrase-fd 8
>                                                 --batch --no-tty
>                                                 --default-key XXXXXXXX
>                                                 --recipient
>                                                 XXXXXXXX
>                                                 --no-secmem-warning
>                                                 --always-trust --encrypt
>                                                 --sign
>                                                 1265 pts/0    SL    
>                                                 0:00      |            
>                                                       \_ gpg
>                                                 --status-fd 6
>                                                 --passphrase-fd 11
>                                                 --logger-fd 5 --batch
>                                                 --no-tty --default-key
>                                                 XXXXXXXX
>                                                 --no-secmem-warning
>                                                 --always-trust --decrypt
>                                                 1267 pts/0    Z    
>                                                  0:00      |            
>                                                       \_ [gpg]
>                                                 <defunct>
>                                                 1272 pts/0    Z    
>                                                  0:00      |            
>                                                       \_ [gpg]
>                                                 <defunct>
>                                                 1275 pts/0    Z    
>                                                  0:00      |            
>                                                       \_ [gpg]
>                                                 <defunct>
>                                                 1278 pts/0    Z    
>                                                  0:00      |            
>                                                       \_ [gpg]
>                                                 <defunct>
>                                                 1282 pts/0    Z    
>                                                  0:00      |            
>                                                       \_ [gpg]
>                                                 <defunct>
>                                                 1286 pts/0    Z    
>                                                  0:00      |            
>                                                       \_ [gpg]
>                                                 <defunct>
>                                                 1289 pts/0    Z    
>                                                  0:00      |            
>                                                       \_ [gpg]
>                                                 <defunct>
>                                                 1292 pts/0    Z    
>                                                  0:00      |            
>                                                       \_ [gpg]
>                                                 <defunct>
>                                                 1296 pts/0    Z    
>                                                  0:00      |            
>                                                       \_ [gpg]
>                                                 <defunct>
>                                                 1299 pts/0    Z    
>                                                  0:00      |            
>                                                       \_ [gpg]
>                                                 <defunct>
>                                                 1302 pts/0    Z    
>                                                  0:00      |            
>                                                       \_ [gpg]
>                                                 <defunct>
>                                                 1305 pts/0    Z    
>                                                  0:00      |            
>                                                       \_ [gpg]
>                                                 <defunct>
>                                                 1308 pts/0    Z    
>                                                  0:00      |            
>                                                       \_ [gpg]
>                                                 <defunct>
>                                                 1311 pts/0    Z    
>                                                  0:00      |            
>                                                       \_ [gpg]
>                                                 <defunct>
>                                                 1314 pts/0    Z    
>                                                  0:00      |            
>                                                       \_ [gpg]
>                                                 <defunct>
>                                                 1317 pts/0    Z    
>                                                  0:00      |            
>                                                       \_ [gpg]
>                                                 <defunct>
>                                                 1320 pts/0    Z    
>                                                  0:00      |            
>                                                       \_ [gpg]
>                                                 <defunct>
>                                                 1324 pts/0    Z    
>                                                  0:00      |            
>                                                       \_ [gpg]
>                                                 <defunct>
>                                                 1327 pts/0    Z    
>                                                  0:00      |            
>                                                       \_ [gpg]
>                                                 <defunct>
>                                                 1330 pts/0    Z    
>                                                  0:00      |            
>                                                       \_ [gpg]
>                                                 <defunct>
>                                                 1333 pts/0    Z    
>                                                  0:00      |            
>                                                       \_ [gpg]
>                                                 <defunct>
>                                                 1334 pts/1    Ss+  
>                                                  0:00      |            
>                                                       \_
>                                                 /usr/bin/ncftp -u
>                                                 ******* backup.server.de
>                                                 <http://backup.server.de>
>                                                 1335 pts/0    R+    
>                                                 0:00      \_ ps -u user xf
> 
> 
> 
> 
>                                                 
> _______________________________________________
>                                                 Duplicity-talk mailing list
>                                                 address@hidden
>                                                 <mailto:address@hidden>
>                                                 
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
>                                                                        
>                                                                      
> 
>                                              
> ------------------------------------------------------------------------
> 
> 
>                                             
> _______________________________________________
>                                             Duplicity-talk mailing list
>                                             address@hidden
>                                             <mailto:address@hidden>
>                                             
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>                                                                        
>                                                      
> 
>                                         
> _______________________________________________
>                                         Duplicity-talk mailing list
>                                         address@hidden
>                                         <mailto:address@hidden>
>                                         
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>                                                                      
> 
>                                     
> _______________________________________________
>                                     Duplicity-talk mailing list
>                                     address@hidden
>                                     <mailto:address@hidden>
>                                     
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
>                                                            
> 
>                                  
> ------------------------------------------------------------------------
> 
> 
>                                 
> _______________________________________________
>                                 Duplicity-talk mailing list
>                                 address@hidden
>                                 <mailto:address@hidden>
>                                 
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>                                                  
> 
>                             _______________________________________________
>                             Duplicity-talk mailing list
>                             address@hidden
>                             <mailto:address@hidden>
>                             
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
>                                        
> 
>                          
> ------------------------------------------------------------------------
> 
> 
>                         _______________________________________________
>                         Duplicity-talk mailing list
>                         address@hidden
>                         <mailto:address@hidden>
>                         
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>                                
> 
>                     _______________________________________________
>                     Duplicity-talk mailing list
>                     address@hidden
>                     <mailto:address@hidden>
>                     http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
>                        
> 
> 
>                  
> ------------------------------------------------------------------------
> 
>                 _______________________________________________
>                 Duplicity-talk mailing list
>                 address@hidden <mailto:address@hidden>
>                 http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>                  
> 
> 
> 
>             _______________________________________________
>             Duplicity-talk mailing list
>             address@hidden <mailto:address@hidden>
>             http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
> 
> 
> 
>         
> ------------------------------------------------------------------------
> 
> 
>         _______________________________________________
>         Duplicity-talk mailing list
>         address@hidden <mailto:address@hidden>
>         http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
> 
> 
> 
>     _______________________________________________
>     Duplicity-talk mailing list
>     address@hidden <mailto:address@hidden>
>     http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk

signature.asc
Description: OpenPGP digital signature