[Duplicity-talk] Duplicity - Resource Unavailable Errors

[Colin Ryan]

I should know the answer to this but can't seem to confirm. Clearly such 
errors are generally flimit/ulimit issues or RAM.

But I'm testing restore a not huge ( ~ 8 GB archive that is only 1 full 
and 45 days of incremental's) and get the following from the following 
command line.

/duplicity restore  -v9 --encrypt-key=xxxxxxx --sign-key=xxxxxx 
--gpg-options='--default-key=xxxxxxx' 
--archive-dir="/opt/aa/service/nas/restores" 
--tempdir="/opt/aa/service/nas/restores" --ignore-errors 
s3+http://URL/<backup set name> "/opt/aa/service/nas/restores/office"

I'm using the -ignore-errors as I'm unpacking as a non-root user other 
than the one that did the backups so it was throwing "no permission" 
errors on what I expect was the chmod/chown stages ( I say this because 
the files themselves were extracted)...Anyhow...

Version is 6.0.5 hand rolled in Debian Lenny.

When it ceases the system is unable to fork any other processes until I 
kill duplicity.

A already running top in another terminal shows that duplicity at this 
point is ~ 200 MB virtual and 11 MB resident but that I have ~ 200M RAM 
free (this is a fairly small footprint system).

I believe it's out of memory for forking but is it normal.

<code>
BEGNj3-tempdir/mktemp-QYdP4e-85
DEBUG:duplicity:Removing still remembered temporary file 
/opt/aa/service/nas/restores/duplicity-BEGNj3-tempdir/mktemp-QYdP4e-85
Removing still remembered temporary file 
/opt/aa/service/nas/restores/duplicity-BEGNj3-tempdir/mktemp-aDi69N-61
DEBUG:duplicity:Removing still remembered temporary file 
/opt/aa/service/nas/restores/duplicity-BEGNj3-tempdir/mktemp-aDi69N-61
Removing still remembered temporary file 
/opt/aa/service/nas/restores/duplicity-BEGNj3-tempdir/mktemp-2YCFpS-81
DEBUG:duplicity:Removing still remembered temporary file 
/opt/aa/service/nas/restores/duplicity-BEGNj3-tempdir/mktemp-2YCFpS-81
Removing still remembered temporary file 
/opt/aa/service/nas/restores/duplicity-BEGNj3-tempdir/mktemp-CXX67i-73
DEBUG:duplicity:Removing still remembered temporary file 
/opt/aa/service/nas/restores/duplicity-BEGNj3-tempdir/mktemp-CXX67i-73
Traceback (most recent call last):
 File "/usr/bin/duplicity", line 1241, in ?
   with_tempdir(main)
 File "/usr/bin/duplicity", line 1234, in with_tempdir
   fn()
 File "/usr/bin/duplicity", line 1188, in main
   restore(col_stats)
 File "/usr/bin/duplicity", line 542, in restore
   restore_get_patched_rop_iter(col_stats)):
 File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line 
519, in Write_ROPaths
   for ropath in rop_iter:
 File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line 
491, in integrate_patch_iters
   for patch_seq in collated:
 File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line 
376, in yield_tuples
   setrorps(overflow, elems)
 File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line 
365, in setrorps
   elems[i] = iter_list[i].next()
 File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line 
110, in difftar2path_iter
   tarinfo_list = [tar_iter.next()]
 File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line 
326, in next
   self.set_tarfile()
 File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line 
320, in set_tarfile
   self.current_fp = self.fileobj_iter.next()
 File "/usr/bin/duplicity", line 579, in get_fileobj_iter
   manifest.volume_info_dict[vol_num])
 File "/usr/bin/duplicity", line 603, in restore_get_enc_fileobj
   fileobj = tdp.filtered_open_with_delete("rb")
 File "/usr/lib/python2.4/site-packages/duplicity/dup_temp.py", line 
114, in filtered_open_with_delete
   fh = FileobjHooked(path.DupPath.filtered_open(self, mode))
 File "/usr/lib/python2.4/site-packages/duplicity/path.py", line 724, 
in filtered_open
   return gpg.GPGFile(False, self, gpg_profile)
 File "/usr/lib/python2.4/site-packages/duplicity/gpg.py", line 135, in 
__init__
   attach_fhs={'stdin': encrypt_path.open("rb"),
 File "/usr/lib/python2.4/site-packages/duplicity/GnuPGInterface.py", 
line 365, in run
   create_fhs, attach_fhs)
 File "/usr/lib/python2.4/site-packages/duplicity/GnuPGInterface.py", 
line 407, in _attach_fork_exec
   process.pid = os.fork()
OSError: [Errno 11] Resource temporarily unavailable

ERROR:duplicity:Traceback (most recent call last):
 File "/usr/bin/duplicity", line 1241, in ?
   with_tempdir(main)
 File "/usr/bin/duplicity", line 1234, in with_tempdir
   fn()
 File "/usr/bin/duplicity", line 1188, in main
   restore(col_stats)
 File "/usr/bin/duplicity", line 542, in restore
   restore_get_patched_rop_iter(col_stats)):
 File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line 
519, in Write_ROPaths
   for ropath in rop_iter:
 File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line 
491, in integrate_patch_iters
   for patch_seq in collated:
 File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line 
376, in yield_tuples
   setrorps(overflow, elems)
 File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line 
365, in setrorps
   elems[i] = iter_list[i].next()
 File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line 
110, in difftar2path_iter
   tarinfo_list = [tar_iter.next()]
 File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line 
326, in next
   self.set_tarfile()
 File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line 
320, in set_tarfile
   self.current_fp = self.fileobj_iter.next()
 File "/usr/bin/duplicity", line 579, in get_fileobj_iter
   manifest.volume_info_dict[vol_num])
 File "/usr/bin/duplicity", line 603, in restore_get_enc_fileobj
   fileobj = tdp.filtered_open_with_delete("rb")
 File "/usr/lib/python2.4/site-packages/duplicity/dup_temp.py", line 
114, in filtered_open_with_delete
   fh = FileobjHooked(path.DupPath.filtered_open(self, mode))
 File "/usr/lib/python2.4/site-packages/duplicity/path.py", line 724, 
in filtered_open
   return gpg.GPGFile(False, self, gpg_profile)
 File "/usr/lib/python2.4/site-packages/duplicity/gpg.py", line 135, in 
__init__
   attach_fhs={'stdin': encrypt_path.open("rb"),
 File "/usr/lib/python2.4/site-packages/duplicity/GnuPGInterface.py", 
line 365, in run
   create_fhs, attach_fhs)
 File "/usr/lib/python2.4/site-packages/duplicity/GnuPGInterface.py", 
line 407, in _attach_fork_exec
   process.pid = os.fork()
OSError: [Errno 11] Resource temporarily unavailable

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
 File "atexit.py", line 24, in _run_exitfuncs
   func(*targs, **kargs)
 File "threading.py", line 634, in __exitfunc
   t.join()
 File "threading.py", line 540, in join
   self.__block.wait()
 File "threading.py", line 203, in wait
   waiter.acquire()
</code>

Cheers and Thanks

Colin