[rdiff-backup-users] debugging weird memory error

[Rolf Braun]

Recently, a system on which I administer backups remotely has not been
able to complete a run of rdiff-backup (the data set is 500 GB but the
backup is completing less than 15% before failing). This is the case
even after I remove the entire directory and retry the backup; in
fact, it started after a server upgrade in which the RAID array was
replaced and the backups were to be restored from the original (there
are also off-site backups). In fact, we have never been able to get a
full backup since the array was replaced. What appears to happen is
that the backup chugs along fine for a time, with some significant
memory usage mainly on the server but no obvious leaks. At some point
the backup starts getting slow and then it hangs altogether for a
while before dying with the memory error listed below. Running "top"
shows less memory usage on the client than the server until around
when the error happens, then the usage balloons on the client while
the server process appears to be idle. It takes about the same amount
of time each try for the backup to fail, and excluding the directory
where it normally fails simply results in it failing elsewhere. A line
from top output on the client:

 PID   UID  REG RPRVT  RSHRD  RSIZE  VPRVT  VSIZE  TH PRT    TIME  %CPU
24908     0 4765 1.25G+  972K  1.07G+ 1.28G+ 1.28G+  1  27   2h08m  90.4

This error occurred both with our on-site version (hacked to sort the
metadata case insensitive) and with stock 1.1.5, deleting all metadata
in between tests. The only differences between a couple months of
error-free backups and these latest attempts are that the disk array
on the server was replaced and the server OS was upgraded to a new
point release of Mac OS X Tiger, although some other work may have
been done that I'm not aware of. There is no indication that any
hardware failure is occurring and there is plenty of disk space
available on the server.

The following error is when trying to do a backup run with stock
rdiff-backup 1.1.5, all metadata (the rdiff-backup-data directory)
removed in advance. Any help figuring out what is going on here would
be appreciated, as I've worked with the software and hacked around
issues we've had for a little while, but this has me stumped.

Exception '' raised of class 'exceptions.MemoryError':
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/robust.py",
line 32, in check_common_error
   try: return function(*args)
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/rpath.py",
line 878, in listdir
   return self.conn.os.listdir(self.path)

Sending back exception  of type exceptions.MemoryError:
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 335, in answer_request
   result = apply(eval(request.function_string), argument_list)
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 483, in readfromid
   if length is None: return cls.vfiles[id].read()
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 302, in read
   if not self.addtobuffer(): break
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 325, in addtobuffer
   try: currentobj = self.iter.next()
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/rorpiter.py",
line 342, in next
   next_elem = self.iter.next()
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/selection.py",
line 132, in Iterate_fast
   try: rpath, val = diryield_stack[-1].next()
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/selection.py",
line 118, in diryield
   for filename in self.listdir(rpath):
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/selection.py",
line 180, in listdir
   dir_listing = robust.check_common_error(error_handler, dir_rp.listdir)
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/robust.py",
line 32, in check_common_error
   try: return function(*args)
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/rpath.py",
line 878, in listdir
   return self.conn.os.listdir(self.path)

Sending back exception  of type exceptions.MemoryError:
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 335, in answer_request
   result = apply(eval(request.function_string), argument_list)
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 483, in readfromid
   if length is None: return cls.vfiles[id].read()
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 302, in read
   if not self.addtobuffer(): break
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 325, in addtobuffer
   try: currentobj = self.iter.next()
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/backup.py",
line 166, in get_sigs
   for src_rorp, dest_rorp in cls.CCPP:
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/backup.py",
line 306, in next
   source_rorp, dest_rorp = self.iter.next()
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/rorpiter.py",
line 103, in Collate2Iters
   for relem1 in riter1:
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 382, in next
   while not type: type, data = self._get()
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 414, in _get
   if not self.buf: self.buf += self.file.read()
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 516, in read
   return self.connection.VirtualFile.readfromid(self.id, length)
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 448, in __call__
   return apply(self.connection.reval, (self.name,) + args)
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 370, in reval
   if isinstance(result, Exception): raise result

Sending back exception  of type exceptions.MemoryError:
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 335, in answer_request
   result = apply(eval(request.function_string), argument_list)
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 483, in readfromid
   if length is None: return cls.vfiles[id].read()
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 302, in read
   if not self.addtobuffer(): break
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 325, in addtobuffer
   try: currentobj = self.iter.next()
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/backup.py",
line 103, in get_diffs
   for dest_sig in dest_sigiter:
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 384, in next
   elif type == "r": return self.get_rorp(data)
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 394, in get_rorp
   rorp.setfile(self.get_file())
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 399, in get_file
   type, data = self._get()
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 414, in _get
   if not self.buf: self.buf += self.file.read()
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 516, in read
   return self.connection.VirtualFile.readfromid(self.id, length)
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 448, in __call__
   return apply(self.connection.reval, (self.name,) + args)
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 370, in reval
   if isinstance(result, Exception): raise result

Sending back exception  of type exceptions.MemoryError:
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 335, in answer_request
   result = apply(eval(request.function_string), argument_list)
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/backup.py",
line 216, in patch
   for diff in rorpiter.FillInIter(source_diffiter, dest_rpath):
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/rorpiter.py",
line 177, in FillInIter
   for rp in rpiter:
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 382, in next
   while not type: type, data = self._get()
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 414, in _get
   if not self.buf: self.buf += self.file.read()
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 516, in read
   return self.connection.VirtualFile.readfromid(self.id, length)
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 448, in __call__
   return apply(self.connection.reval, (self.name,) + args)
 File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 370, in reval
   if isinstance(result, Exception): raise result