[rdiff-backup-users] Increased resource usage in rdiff-backup 0.12.3

[Luke Mewburn]

I recently upgraded from 0.10.x to 0.12.3, and noticed that
rdiff-backup has increased resource usage requirements.

For example, on one day a backup failed with
        File "/usr/pkg/lib/python2.2/gzip.py", line 43, in __init__
        IOError: [Errno 24] Too many open files:

The next next run of rdiff-backup for that same file system noticed
the previous run had failed, tried to regress, and failed with a
memory error:

        Previous backup seems to have failed, regressing destination now.
        Traceback (most recent call last):
          File "/usr/pkg/lib/python2.2/site-packages/rdiff_backup/rpath.py", 
line 873,
        in getincbase
            return self.__class__(self.conn, self.base, self.index[:-1] +
          File "/usr/pkg/lib/python2.2/site-packages/rdiff_backup/rpath.py", 
line 549,
        in __init__
            else: self.data = self.conn.C.make_file_dict(self.path)
        MemoryError

I could reproduce the latter if I ran rdiff-backup manually with
--check-destination-dir.


Further investigation led me to invoking rdiff-backup with a larger
datasize ulimit to get the "--check-destination-dir" to succeed; I had
to increase the softlimit datasize from 128MB to 512MB.
Interesting, I only needed this for the --check-destination-dir.

A "normal" run for that file system only needs ~ 15MB of RAM,
but a --check-destination-dir regression run was using ~ 170MB of RAM!
This disparity between the normal and regression runs is astounding.


For what it's worth, I prevent the initial error (Too many open files
in gzip.py) from occurring by cranking the open files rlimit from 64
to 512, just to be "safe".  My do-rdiff-backup script now has:
        ulimit -S -d 524288             # up from 131072
        ulimit -S -n 512                # up from 64


I don't think I had these issues with rdiff-backup 0.10.x.  Is there
any reason as to why rdiff-backup needs much higher resource limits
now?


BTW: thanks for a great product :)


Luke.