[Rdiff-backup-bugs] [bug #22183] Feature request - speed improvements at cost of integrity for large systems

[Guru Evi]

URL:
  <http://savannah.nongnu.org/bugs/?22183>

                 Summary: Feature request - speed improvements at cost of
integrity for large systems
                 Project: rdiff-backup
            Submitted by: guruevi
            Submitted on: Wednesday 01/30/2008 at 21:59
                Category: None
                Severity: 3 - Normal
              Item Group: None
                  Status: None
                 Privacy: Public
             Assigned to: None
             Open/Closed: Open
         Discussion Lock: Any

    _______________________________________________________

Details:

Hi,

I just would like to know if it were possible, to implement a switch somehow
that would improve the speed of rdiff-backup significantly by not comparing
file-by-file by calculating SHA1 (I believe that is the approach you take for
comparing files). 

I have rather large binary files (in the order of hundreds of megabytes) or
very much small (1000's of 16k files from a medical scanner) on a very large
file system (9 terabytes) and just a run through with 6 Gbyte of changed files
takes about 72 hours for rdiff-backup and takes up a whole CPU (not really
what we like). The metadata file which I believe contains all the SHA1 hashes
is a whopping 3 Gigabyte. The files itself however never change, they either
get created or removed.

If there were a switch that could do turn off sha1 hashing for something way
less expensive (md4) or something else that would save a lot since (depending
on your machine) md4 is about 10-200% faster.

If you have additional information or tips (maybe insert it into the
documentation) on how to speed up rdiff-backup, I would like to know. I'm
going to switch from SSH to straight NFS (currently pushing the ACL for the
backup user), I already tried turning off compression but that didn't seem to
help a whole lot (you probably do the compression after a complete backup?)




    _______________________________________________________

Reply to this item at:

  <http://savannah.nongnu.org/bugs/?22183>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.nongnu.org/