rdiff-backup-users
[Top][All Lists]
Advanced
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[rdiff-backup-users] Slow remote backups

From: Peter Howell
Subject: [rdiff-backup-users] Slow remote backups
Date: Sun, 22 Feb 2004 10:20:53 -0000

Hi

Problem: Using 0.12.6 rdiff seems to be detecting that more files have changed than there really are. I am running 0.10.2 and 0.12.6 in parallel and can compare.

I have been using version 0.10.2 for over a year now with one on site machine and another remote off site using the ssh keys setup and compression set to 'yes'. The remote machine would connect to the onsite rdiff machine and this worked fine. We have about 30G in 260000 files which get backed up every day on site and off site. We change about 30000 files per day with a net change of about 2-300M but the total size of the files changes is about 10G hence the differential backup is essential. The connection speed of the remote machine is 150kb and the remote backup took about 4 hours, as long as no one did any thing stupid in the office like renaming directories with large trees underneath them!

I have recently upgraded the remote machine (from suse to Debian-testing) and switched to 0.13.3. I set up rdiff 0.13.3 on another host so to leave the original on site machine running 0.10.2. The new remote backup works but it takes about 24 hours to complete. Repeated backups get no quicker.

Comparing the logs from one of the 15 rdiff-backup lines in the backup script between 0.10.2 and 0.12.6 , each backup which should be the almost identical (variation caused by different start backup times) the 0.12.6 seems to find many more files that have changed. Automated processes are changing the databases so I would expect there to be differences in new and changed files but not this many!

rdiff-backup  0.10.2(on site)
StartTime 1077314222.38 (Fri Feb 20 21:57:02 2004)
EndTime 1077320608.93 (Fri Feb 20 23:43:28 2004)
ElapsedTime 6386.55 (1 hour 46 minutes 26.55 seconds)
SourceFiles 151044
SourceFileSize 7129092134 (6.64 GB)
MirrorFiles 151004
MirrorFileSize 7069675964 (6.58 GB)
NewFiles 48
NewFileSize 50083500 (47.8 MB)
DeletedFiles 8
DeletedFileSize 15155134 (14.5 MB)
ChangedFiles 12380
ChangedSourceSize 5736262719 (5.34 GB)
ChangedMirrorSize 5711774915 (5.32 GB)
IncrementFiles 12436
IncrementFileSize 19071706 (18.2 MB)
TotalDestinationSizeChange 78487876 (74.9 MB)

rdiff-backup 0.12.6 (remote)
StartTime 1077398625.00 (Sat Feb 21 21:23:45 2004)
EndTime 1077436593.75 (Sun Feb 22 07:56:33 2004)
ElapsedTime 37968.75 (10 hours 32 minutes 48.75 seconds)
SourceFiles 151601
SourceFileSize 7101188536 (6.61 GB)
MirrorFiles 150302
MirrorFileSize 6886185213 (6.41 GB)
NewFiles 1345
NewFileSize 164666434 (157 MB)
DeletedFiles 46
DeletedFileSize 33064 (32.3 KB)
ChangedFiles 150256
ChangedSourceSize 6936522102 (6.46 GB)
ChangedMirrorSize 6886152149 (6.41 GB)
IncrementFiles 151647
IncrementFileSize 30402265 (29.0 MB)
TotalDestinationSizeChange 245405588 (234 MB)

I can explain some of these changes as the time of the remote and the on site backup where different. And there were a number of new files added between the on site backup running and the remote backup. However the puzzling bit is the 'Changed files'. In 0.12.6 we have almost all the files changed and in 0.10.2 only about 10% of the files have been marked as changed.

The three machines invloved are running

on site backup rdiff 0.10.2 suse 8.2 - rpms installed from the rdiff site

on site backup remote host rdiff 0.12.6 suse 8.2 - rpms installed from rdiff site
remote rdiff 0.12.6 - debian (testing) /deb created by alien from rpm from rdiff site

There may be an issue here in that I had originally installed version 0.13.3 on the debian remote using apt-get (which had the same problem) and then removed it and put on version 0.12.6. I have assumed the librsync files have not changed. All machines are running python 2.2.

The only thing I can think of is that there is something different in the versions of librsync that I am running I dont think there has been any changes here. I dont know how to find out what version of librsync I am running?

Any ideas?

As a way of thanks to all those involved in writing and maintaining rdiff-backup - What a fantastic program! Over the last year it has saved me and others on many occassions plus given the business a level of data secutrity that I used to only dream about. Thanks again :-)



______________________________________________________________

This email is confidential and intended solely for the use of the individual to whom it is addressed. Any views or opinions presented are solely those of the author and do not necessarily represent those of TEAM (EAA Ltd). If you are not the intended recipient, be advised that you have received this email in error and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited.

______________________________________________________________


reply via email to
[Prev in Thread] Current Thread [Next in Thread]