Re: [rdiff-backup-users] Wasted bandwidth and accounting for it

[Gavin]

Hi Chris,

There is talk of kicking off a new version or rewrite of rdiff-backup
but not much in the way of
feature addition happening at the moment (from what I can tell as a user).

Anyway the best is to take snapshots of changing files and then
rdiff-backup those since rdiff-backup
does not handle changing files. This goes for SQL and SVN etc repo's
too, use their hotcopy or dump
functions and then backup the dump. Backupninja can be used to setup
such a backup system.

Cheers
Gavin

Chris Wilson wrote:
> Hi all,
>
> I have been a pretty happy user of rdiff-backup for many years.
> Currently all servers that I control use rdiff-backup 1.0.x for local
> and offsite backups. The reason for sticking with an ancient version
> is simply that I cannout upgrade my backup servers and all clients at
> the same time, and cross-version compatibility is difficult.
>
> I recently discovered that one of my offsite hosts had been backing up
> an increasing amount of data each day:
>
>   http://i41.tinypic.com/2qajtyx.png
>
> (this host is shown in dark green). When I investigated, the session
> statistics showed that the change in the destination was small:
>
> $ sudo cat
> rdiff-backup-data/session_statistics.2010-05-07T00:17:43+01:00.data
> StartTime 1273187863.00 (Fri May  7 00:17:43 2010)
> EndTime 1273229597.92 (Fri May  7 11:53:17 2010)
> ElapsedTime 41734.92 (11 hours 35 minutes 34.92 seconds)
> SourceFiles 144698
> SourceFileSize 11994143167 (11.2 GB)
> MirrorFiles 144697
> MirrorFileSize 12043475927 (11.2 GB)
> NewFiles 5
> NewFileSize 12542425 (12.0 MB)
> DeletedFiles 4
> DeletedFileSize 61886561 (59.0 MB)
> ChangedFiles 74
> ChangedSourceSize 14252455 (13.6 MB)
> ChangedMirrorSize 14241079 (13.6 MB)
> IncrementFiles 84
> IncrementFileSize 8256201 (7.87 MB)
> TotalDestinationSizeChange -41076559 (-39.2 MB)
> Errors 0
>
> Even though this backup took 11.5 hours to finish, and in fact
> transferred 2.6 GB of data!
>
> I looked into the error log, and this gave me the clue that some huge
> files (e.g. 2GB Argus packet logs) were being tranferred and then
> discarded because they were still changing on the sending side.
>
> First of all, the files transferred and then discarded are not logged
> anywhere in the session statistics. I think it would be useful to
> track this as a metric. Could I request this as a feature, if 1.2.x
> doesn't already do it? (we will upgrade eventually).
>
> I think in some cases, for example log files, I would prefer to keep
> the "corrupted" copy as part of the snapshot, as I know that the file
> will just be growing and I'd rather not discard and transfer it again
> every time. Could I request this as a feature for 1.2.x as well?
>
> Perhaps it would even be possible, if the checksum fails, to compare
> the checksums of the first X bytes (the recorded length when the
> transfer started), and if these match, to truncate the file on the
> destination to that length? Could I request that as another feature
> for 1.2.x?
>
> Cheers, Chris.