Re: [rdiff-backup-users] why does exclude move files to increments?

[Chris Wilson]

Hi Brandon,

> I want to use rdiff-backup to do incrementals, but I also only want to 
> process files that have changed in the last 24 hours. How come when I 
> use exclude directives that the directories excluded in subsequent 
> backups are moved to incrementals?

This is by design. The principle is that if you restore from the 
repository to a date before you excluded those files, then the files will 
be restored. Of course they have to be stored somewhere on the server in 
order to be restored. rdiff-backup treats them like any other file that 
disappeared (e.g. was deleted) from the source filesystem, and creates a 
compressed increment that describes the change, or in this case, how to 
recreate the file with its original contents.

This only happens once, when the file is excluded, and after that it won't 
be touched until you purge the snapshot that contains it from the 
repository.

> This is causing huge amounts of unnecessary I/O on my disk.

But only once, the first time that you exclude each file.

> I want to create a baseline rdiff-backup and then only push updates 
> daily by using mtime.

That is what rdiff-backup does for you.

> I have 300GB worth of data and it takes an insane 16 hours to run. 
> That's fine for a baseline,but not an incremental.

Sorry, rdiff-backup is quite slow in my opinion (I have a similar 
problem). I think it's inefficient in marshaling data to send over the 
network. I've worked around it by rsyncing my data to the backup server 
and running rdiff-backup locally there, but that may not work for you, 
especially if you don't want two copies of your data on the backup server.

> Does rdiff-backup to a file compare on every single run?

I think it compares checksums of local file blocks against checksums of 
remote file blocks, to determine which blocks have changed and must be 
sent again. Just as rdiff does.

> I create my baseline rdiff-backup with the following:
> rdiff-backup --print-statistics
> --exclude-other-filesystems --exclude-sockets
> --exclude-device-files --exclude-fifos --include /test
> --exclude '**' / /backup/testing
...
> Now when I run rdiff-backup on just those files that
> have changed, it moves the non-changed files to
> increments! Why?

Because you excluded all files from the backup with --exclude '**'.

> I am simulating the non-changed files
> by excluding dir2 from my processing. I only want to
> update the rdiff-backup tree for changed files based
> on mtime to avoid the processing and I/O caused by
> comparing all files' signatures.

I think that is not a valid test.

> Here is my command for only processing changed files:

> find /test -mtime 0 | rdiff-backup --print-statistics
> --exclude-other-filesystems --exclude-sockets
> --exclude-device-files --exclude-fifos --exclude
> /test/dir2 --include-filelist-stdin --exclude '**' /
> /backup/testing

Umm, rdiff-backup will ignore the output of "find /test -mtime 0" that you 
pipe into it. It works out which files have changed for itself. Just run 
it the same way each time (without the find command). It will do what you 
want.

Hope that helps.

Cheers, Chris.
--
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |