Re: [rdiff-backup-users] Q. on max-file-size behavior

[Whit Blauvelt]

On Sun, Mar 14, 2010 at 10:27:33PM +0100, Maarten Bezemer wrote:

> " When backing up, if a file is excluded, rdiff-backup acts as if that
>   file does not exist in the source directory."

Okay. But that's not the only option for how it could act. It's acting as if
something were true that's not. It wouldn't take much for it to represent
the true situation instead. I spelled out how it could do that in the prior
post. Tracking reality, when it doesn't cost much, should be preferred.

> I think your problem is not with the gzipping. I think you want to
> use rdiff-backup in a way it was never designed to be used.

Sure. That's the beauty of *nix utilities. Their use, for the best of them,
is dependent on the creativity and vision of the user, not just on that of
the original author. The more flexible and general purpose the utility's
design, the more it lends itself to prospering in the *nix ecology.

> That's just a Bad Idea (tm). The whole idea of "restore to a
> specific point in time" implies that you then get back the tree as
> it was at the time you specified. Not a tree with small files from
> that date/time, and with large files from an earlier date.

It might be a bad idea on your systems, and a good idea on mine.

> So, why not just use both --max-file-size and --min-file-size on two
> separate backup trees? That would exclude the large files from the
> smallfiles-tree, and the small files from the largefiles-tree, so no
> redundancy. And you can use different backup schedules for both
> trees.

Intriguing idea. 

> To make things more easy, I think I'd just create two backup trees,
> based on file paths. Huges files with sizes like you mentioned
> usually show up on well-defined places in a file system, and not
> just between a normal user's mozilla preferences file and a list of
> recently opened documents. 

Okay, but doesn't apply here. I'm talking about corporate servers handling
financial data. There are no "normal user" accounts on them. I've been using
rsync and rsnapshot for years in various areas of this, but for some parts
of the overall system rdiff-backup looks advantageous. So I'm seeing how far
I can go to bend it to our use. For my own workstations rsnapshot is more
ideal. But for the financial data servers, I can see some real advantages to
integrating rdiff-backup into our scheme. I'm sure some of my uses will be
"off label," not anticipated. I do that alot with software. It's what they
pay me for.

> If you want to compromise, you don't get what you want, and also you
> get things you don't want. That's not only a matter of language,
> it's just something you don't want when designing a backup system.
> If you want speed (assuming, for the sake of argument, that gzipping
> is your only problem), just get larger disks. Extra 1TB of disk
> space costs way less than changing rdiff-backup to something it was
> never designed to be.

Changing? No, augmenting. The previously-designed-in aspects would stay the
same. Nor are we talking about just 1TB, or consumer-quality drives.

> Actually, your suggestion would only help for large files being
> deleted (or excluded) from the source tree. For your suggestion to
> be really useful, you would need to have a source tree that has this
> happening on a regular basis. And in that case, the time spent in
> gzipping will be so much less of a problem than the amount of disk
> space that will be used by all the increments. (Or you would need to
> keep such a short history that you shouldn't be using rdiff-backup
> at all, making this discussion moot anyway.)

Eh. You don't know our usage pattern. Actually, for the files we're
handling, it would make sense. You assume too much. Not everyone's patterns
match those you're familiar with. General purpose tools are best, but
lacking those, we make due with what we can. Thanks for the spirited
and stimulating conversation. 

Whit