Re: [rdiff-backup-users] Atomicity of backups?

[Dave Steinberg]

I would expect that you'd have to implement more or less the equivelant
of a database redo log.  There's no shortage of reading material on this
topic, thankfully.  Maybe something simpler would suffice?

You would have to fsync() after every write.  Or use a raw device
(hah!).  There's no other way to avoid OS level buffering.  Of course
you could start clustering write()'s, and fsync() to group these
together.  Gah...  I'm afraid.

Case (1):  I do believe that a 'mv' is atomic to the OS, but groups of
'mv's aren't.  If I understand your explanation correctly, the temp
files are moved into position where the old ones lived.  You might want
to log their metadata (including where they're supposed to go), in the
event of failure and then checkpoint it when its complete.  That seems
reasonable, and not so hard, to me.  Individual files would always be
consistent, and any leftovers can be picked up on the next run.

Case (2) seems moot, as a lack of data would prevent any sort of
roll-forward that you'd like.  If the files themselves haven't been
processed, why shouldn't you get half-new and half-old?  The new ones
are guarenteed to be consistent by Case (1), and well, the old ones are
already consistent!

I'd assume that the operator would rerun the backup if there was a
crash.  It suffices for rdiff-backup to be able to return to a
consistent state given any time specification.

Just some thoughts,

--
Dave Steinberg

----- Original Message -----
From: "Ben Escoto" <address@hidden>
To: "Nathaniel Smith" <address@hidden>
Cc: <address@hidden>
Sent: Tuesday, December 10, 2002 12:23 PM
Subject: Re: [rdiff-backup-users] Atomicity of backups?

>>>>> "NS" == Nathaniel Smith <address@hidden>
>>>>> wrote the following on Tue, 10 Dec 2002 01:31:28 -0800

  NS> Conceptually, it seems like you need something like journalled
  NS> commits, where after a crash it's always possible to revert to
  NS> the last-good-version.  Even just making a full copy of the
  NS> mirror directory before modifying it would be a good simple
  NS> solution, albeit somewhat hard on the disk space :-).  What does
  NS> it do now?

Well, there are two possibilies:  1) crash in the middle of modifying
one file, 2) crash in the middle of modifying the directory as a
whole.

    For 1), the strategy is to try to not make this any worse than 2
in general.  So rdiff-backup writes new mirror files and increments to
temp files, and then moves them both into position right after
another.  However, there is a (fairly small) period of
vulnerability---if a hard crash happens right in between, then data for
that one file could be lost.  The temp files would still be there, but
manual recovery might be necessary.

    For 2), the directory is still in a consistent state as far as
rdiff-backup is concerned, but it may not be the state of the source
directory at any particular time (it really isn't anyway though, since
the directory can get modified while rdiff-backup is working).  So if
a session crashes halfway through and you try to restore to that time,
the first half of the restore will have new files, and the second half
will have older files from the last successful session.


But now that you mention it, and I am thinking in those terms, that
journalling stuff sounds like a good idea (especially since, as I
mentioned before, new metadata stuff is going to make this harder).
So I guess the basic idea is I write what I'm going to do to a file,
and then do it, and the next instance reads the file and sees when the
crash happened?  Do I have to keep flush()ing the file then?  In
practice does this make things much slower?  If anyone has any opinion
on whether journalling would be appropriate here, or wants to point me
towards some basic information on the topic, that might be useful.


--
Ben Escoto