Re: [rdiff-backup-users] Finding/designing a tar replacement

[dean gaudet]

> Neat, that's a good idea.  The strategy of using volumes as file
> systems should be an important consideration.  I didn't even know that
> Linux could mount an iso image.  (How can I try this?)

you want loopback device support -- either built in or as a module.
(i use this stuff all the time because i maintain image files for
simulators.)

then it's a simple matter of doing this:

        mount -o loop /path/to/isofile /mnt

to mount an image.  (isofile must be local -- loopback does not work over
nfs.)

i regularly make images of disks or filesystems.  to make a file system
image you can do this:

        dd if=/dev/hda1 of=/path/to/image.img

that'll make a copy of the entire hda1 partition... which you could then
mount as a loopback file.  (and you can later copy it right back to the
partition with dd.)

oh note -- it's generally important to use dd for image manipulation
rather than cat(1) because some cat(1) will deliberately create sparse
files, and loopback device does not support sparse files.  dd never
creates a sparse file.

full disk images are more of a pain to explain, you need to know about
partition tables and such... so i won't even try.

you can make a blank image and then create a filesystem on it:

        dd if=/dev/zero of=/path/to/newimage.img bs=1M count=256

would create a 256MB file of all zeros... then do this:

        losetup /dev/loop0 newimage.img
        mke2fs /dev/loop0
        losetup -d /dev/loop0

now newimage.img is a 256MB ext2 filesystem image... which you can
mount as described earlier.

one thing about iso9660 images which probably makes them inappropriate
for duplicity is that they require an upfront calculation of the
table of contents... and you need all the file metadata (such as
length) to do this.  i'm pretty sure this TOC has to be adjacent in
the image.  so this makes it hard to build the TOC on-the-fly as
you're generating the diffs... almost forces you to a two pass
technique.  (you could probably do a "two pass" technique which
generates a TOC file and a file containing all the diffs then
join them together... but i feel like you might as well make your
own fileformat if you're going this far.)

ext2 or ext3 along with resize2fs may be more what you're looking for.
with these you could create an image like i just described, then begin
filling it.  when you fill the image you unmount it, and do this:

        dd if=/dev/zero bs=1M count=256 >>newimage.img
        losetup /dev/loop0 newimage.img
        resize2fs /dev/loop0
        losetup -d /dev/loop0
        mount -o loop newimage /mnt

that will append 256MB to your image file, then resize the filesystem
to make use of the extra 256MB.

a resize like this is a bit of work because it has to go rewrite
all the superblocks, which might require it to move some blocks
around.  so you want to grow your image file in some large enough
increment.

once you're done you can figure out exactly how many blocks you need
for the image, and use resize2fs to shrink the filesystem then use the
trunc(2) system call to shrink the image.  (this is a bit tricky to get
right -- you need a few extra blocks probably.)

you probably want to play with mke2fs parameters such as the block
size -- you shouldn't really need 4096 byte blocks for a backup
like this, you can probably use 1024 byte blocks for better space
utilisation.  you need to also be careful about inode usage ...
i forget if resize2fs can increase the number of inodes or not.

finally -- there is a libext2 or some such similar name which you can
link against to modify such image files without mounting them (i think
it supports writes, maybe it's read-only though!).  you might be able
to resize2fs without the losetup step i described above... this would
eliminate the requirement of running as root for access to mount/losetup.

anyhow... i'm not so sure this is really the best approach (it's certainly
not very portable even though other freenix probably have loopback devices
:).  but using a filesystem directly certainly gives you an answer to a
lot of your metadata storage needs.

-dean