Re: mkisofs mastering sparse files - end result file full of zeros

[Thomas Schmitt]

Hi,

(What happened about the suspicion that xorriso zeros sparse files ?
Any new insight ?)


Graham Leggett wrote:
> [root@arnie optical]# /usr/bin/growisofs -dry-run -quiet -iso-level 3 -f -Z
> /dev/sr1 -r -J -A 'Example VMs' -V 'Example VMs 2024-11-25'
> /var/spool/backup/optical/virtual-machines
> ...
> :-( /dev/sr1: 24438784 blocks are free, 48920021 to be written!

xorriso told growisofs how many blocks it would write. growisofs asked the
drive about the free size on the medium.

(xorriso would probably detect the same 24438784 blocks of capacity if you
let it do the burning. If run under growisofs then libburn writes to a
pipe, which it handles as emulated drive. See below for a proposal to
run xorriso as integrated burn program.)


> The size of the sparse data is as follows, and 40G should comfortably fit on
> a 50G bluray.
> [root@arnie optical]# du -c -h -L virtual-machines/
> ...
> 40G virtual-machines/

"du" by default does not give you the sum of the file sizes but the
amount of storage that is used to represent them in their hosting
filesystem (ext4 in your case ?).


> [root@arnie optical]# du --apparent-size -c -h -L virtual-machines/
> ...
> 94G virtual-machines/

And this is what libisofs sees as sizes when it reads the files.

Alternatively you may run

  ls -lr virtual-machines/

I expect that it will show you sizes which sum up to a bit less than
48920021 * 2048 = 100,188,203,008 bytes (libisofs adds some metadata).


> The not-enough-space would suggest it is taking into
> account the non-sparse size for calculating space, and not the sparse size.

Sparse file representation is an inner detail of filesystems.
ISO 9660 specs do not provide means to represent gaps in files.
So libisofs has to record the bytes without gaps. Your 40 GB on disc
would indeed expand to 94 GB in the ISO 9660 filesystem.

The problem is not that i could not invent an extension to ISO 9660,
but that the operating system kernels would need to learn how to recognize
it and how to interpret it.
(See below my musings about "zisofs".)

If you run growisofs with real mkisofs or its clone "genisoimage",
then i am sure you will get the size overflow message, too.


> I wonder if there is something odd on the code path when symlinks are used.

I would be astonished. There is not much magic with symbolic links.
They just contain a file path, which may be used to open and read the
link target.
So if libisofs records the link object, you get in the ISO the size of the
file path, usually a few dozen bytes. If its records the link target,
you get the "ls -l" size of the target. In your case 94 gigabytes.
Anything inbetween would be an oddity of system calls open(2) and read(2).


---------------------------------------------------------------------
Why i do not propose zisofs for now:

If the Blu-ray shall be read by Linux kernels only and the files would
be smaller than 4 GiB, then zisofs compression would be a way to get
something similar to sparse files. Linux kernels know zisofs and can
uncompress the file content on the fly.

This compression format has a shortcut to represent data chunks which
contain only 0-bytes. Chunk size may be 32 KiB, 64 KiB, But the
uncompressed file size is limited to 4 GiB - 1.
There is a specification for 64-bit zisofs v2. xorriso can write it and
extract files from it. But the Linux kernel does not implement it.


-----------------------------------------------------------------------
The equivalent of your growisofs run done with native xorriso commands,
as documented in man 1 xorriso:

  xorriso -report_about mishap \
          -follow link \
          -outdev /dev/sr1 \
          -joliet on \
          -application_id 'Example VMs' \
          -volid 'Example VMs 2024-11-25' \
          -map /var/spool/backup/optical/virtual-machines / \
          -find / -exec mkisofs_r --

(The -find command performs the equivalent of mkisofs option -r.
Due to the rules for xorriso commands it has to be given after all
files have been mapped in, not before.)

-----------------------------------------------------------------------


Have a nice day :)

Thomas