Re: [Libcdio-devel] Invalid/bad directory errors with openSUSE Leap 15.0 ISO

[Thomas Schmitt]

Hi,

the disease seems not new. I get similar errors with:
  openSUSE-Leap-42.1-NET-x86_64.iso
  openSUSE-13.1-NET-x86_64.iso

To blame is

  Preparer   : KIWI - http://opensuse.github.com/kiwi

which obviously fiddles with the ISOs after they were produced by the
SuSE version of genisoimage. (Among other peculiarities, the ISOs have
two heads and an EFI System partition between them.)

The damaged directory record in
  openSUSE-Leap-15.0-DVD-x86_64.iso
belongs to file

  /CHECKSUMS.asc

In the older two ISOs it is file /content.asc .

One would have to file a bug to SuSE/Kiwi pointing out ECMA-119, 7.3.3
"Both-byte orders" and 9.1.4 "Data Length (BP11 to BP18)", which prescribe
that the byte count of a data file is to be recorded as first a
little-endian 32-bit number and then a big-endian 32-bit number.

Kiwi obviously only patches the little-endian bytes into the ISO when
it sets the final size of the affected files (which are GPG signatures
probably made after the ISO was finished).
 
The file in openSUSE-Leap-15.0 bears as size:

  e1  01  00  00  00  00  08  00

where the first four bytes are correct and the next four are wrong.
Probably because only the first four were overwritten by Kiwi, when it
patched in the new size after manipulating the file content.

Since the error is noticed during reading of the file's directory record,
libcdio has not yet learned its name but only the location of the damaged
record.

Currently libcdio reacts quite ungraceful on the situation.
Linux and libisofs tolerate it:

  # mount openSUSE-Leap-15.0-DVD-x86_64.iso /mnt/iso
  $ ls -l /mnt/iso/CHECKSUMS.asc
  -r--r--r-- 1 root root 481 May 16 19:45 /mnt/iso/CHECKSUMS.asc

and

  $ xorriso -indev openSUSE-Leap-15.0-DVD-x86_64.iso -lsdl /CHECKSUMS.asc --
  ...
  -r--r--r--    1 0        0             481 May 16 19:45 '/CHECKSUMS.asc'


Tolerance would probably have to be implemented deep down in
  from_733_with_err (uint64_t p, bool *err)
in
  include/cdio/bytesex.h
which detects the situation and issues the first warning message.


---------------------------------------------------------------------------
 
We should try to fid out why Pete's libcdio clone bails out with
 
  ++ WARN: Bad directory information for docu

whereas mine (quite fresh) reports

  ++ WARN: Invalid directory stat at offset 510

---------------------------------------------------------------------------
Here's he long story how i came to above conclusion 

Pete Batard wrote:
> ++ WARN: from_733: broken byte order

This comes from inline functioni
  from_733_with_err (uint64_t p, bool *err)
in
  include/cdio/bytesex.h
if
  if (uint64_swap_le_be (p) != p) {
which effectively tests whether p as 64 bit byte string complies to
ECMA-119, 7.3.3 "Both-byte orders", which prescribes to store a 32-bit
unsigned integer first as little-endian and then as big-endian byte string.


> ++ WARN: Bad directory information for docu

Function _fs_iso_stat_traverse() in
  lib/iso9660/iso9660_fs.c
reports this if _iso9660_dir_to_statbuf() in the same file returns NULL.
_iso9660_dir_to_statbuf() calls from_733_with_err() on two occasions
when it looks for the block address and for the size of the file.
Those are the bytes from offsets 2 to 9 and and 10 to 17 (BP3 to BP10,
BP 11 to BP18 in ECMA-119 speak).


> docu/RELEASE-NOTES.en.txt

I downloaded the ISO and now look at the directory record of that file
at offset 112228, i see

   cc  f4  02  00  00  02  f4  cc  53  3c  00  00  00  00  3c  53
  204 244   2   0   0   2 244 204  83  60   0   0   0   0  60  83

which says block address 193740 and size 24723 bytes, and looks exactly
like the prescribed byte waste by mirroring.
So it's not that file but probably one of its siblings or a sibling of
its boss directory.

Hrmpf. My clone does not get there. It rather reports:

  ++ WARN: Invalid directory stat at offset 510

Well, "offset" is welcome. But the directory's LBA would have been nice.
The message stems from iso9660_ifs_readdir() in lib/iso9660/iso9660_fs.c .
I augment it by:

-          cdio_warn("Invalid directory stat at offset %lu", (unsigned 
long)offset);
+          cdio_warn("Invalid directory stat at LBA %u , offset %lu",
+                    p_stat->lsn, (unsigned long)offset);

This yields:

  ++ WARN: Invalid directory stat at LBA 29 , offset 510

That would be absolute byte offset 59902. Two bytes further i see

  05  e4  00  00  00  00  e4  05  e1  01  00  00  00  00  08  00

So the specs violation is in the file size field: 0x1e1 != 0x800.

The file goes by the ISO 9660 name "CHECKSUM.ASC;1".
Rock Ridge name is "CHECKSUMS.asc".

xorriso lists it with size 481 = 0x1e1. So the 0x800 alternative is wrong.
The content as extracted by xorriso looks plausible:

  -----BEGIN PGP SIGNATURE-----
  Version: GnuPG v1.0.7 (GNU/Linux)

  iQEVAwUAWvxud7iLL9Q9vcKEAQgmPggAwnGfRxlVpfjjBF07FQS7jQqeOQ47/fcS
  KwHWsxab0uoNNKvWV8QtGOLKSfKxdwDioE+cJU+7ZiPXjrHI7KsCyy+xV6GE75l+
  BEF+Ud4xuKxV7KF1jnG/8eHeVXywvA4Q4i2AyBGcIjrr1a10p0g+GyIt7ueaJJ1i
  1BGGn5AGyi6chE2MAGsBoDk4AxS3yUXKWgF4lGU7H2ZBNiNSFYy3OeRfKzmrLJPG
  C0orEG75pm3hM9/7v3AbnMHN0pGr3NRks/RJREBYGFTsiu7sDQ7RMNLdpa/0xpDn
  hEFI5t+0p/KIuKTISOfiJZDUGtooIF331K37P36hi9q3lvfmQIlTtg==
  =o7aX
  -----END PGP SIGNATURE-----

---------------------------------------------------------------------------

Have a nice day :)

Thomas