Re: Tarlz 0.26 'make check' test failure on Debian 12

[Aren Tyr]

Hello Antonio

Thanks for the informative reply.

On 23/01/2025 01:54, Antonio Diaz Diaz wrote:

> No, it is not normal. I always use level -9 and never had a problem. If
> level -9 has a problem, I'm very interested in fixing it.
>
> Level -9 may run out of memory on some machines, but if it happens it is
> clearly diagnosed.
>
> Level -9 uses more memory than level -6, and is therefore more frequently
> affected by RAM errors. I recommend to repeat the exact command where you
> used level -9 and compare the resulting archive with the faulty one:
> - If they are identical, it is a bug in tarlz. If this is the case, please,
> send the output of 'lzip -tvvvv'.
> - If they are different and the second is correct, it may be a bug (data
> race), or a memory problem.
> - If they are different but both are faulty, it may be also a bug (data
> race), or a memory problem. In this case try creating the archive with one
> thread (--threads=1) to discard a data race.

So it looks like your suspicion was correct, in that it [must surely be/is] a 
memory/RAM error (it is interesting to know that this is still a pertinent 
hardware issue on modern machines). The machine I am running this on has 128GB 
RAM, so quantity/resources is never an issue.

Anyway, I tried creating a fresh archive with -9 again today, and could not 
reproduce the issue. This would add further weight to it being a RAM issue 
since the machine was powered off and obviously the live state entirely 
different from 24 hours previous. I ran lzip -tvvvv on the resultant achive as 
you suggested, and got this output:


   files.tar.lz: dict   32 MiB,  1.331:1, 75.16% ratio, 24.84% saved. CRC 
436ED216,  66670080 out, 50107520 in. ok
   files.tar.lz: dict   32 MiB,  1.008:1, 99.24% ratio,  0.76% saved. CRC 
4C463A53,  40073728 out, 39767927 in. ok
   files.tar.lz: dict   32 MiB,  1.132:1, 88.37% ratio, 11.63% saved. CRC 
C640AE66, 319841792 out, 282650901 in. ok
   files.tar.lz: dict   32 MiB,  1.279:1, 78.18% ratio, 21.82% saved. CRC 
DAC64363,  67428352 out, 52712237 in. ok
   files.tar.lz: dict   32 MiB,  1.100:1, 90.95% ratio,  9.05% saved. CRC 
177B3C4F,  89001472 out, 80946035 in. ok
   files.tar.lz: dict   32 MiB,  1.473:1, 67.88% ratio, 32.12% saved. CRC 
37D5EEB6,  67553280 out, 45854099 in. ok
   files.tar.lz: dict   32 MiB,  1.755:1, 56.99% ratio, 43.01% saved. CRC 
937F7099,  65571840 out, 37372377 in. ok
   files.tar.lz: dict   32 MiB,  0.998:1, 100.20% ratio, -0.20% saved. CRC 
D8D5FFD8,  67117056 out, 67252004 in. ok
   files.tar.lz: dict   32 MiB, 44.039:1,  2.27% ratio, 97.73% saved. CRC 
7E5629BD,  67116032 out,  1524000 in. ok
   files.tar.lz: dict   32 MiB,  1.726:1, 57.92% ratio, 42.08% saved. CRC 
94F7AA52,  69840896 out, 40453016 in. ok
   files.tar.lz: dict   32 MiB,  1.254:1, 79.73% ratio, 20.27% saved. CRC 
7CD97C5E,  66978304 out, 53402326 in. ok
   files.tar.lz: dict   32 MiB,  1.424:1, 70.21% ratio, 29.79% saved. CRC 
261712D3,  67108864 out, 47114950 in. ok
   files.tar.lz: dict   20 MiB,  1.571:1, 63.66% ratio, 36.34% saved. CRC 
0256CCC7,  19752960 out, 12575533 in. ok
   files.tar.lz: dict   32 MiB,  1.055:1, 94.78% ratio,  5.22% saved. CRC 
E6D8309F, 562729984 out, 533335546 in. ok
   files.tar.lz: dict   32 MiB,  1.236:1, 80.91% ratio, 19.09% saved. CRC 
1C66D6C4,  62746112 out, 50765045 in. ok
   files.tar.lz: dict   32 MiB,  1.825:1, 54.79% ratio, 45.21% saved. CRC 
BB6BC655,  67108352 out, 36765408 in. ok
   files.tar.lz: dict   32 MiB,  1.256:1, 79.61% ratio, 20.39% saved. CRC 
700E4C9C,  84463104 out, 67237671 in. ok
   files.tar.lz: dict 1408 KiB,  4.658:1, 21.47% ratio, 78.53% saved. CRC 
295576E3,   1430016 out,   307011 in. ok
   files.tar.lz: dict    4 KiB, 23.273:1,  4.30% ratio, 95.70% saved. CRC 
EFB5AF2E,      1024 out,       44 in. ok

So no issues. So I can only assume it is a hardware issue rather than bug with 
tarlz.

One minor curiosity, though I assume it is probably just to do with the 
threading model (?): creating the archive with the -v flag so it shows the 
files being added, stopped producing any console output after a while, until 
the process successfully completed. This doesn't really matter, but it is nice 
to be able to watch the files being added. I can of course watch the progress 
by monitoring the growing size of the new archive on the filesystem.

> But remember that it does not matter how safe is the format if the archive
> is created corrupt because of a bug in the tool or a faulty RAM. Check
> everything twice at creation time.

Thanks for the advice - from now on I'm going to always run the lzip '-tvvvv' 
command on the resultant created archive. I'll probably also just leave it at 
default compression, since the saving was minimal.

I presume I am safe in assuming that if that command completes all ok, and 
every checksum is correct, then the archive integrity CAN be trusted :-)

Best regards

Aren.