Re: Tarlz 0.26 'make check' test failure on Debian 12

[Antonio Diaz Diaz]

Aren Tyr wrote:
> Pleased to hear the feedback is useful.

Yes, thanks. Feedback is especially useful for tarlz, which is the most 
complex and less mature tool in the lzip family. (Note that the version 
number is still below 1.0).

> > I think I'm going to limit the --mtime test to dates since the epoch up to
> > the year 2242 to avoid these incompatibilities with touch.
>
> Sounds like a sensible course of action.

I have already fixed it. The change will appear in the next version. Thanks.

> Anyway, here is where the possible interest lies: using compression level '-9' with 
> everything else as per defaults appears to have generated a faulty archive. In 
> particular, listing the entire contents of the tarlz archive via '-tf' caused it to 
> abort/stop with "data error" with the AppImage.tgz file as the last entry, so I 
> presume that was what it was failing on.

This is in fact a serious problem that needs to be diagnosed ASAP.

> 1. It is "normal" that using maximum compression level '-9' can cause some 
> failures/problems with created archives?

No, it is not normal. I always use level -9 and never had a problem. If 
level -9 has a problem, I'm very interested in fixing it.

Level -9 may run out of memory on some machines, but if it happens it is 
clearly diagnosed.

Level -9 uses more memory than level -6, and is therefore more frequently 
affected by RAM errors. I recommend to repeat the exact command where you 
used level -9 and compare the resulting archive with the faulty one:
- If they are identical, it is a bug in tarlz. If this is the case, please, 
send the output of 'lzip -tvvvv'.
- If they are different and the second is correct, it may be a bug (data 
race), or a memory problem.
- If they are different but both are faulty, it may be also a bug (data 
race), or a memory problem. In this case try creating the archive with one 
thread (--threads=1) to discard a data race.

> 2. If not, I can try to get you some more diagnostic information, I presume I 
> can simply use some verbosity switches to get tarlz to be more informative 
> about the underlying issue.

You may try to diagnose the exact defect in the faulty archive with lzip 
because it gives the most precise diagnostics:
  lzip -tvvvv large_archive.tar.lz

The most important thing is to find out if the problem is in tarlz or in the 
machine (RAM).

> 3. I presume the fact that the archive lists all files without apparent issue 
> with default compression settings means its integrity can be trusted.

'--list' can't be trusted to detect corruption in multimember archives 
unless multi-threading is disabled (--threads=0). See 
http://www.nongnu.org/lzip/manual/tarlz_manual.html#Multi_002dthreaded-decoding

"On the other hand, multi-threaded --list won't detect corruption in the tar 
member data because it only decodes the part of each lzip member 
corresponding to the tar member header."

I usually check tar.lz archives with 'tarlz -d' just after creation, and at 
any later time with 'lzip -tv' or 'plzip -tv' to make sure that all the data 
is checked.

> I am just checking because my intention was to switch over to using tarlz as my 
> default backup format precisely because of the extra integrity/recovery/safety 
> it provides versus a stock tar.xz archive.

Excelent election! :-)

But remember that it does not matter how safe is the format if the archive 
is created corrupt because of a bug in the tool or a faulty RAM. Check 
everything twice at creation time.

Best regards,
Antonio.