Re: Tarlz 0.26 'make check' test failure on Debian 12

[Aren Tyr]

Hi Antonio/wrotycz

Re: memory errors, I've installed 'memtester', so I'll give that a try to see 
if it produces anything interesting. Given that generally speaking I've never 
had any issues on my PC and have used it for years, any error rate if it occurs 
must be minor/benign, but it does highlight the need for caution particularly 
when backing up precious data into compressed archives, for example. Will be 
interesting, anyway.

On 24/01/2025 00:13, Antonio Diaz Diaz wrote:

> Memory errors happen even in the most expensive hardware. They almost ruined
> the mission of the Mars Science Laboratory[1]. I guess that in a machine
> with 128 GiB of RAM without ECC, a bit flip may happen almost daily[2].
> Maybe you should test your RAM to get an idea of how reliable it is. (And it
> still could be a bug in tarlz).

I dug into my actual hardware using the tool 'inxi' (command "inxi -m -xxx") 
and the output showed that my Corsair memory is all "synchronous unbuffered 
(unregistered)" UDIMMs, so I presume that means no error correction capability.

In any case, this "false positive" (i.e. no bug, seems that way) has been 
enormously useful/beneficial: I've now modified my script to make it much more 
robust as a result, whereas had nothing wrong ever happened, I'd have likely 
blithely (and incorrectly) just relied on occasional listing and spot-checking 
and assumed everything was perfect. So it now does the following steps:

1. Compresses the tarlz archive with maximum compression.
2. Immediately runs tarlx -df <archive> to diff it against all the original 
files.
3. runs lzip -tv <archive> to further verify the integrity of the archive.

If step (2) runs with no resultant output (i.e. no diff's found), and step (3) 
results in verifying as 'ok', I take that to be a pretty strong indication that 
the data has been correctly/safely archived.

---

Thanks for the help/insight. One suggestion I have is you might wish to 
append/add a small section into the tarlz documentation as a quick checklist 
for "verifying your archive integrity", or "recommended best practices for very 
important data", perhaps even with an short example sh/bash script showing how 
the tarlz -df and lzip -tv can be used to verify an archive.

Since the primary use-case for this tool is to generate extremely robust 
archives that have both error checking and a level of recoverability (plus 
extremely fast decompression, etc.), a few pointers/tips/examples in the 
documentation here might nudge potential users in the right direction and also 
highlight the real unique proposition this tool presents versus more 
conventional/established archive/compression candidates that are currently 
used. All of this might help to increase adoption of this excellent format. 
Anyway, I'll be using tarlz for all of my long term archive needs going forward.

Best regards

Aren.