bug#23113: parallel gzip processes trash hard disks, need larger buffers

[Jim Meyering]

On Tue, Apr 12, 2016 at 9:55 AM, Chevreux, Bastien
<address@hidden> wrote:
> Mark,
>
> I knew about pigz, albeit not about -b, thank you for that. Together with -p 
> 1 that would replicate gzip and implement input buffering well enough to be 
> used in parallel pipelines (where you do not want, e.g., 40 pipelines running 
> 40 pigz with 40 threads each).
>
> Questions: how stable / error proof is pigz compared to gzip? I always shied 
> away from it as gzip is so much tried and tested that errors are unlikely ... 
> and the zlib.net homepage does not make an "official" statement like "you 
> should all now move to pigz, it's good and tested enough." Additional 
> question: is there a pigzlib planned? :-)

I expect pigz is stable enough to use with very high confidence.
Paul and I are notoriously picky about such things, and would not be
considering how to deprecate gzip in favor of pigz or to make gzip a
wrapper around pigz if we did not have that level of confidence.

One question for Mark: do you know if pigz has been subjected to AFL's
coverage-adaptive fuzzing? If not, it'd be great if someone could find
the time to do that. If someone does that, please also test an
ASAN-enabled binary and tell us how long the tests ran with no trace
of failure.

For reference, here's what happened when AFL was first applied to
linux file system driver code:
https://events.linuxfoundation.org/sites/events/files/slides/AFL%20filesystem%20fuzzing,%20Vault%202016.pdf.
If you read nothing else, look at slide 3, with its table of file
system type vs. the amount of time each driver withstood AFL-driven
abuse before first failure.

FYI, anyone can close one of these "issues," and I'm doing so simply
by replying to the usual address@hidden address, but with an
inserted "-done" before the "@": address@hidden