Re: spec compliance: header CRC?

[Greg Roelofs]

Paul Eggert wrote:

>>      http://gregroelofs.com/test/testCompressThenConcat.txt.gz

> Thanks, I've verified that the new code works with that example.
> It's a bit much to turn that into a test case.  Perhaps if I
> find time I'll write a smaller one.

I just attached to https://issues.apache.org/jira/browse/MAPREDUCE-1927
a small zipfile containing a script and some binary bits to generate all
32 non-encrypted gzip header variants.  You can either grab that or find
the results here:

        http://gregroelofs.com/test/all-gz-header-types-20100812.zip

The script is dependent on this CRC checker:

        http://gregroelofs.com/code/check-latest.tgz

(There's probably some standard util available these days, but I didn't
look.)

Oh, and the gzip file above (3409 bytes) was slightly broken; I had
overlooked the part about extra-field subfields, so it was missing the
two ID bytes and two subfield-length bytes.  I just replaced it with a
3413-byte corrected version.  (gzip doesn't look inside it anyway, but
if you want a correct test case, here you go.  All 16 of the new ones
share the same fix, btw.)

I've also tested your 20100703 header-CRC patch (backported to 1.4), and
it works great--thanks again.  Related fixes/suggestions:

 - the "header16 != crc16" error message in gzip.c would look nicer with
   a pair of "%04x" instead of "%x".  (A CPAN Perl module has a zero-the-
   upper-byte bug, and the mismatched one-byte vs. two-byte error messages
   looked weird.)

 - algorithm.doc needs updating:
   - "bit 1 set: continuation of multi-part gzip file" -> "bit 1 set: header
     CRC-16 present"
   - new "? bytes  optional 16-bit header CRC" line immediately after
     "? bytes  optional file comment, zero terminated"
   - should mention spec at http://www.ietf.org/rfc/rfc1952.txt

Also, are you aware of _any_ gzip utilities that support encryption?  The
full, drop-in crypt.c and crypt.h have been available for a decade, but
it appears that the gzip code is explicitly not wired to support it.  Does
anything out there do so?  If not, you might want to add another note to
algorithm.doc mentioning that crypto's not official in either the code
sense or the spec sense.  (Or did an older version of gzip support it in
gzip files?)

Thanks,
  Greg