[Gcl-devel] RE: Better tmpfile code for gzipped streams

[Mike Thomas]

Hi Camm.

| Hi Mike!  I've used zlib before, and it would be easy to add in this
| context.  But isn't it even less likely that the avreage Windows user
| would have this lib on their system?

.....

| This particular code only comes in when the unaltered filename cannot
| be found, the .gz appended filename can, and compressed files are
| allowed.  I.e. any given port can disable this by setting the right
| variable.  Perhaps this should appear in the error message.
|
| Is it a bad assumption that if users have .gz file available, that
| zcat would also be available?
|
| Anyway, thoughts appreciated.

My line of attack would have been to link statically on Windows.

Certainly the dll is not a standard part of Windows.  Zcat is something
which comes with Cygwin but not, for example, in MSYS.  Cygwin's usually not
available on typical Windows machines and even if it is, can't be guaranteed
to be in the system path (otherwise different sets of dev tools interfere
with one another - my machine at work has three different sets of Unix
emulation build systems for example).  WinZip opens "tar.gz" and many other
archive formats using internal routines so there is no need for zcat and
friends usually.

I don't see much need for the .gz thing on Windows anyway, but I suppose GCL
is used in ways and places I can't even imagine.


| I'm not sure on the pros
| vs. cons of another lib dependency vs. external program dependency.
| We use objdump in (disassemble, for example.

My opinion is that as objdump comes with MinGW32 gcc it is OK to use it.  If
you don't have gcc you can't compile anything from GCL anyway.

Sorry to be butting in with opinions without much code cutting to back it up
at the moment!!

Cheers

Mike Thomas.

|
| Take care,
|
| "Mike Thomas" <address@hidden> writes:
|
| > Hi Camm.
| >
| > You've been very busy lately!  I built 2.6.1 for the first time
| yesterday
| > and doubt I'll get back to it this week.
| >
| > To your question, zcat isn't an 'everyday' Windows program, but
| people can
| > download a copy off the internet.
| >
| > I haven't tested the pipe mechanism you've outlined on Windows
| but the calls
| > seem OK on the face of it.
| >
| > Have you considered the free library, zlib, use of which would keep
| > end-users independent of stray executables?  From the manual:
| >
| > http://www.gzip.org/zlib/manual.html
| >
| >
| >
| ==================================================================
| ==========
| > ====
| >
| > The 'zlib' compression library provides in-memory compression and
| > decompression functions, including integrity checks of the uncompressed
| > data. This version of the library supports only one compression method
| > (deflation) but other algorithms will be added later and will
| have the same
| > stream interface.
| > Compression can be done in a single step if the buffers are large enough
| > (for example if an input file is mmap'ed), or can be done by
| repeated calls
| > of the compression function. In the latter case, the application must
| > provide more input and/or consume the output (providing more
| output space)
| > before each call.
| >
| > The library also supports reading and writing files in gzip (.gz) format
| > with an interface similar to that of stdio.
| >
| > The library does not install any signal handler. The decoder checks the
| > consistency of the compressed data, so the library should never
| crash even
| > in case of corrupted input.
| >
| > ...
| >
| > int compress (Bytef *dest, uLongf *destLen, const Bytef *source, uLong
| > sourceLen);
| > int compress2 (Bytef *dest, uLongf *destLen, const Bytef *source, uLong
| > sourceLen, int level);
| > int uncompress (Bytef *dest, uLongf *destLen, const Bytef *source, uLong
| > sourceLen);
| > typedef voidp gzFile;
| > gzFile gzopen (const char *path, const char *mode);
| > gzFile gzdopen (int fd, const char *mode);
| > int gzsetparams (gzFile file, int level, int strategy);
| > int gzread (gzFile file, voidp buf, unsigned len);
| > int gzwrite (gzFile file, const voidp buf, unsigned len);
| > int VA gzprintf (gzFile file, const char *format, ...);
| > int gzputs (gzFile file, const char *s);
| > char * gzgets (gzFile file, char *buf, int len);
| > int gzputc (gzFile file, int c);
| > int gzgetc (gzFile file);
| > int gzflush (gzFile file, int flush);
| > z_off_t gzseek (gzFile file, z_off_t offset, int whence);
| > z_off_t gztell (gzFile file);
| > int gzrewind (gzFile file);
| > int gzeof (gzFile file);
| > int gzclose (gzFile file);
| > const char * gzerror (gzFile file, int *errnum);
| >
| >
| >
| ==================================================================
| ==========
| > ====
| >
| > An example from the source is included below for your edification.
| >
| > Cheers
| >
| > Mike Thomas.
| >
| >
| ==================================================================
| ==========
| > ====
| >
| > /* minigzip.c -- simulate gzip using the zlib compression library
| >  * Copyright (C) 1995-2002 Jean-loup Gailly.
| >  * For conditions of distribution and use, see copyright notice
| in zlib.h
| >  */
| >
| > /*
| >  * minigzip is a minimal implementation of the gzip utility. This is
| >  * only an example of using zlib and isn't meant to replace the
| >  * full-featured gzip. No attempt is made to deal with file systems
| >  * limiting names to 14 or 8+3 characters, etc... Error checking is
| >  * very limited. So use minigzip only for testing; use gzip for the
| >  * real thing. On MSDOS, use only on file names without extension
| >  * or in pipe mode.
| >  */
| >
| > /* @(#) $Id$ */
| >
| > #include <stdio.h>
| > #include "zlib.h"
| >
| > #ifdef STDC
| > #  include <string.h>
| > #  include <stdlib.h>
| > #else
| >    extern void exit  OF((int));
| > #endif
| >
| > #ifdef USE_MMAP
| > #  include <sys/types.h>
| > #  include <sys/mman.h>
| > #  include <sys/stat.h>
| > #endif
| >
| > #if defined(MSDOS) || defined(OS2) || defined(WIN32)
| > #  include <fcntl.h>
| > #  include <io.h>
| > #  define SET_BINARY_MODE(file) setmode(fileno(file), O_BINARY)
| > #else
| > #  define SET_BINARY_MODE(file)
| > #endif
| >
| > #ifdef VMS
| > #  define unlink delete
| > #  define GZ_SUFFIX "-gz"
| > #endif
| > #ifdef RISCOS
| > #  define unlink remove
| > #  define GZ_SUFFIX "-gz"
| > #  define fileno(file) file->__file
| > #endif
| > #if defined(__MWERKS__) && __dest_os != __be_os && __dest_os !=
| __win32_os
| > #  include <unix.h> /* for fileno */
| > #endif
| >
| > #ifndef WIN32 /* unlink already in stdio.h for WIN32 */
| >   extern int unlink OF((const char *));
| > #endif
| >
| > #ifndef GZ_SUFFIX
| > #  define GZ_SUFFIX ".gz"
| > #endif
| > #define SUFFIX_LEN (sizeof(GZ_SUFFIX)-1)
| >
| > #define BUFLEN      16384
| > #define MAX_NAME_LEN 1024
| >
| > #ifdef MAXSEG_64K
| > #  define local static
| >    /* Needed for systems with limitation on stack size. */
| > #else
| > #  define local
| > #endif
| >
| > char *prog;
| >
| > void error            OF((const char *msg));
| > void gz_compress      OF((FILE   *in, gzFile out));
| > #ifdef USE_MMAP
| > int  gz_compress_mmap OF((FILE   *in, gzFile out));
| > #endif
| > void gz_uncompress    OF((gzFile in, FILE   *out));
| > void file_compress    OF((char  *file, char *mode));
| > void file_uncompress  OF((char  *file));
| > int  main             OF((int argc, char *argv[]));
| >
| > /*
| >
| ==================================================================
| =========
| >  * Display error message and exit
| >  */
| > void error(msg)
| >     const char *msg;
| > {
| >     fprintf(stderr, "%s: %s\n", prog, msg);
| >     exit(1);
| > }
| >
| > /*
| >
| ==================================================================
| =========
| >  * Compress input to output then close both files.
| >  */
| >
| > void gz_compress(in, out)
| >     FILE   *in;
| >     gzFile out;
| > {
| >     local char buf[BUFLEN];
| >     int len;
| >     int err;
| >
| > #ifdef USE_MMAP
| >     /* Try first compressing with mmap. If mmap fails (minigzip
| used in a
| >      * pipe), use the normal fread loop.
| >      */
| >     if (gz_compress_mmap(in, out) == Z_OK) return;
| > #endif
| >     for (;;) {
| >         len = fread(buf, 1, sizeof(buf), in);
| >         if (ferror(in)) {
| >             perror("fread");
| >             exit(1);
| >         }
| >         if (len == 0) break;
| >
| >         if (gzwrite(out, buf, (unsigned)len) != len) error(gzerror(out,
| > &err));
| >     }
| >     fclose(in);
| >     if (gzclose(out) != Z_OK) error("failed gzclose");
| > }
| >
| > #ifdef USE_MMAP /* MMAP version, Miguel Albrecht <address@hidden> */
| >
| > /* Try compressing the input file at once using mmap. Return Z_OK if
| >  * if success, Z_ERRNO otherwise.
| >  */
| > int gz_compress_mmap(in, out)
| >     FILE   *in;
| >     gzFile out;
| > {
| >     int len;
| >     int err;
| >     int ifd = fileno(in);
| >     caddr_t buf;    /* mmap'ed buffer for the entire input file */
| >     off_t buf_len;  /* length of the input file */
| >     struct stat sb;
| >
| >     /* Determine the size of the file, needed for mmap: */
| >     if (fstat(ifd, &sb) < 0) return Z_ERRNO;
| >     buf_len = sb.st_size;
| >     if (buf_len <= 0) return Z_ERRNO;
| >
| >     /* Now do the actual mmap: */
| >     buf = mmap((caddr_t) 0, buf_len, PROT_READ, MAP_SHARED,
| ifd, (off_t)0);
| >     if (buf == (caddr_t)(-1)) return Z_ERRNO;
| >
| >     /* Compress the whole file at once: */
| >     len = gzwrite(out, (char *)buf, (unsigned)buf_len);
| >
| >     if (len != (int)buf_len) error(gzerror(out, &err));
| >
| >     munmap(buf, buf_len);
| >     fclose(in);
| >     if (gzclose(out) != Z_OK) error("failed gzclose");
| >     return Z_OK;
| > }
| > #endif /* USE_MMAP */
| >
| > /*
| >
| ==================================================================
| =========
| >  * Uncompress input to output then close both files.
| >  */
| > void gz_uncompress(in, out)
| >     gzFile in;
| >     FILE   *out;
| > {
| >     local char buf[BUFLEN];
| >     int len;
| >     int err;
| >
| >     for (;;) {
| >         len = gzread(in, buf, sizeof(buf));
| >         if (len < 0) error (gzerror(in, &err));
| >         if (len == 0) break;
| >
| >         if ((int)fwrite(buf, 1, (unsigned)len, out) != len) {
| >         error("failed fwrite");
| >     }
| >     }
| >     if (fclose(out)) error("failed fclose");
| >
| >     if (gzclose(in) != Z_OK) error("failed gzclose");
| > }
| >
| >
| > /*
| >
| ==================================================================
| =========
| >  * Compress the given file: create a corresponding .gz file and
| remove the
| >  * original.
| >  */
| > void file_compress(file, mode)
| >     char  *file;
| >     char  *mode;
| > {
| >     local char outfile[MAX_NAME_LEN];
| >     FILE  *in;
| >     gzFile out;
| >
| >     strcpy(outfile, file);
| >     strcat(outfile, GZ_SUFFIX);
| >
| >     in = fopen(file, "rb");
| >     if (in == NULL) {
| >         perror(file);
| >         exit(1);
| >     }
| >     out = gzopen(outfile, mode);
| >     if (out == NULL) {
| >         fprintf(stderr, "%s: can't gzopen %s\n", prog, outfile);
| >         exit(1);
| >     }
| >     gz_compress(in, out);
| >
| >     unlink(file);
| > }
| >
| >
| > /*
| >
| ==================================================================
| =========
| >  * Uncompress the given file and remove the original.
| >  */
| > void file_uncompress(file)
| >     char  *file;
| > {
| >     local char buf[MAX_NAME_LEN];
| >     char *infile, *outfile;
| >     FILE  *out;
| >     gzFile in;
| >     int len = strlen(file);
| >
| >     strcpy(buf, file);
| >
| >     if (len > SUFFIX_LEN && strcmp(file+len-SUFFIX_LEN,
| GZ_SUFFIX) == 0) {
| >         infile = file;
| >         outfile = buf;
| >         outfile[len-3] = '\0';
| >     } else {
| >         outfile = file;
| >         infile = buf;
| >         strcat(infile, GZ_SUFFIX);
| >     }
| >     in = gzopen(infile, "rb");
| >     if (in == NULL) {
| >         fprintf(stderr, "%s: can't gzopen %s\n", prog, infile);
| >         exit(1);
| >     }
| >     out = fopen(outfile, "wb");
| >     if (out == NULL) {
| >         perror(file);
| >         exit(1);
| >     }
| >
| >     gz_uncompress(in, out);
| >
| >     unlink(infile);
| > }
| >
| >
| > /*
| >
| ==================================================================
| =========
| >  * Usage:  minigzip [-d] [-f] [-h] [-1 to -9] [files...]
| >  *   -d : decompress
| >  *   -f : compress with Z_FILTERED
| >  *   -h : compress with Z_HUFFMAN_ONLY
| >  *   -1 to -9 : compression level
| >  */
| >
| > int main(argc, argv)
| >     int argc;
| >     char *argv[];
| > {
| >     int uncompr = 0;
| >     gzFile file;
| >     char outmode[20];
| >
| >     strcpy(outmode, "wb6 ");
| >
| >     prog = argv[0];
| >     argc--, argv++;
| >
| >     while (argc > 0) {
| >       if (strcmp(*argv, "-d") == 0)
| >     uncompr = 1;
| >       else if (strcmp(*argv, "-f") == 0)
| >     outmode[3] = 'f';
| >       else if (strcmp(*argv, "-h") == 0)
| >     outmode[3] = 'h';
| >       else if ((*argv)[0] == '-' && (*argv)[1] >= '1' &&
| (*argv)[1] <= '9'
| > &&
| >            (*argv)[2] == 0)
| >     outmode[2] = (*argv)[1];
| >       else
| >     break;
| >       argc--, argv++;
| >     }
| >     if (argc == 0) {
| >         SET_BINARY_MODE(stdin);
| >         SET_BINARY_MODE(stdout);
| >         if (uncompr) {
| >             file = gzdopen(fileno(stdin), "rb");
| >             if (file == NULL) error("can't gzdopen stdin");
| >             gz_uncompress(file, stdout);
| >         } else {
| >             file = gzdopen(fileno(stdout), outmode);
| >             if (file == NULL) error("can't gzdopen stdout");
| >             gz_compress(stdin, file);
| >         }
| >     } else {
| >         do {
| >             if (uncompr) {
| >                 file_uncompress(*argv);
| >             } else {
| >                 file_compress(*argv, outmode);
| >             }
| >         } while (argv++, --argc);
| >     }
| >     exit(0);
| >     return 0; /* to avoid warning */
| > }
| >
| >
| >
| >
| >
|
| --
| Camm Maguire                                          address@hidden
| ==========================================================================
| "The earth is but one country, and mankind its citizens."  --  Baha'u'llah
|
|