Re: [Lynx-dev] Problem with GZIP and HTTP chunkies

[Thomas Dickey]

On Fri, 2 Jul 2004, Giuseppe Scrivano wrote:

> Thomas Dickey wrote:
> >On Thu, 1 Jul 2004, Giuseppe Scrivano wrote:
> >>Hi,
> >>It seems that lynx doesn't like HTTP chunkies when the data is
> >>compressed with GZIP.  This feature is supported well on other browsers
> >>and the absence of this in lynx cause problems with web servers that
> >>send GZIP data in HTTP chunkies.
> >Do you have a test-URL?
> Hi,
> Really I am the project administrator for the web server project
> MyServer(http://www.myserverproject.net) and abilitating the GZIP
> compression in it I am unable to see pages using lynx while it works
> well with other browsers(tested on Mozilla, Firefox, Internet Explorer,
> Netscape). It seems so that the problem is in lynx, as I can see in the
> GZIP temporary file HTTP chunks data too.

I've done some reading to see what's going on.  Now that my attention is
on it, I seem to recall some discussion of chunked on the mailing list a
few months ago (but I don't see something like that in my email - am not
sure).

However, in this case I believe it's a bug in myserver.  According to RFC
2068 (end of section 3.6):

   All HTTP/1.1 applications MUST be able to receive and decode the
   "chunked" transfer coding, and MUST ignore transfer coding extensions
   they do not understand. A server which receives an entity-body with a
   transfer-coding it does not understand SHOULD return 501
   (Unimplemented), and close the connection. A server MUST NOT send
   transfer-codings to an HTTP/1.0 client.

Lynx is making a GET HTTP/1.0, which makes it fall into that category.
Comparing with apache, I see some relevant discussion (comments in
http_protocol.c):

    /* The following convoluted conditional determines whether or not
     * the current connection should remain persistent after this response
     * (a.k.a. HTTP Keep-Alive) and whether or not the output message
     * body should use the HTTP/1.1 chunked transfer-coding.  In English,
     *
     *   IF  we have not marked this connection as errored;
     *   and the response body has a defined length due to the status code
     *       being 304 or 204, the request method being HEAD, already
     *       having defined Content-Length or Transfer-Encoding: chunked, or
     *       the request version being HTTP/1.1 and thus capable of being set
     *       as chunked [we know the (r->chunked = 1) side-effect is ugly];
     *   and the server configuration enables keep-alive;
     *   and the server configuration has a reasonable inter-request timeout;
     *   and there is no maximum # requests or the max hasn't been reached;
     *   and the response status does not require a close;
     *   and the response generator has not already indicated close;
     *   and the client did not request non-persistence (Connection: close);
     *   and    we haven't been configured to ignore the buggy twit
     *       or they're a buggy twit coming through a HTTP/1.1 proxy
     *   and    the client is requesting an HTTP/1.0-style keep-alive
     *       or the client claims to be HTTP/1.1 compliant (perhaps a proxy);
     *   THEN we can be persistent, which requires more headers be output.
     *
     * Note that the condition evaluation order is extremely important.

and compare with the request by lynx:

GET / HTTP/1.0\r
Host: xxxxxxxxxxxxxxxxxxxxxxxxx\r
Accept: text/html, text/plain, text/sgml, */*;q=0.01\r
Accept-Encoding: gzip, compress, bzip2\r
Accept-Language: en\r
User-Agent: Lynx/2.8.6dev.4 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.7d\r

It would be interesting to implement the HTTP 1.1 features, but I think
that is a different aspect - some further work must be done on myserver...

-- 
Thomas E. Dickey
http://invisible-island.net
ftp://invisible-island.net