Re: lynx-dev RFC959 non-compliance in Lynx hangs the client

[Klaus Weide]

On Wed, 8 Sep 1999, David Woolley wrote:
[Gregory:]
> > [ C->S ] RETR /
> 
> This should only actually happen when a semantically broken URL is
> used.  Correctly formed URLs never result in /s being sent to Unix
> servers, as they are always implemented by stepping one directory
> at a time from the login directory.  Forcing a / into the URL is used
> for broken servers or to work round incorrectly specified URLs.  If
> you didn't force in the initial /, then Lynx has a real problem.

1) Afaik Lynx has never conformed to the specs in this area, at
least when talking to UNIX servers.  Note that's the URL specs, not
FTP specs.  There has been discussion about it in the past, which
should be somewhere in the archives.  Like it or not (I don't
particularly), it's documented behavior, see in the "Supported URLs"
under "ftp URL".

2) There is a completely valid way to say "I want /" in a URL, 
using the form <ftp://example.com/%2F>.  Of course all kinds of
clients that supposedly understand URL syntax for FTP resources
turn out to not interpret this correctly...

> Incidentally, the Lynx FTP code is based on a version of the W3C reference
> implementation, so it would probably be advisable to check whether they have
> diverged in the areas that you are considering.  NB a lot of people will
> be using the W3C proxy, built using an old version of the W3C library.

Good catch - I tried it, yes CERN/3.0 has the same problem as lynx
with ftp.wu-ftpd.org.  Only here it also appears for file 'RETR'ieval,
not just for directory listings.  So Gregory can add a third application
to the list of clients that have problems with the changed behavior.

Strace shows the httpd hanging in a read() on the control connection while
the server has already closed the data connection.  So it seems to be
exactly the same kind of problem as with Lynx..  Other differences found:
the httpd does the FTP interpretation according to specs; and it uses PASV
while I didn't test lynx with PASV (which would require compiling with
-DNOPORT, and then afaik PORT is not available - I probably never compiled
lynx that way).

Once on the topic of proxies (or rather, HTTP->FTP gateways), I decided
to also check squid (which I have running as a personal cache locally).
That is squid 2.1.2-1 as packaged by Debian GNU/Linux (not the latest
version).  Watching the TCP traffic, I see squid has partially the same
problem, although it manages to hide the effect from *its* (HTTP) client:
when it detects the close of the FTP data connection, it also closes the
connection to its client (after having sent the data received from the
FTP server), so the browser client doesn't notice anything is wrong.
But the squid process (or thread - it's using threads here, but I don't
know how the work is distributed among them) still does not close the
data connection to the FTP server.  It is waiting for the response on
the control connection before doing that.  After a timeout (3 minutes
in my case - I seems to be controlled by squid's 'read_timeout'
configuration option, which has a default of 15 minutes but I changed it)
squid finally closes both the data and control connections to the FTP
server (apparently at the same time).  This ultimately results in RST
traffic when ftp.wu-ftpd.org's response that it had delayed until now
hits the now closed control connection port.

I also checked that the same pattern does not occur with a "normal" FTP
server, i.e. one that sends the final control connection response as
soon as it is done sending data.

So add this as a fourth client the list.  Users won't directly notice,
but cache operators probably will notice that threads are kept occupied
waiting for a response and sockets unnecessarily kept open.  The changed
WU-FTPD behavior looks more and more like a Bad Idea, for deployment in
the real world.

   Klaus