Re: 2.8.3dev.8 patch 3 (was: lynx-dev RFC959 non-compliance)

[Gregory A Lundberg]

> Well it seems that wouldnt be an FTP server compliant to RFC 959 then:
> 
>  [ 5.4 ]
>       The table below lists alternative success and failure replies for
>       each command.  These must be strictly adhered to; a server may
>       substitute text in the replies, but the meaning and action implied
>       by the code numbers and by the specific command reply sequence
>       cannot be altered.

RFC 1123 ammends RFC 959:

    4.1.2.11  FTP Replies: RFC-959 Section 4.2, Page 35

            A Server-FTP MUST send only correctly formatted replies on
            the control connection.  Note that RFC-959 (unlike earlier
            versions of the FTP spec) contains no provision for a
            "spontaneous" reply message.

            A Server-FTP SHOULD use the reply codes defined in RFC-959
            whenever they apply.  However, a server-FTP MAY use a
            different reply code when needed, as long as the general
            rules of Section 4.2 are followed. When the implementor has
            a choice between a 4xx and 5xx reply code, a Server-FTP
            SHOULD send a 4xx (temporary failure) code when there is any
            reasonable possibility that a failed FTP will succeed a few
            hours later.

            A User-FTP SHOULD generally use only the highest-order digit
            of a 3-digit reply code for making a procedural decision, to
            prevent difficulties when a Server-FTP uses non-standard
            reply codes.

            A User-FTP MUST be able to handle multi-line replies.  If
            the implementation imposes a limit on the number of lines
            and if this limit is exceeded, the User-FTP MUST recover,
            e.g., by ignoring the excess lines until the end of the
            multi-line reply is reached.

            A User-FTP SHOULD NOT interpret a 421 reply code ("Service
            not available, closing control connection") specially, but
            SHOULD detect closing of the control connection by the
            server.

            DISCUSSION:
                 Server implementations that fail to strictly follow the
                 reply rules often cause FTP user programs to hang.
                 Note that RFC-959 resolved ambiguities in the reply
                 rules found in earlier FTP specifications and must be
                 followed.

                 It is important to choose FTP reply codes that properly
                 distinguish between temporary and permanent failures,
                 to allow the successful use of file transfer client
                 daemons.  These programs depend on the reply codes to
                 decide whether or not to retry a failed transfer; using
                 a permanent failure code (5xx) for a temporary error
                 will cause these programs to give up unnecessarily.

                 When the meaning of a reply matches exactly the text
                 shown in RFC-959, uniformity will be enhanced by using
                 the RFC-959 text verbatim.  However, a Server-FTP
                 implementor is encouraged to choose reply text that
                 conveys specific system-dependent information, when
                 appropriate.

> >   In addition, a well-written client SHOULD be able to process any
> >   unexpected response.  For example, RFC959 does not specify that a 5xx
> >   code might be sent at the end of a data transfer.  If a server chose
> >   too, however, the client should recognize the error as a permanent
> >   failure and terminate processing the current command sequence.
> > 
> >   The RFC does not lay out how to handle [6-9]yz responses.  MY
> >   reccomendation, based upon usage in FTPSEC and other RFCs, is to
> >   treat these the same as 2yx .. final positive response to the
> >   previous command.
> 
> Do you have some specific pointers?  Treating arbitrary [6-9]yz as 2xy
> doesn't sound like a safe thing (without knowing anything about how they
> _are_ being used by protocol extensions).  The alternative (below) seems
> like the safe choice.

The intention is that a client could be used to carry out a conversation
using commands and replies for which it has no specific knowledge.  Some
controlling entity would know the commands and interpret the replies.

> >   It would ve valid for a client, however, to treat these as non-protocol
> >   replies and immedeately terminate the entire session as not being FTP.

Such a client would probably not be usable for any extended FTP features.

Given the nature of web browsers, which are intended solely for human use
and are rarely, if ever, usable by automata, walking away from the entire
problam is a valid choice.

Or, at least, this was the case.  If we consider the effect of client-side
Java scripts, continueing the FTP conversation may indeed be the correct
choice.

> > There is a comment there that Lynx is making an invalid assumption about
> > the way the PORT command works.
> 
> Yes, I see what you mean...  Do you kown of any real-life servers that
> are strictly compliant in this respect?
> 
> It sounds like a pain to deal with.  I bet you woud discover more than
> two clients that have problems with that...

The server would have to look at the data connection.  If anything had been
sent, it would have to close the connection to ensure a clean flush.  If a
client cannot handle that, there is nothing the server can do.

> > The traced Lynx session attempts a PORT/RETR/CWD/LIST sequence.  This
> > should be PORT/RETR/CWD/PORT/LIST since the server SHOULD open the data
> > connection immedeately upon receipt of PORT, and SHOULD close that
> > connection after any transfer command, regardless of the success or failure
> > of that command.
> 
> Shouldn't we have to accept() the first connection too?  (Just to drop
> it immediately, I guess.) What if we don't know whether the server behaves
> "normally" or "strictly".  (No I haven't thought this through.)

Issuing the PORT command advises the server you are _now_ ready to set up a
data connection on the specified PORT.

If I were writing a client, I'd use the follow pseudo code:

  Issue PORT
    At any point after issuing the PORT, the server may attempt to connect
    to the specified port.

  do ; Await reply from server ; while 1yz
    If 4yz, tell the user to try again in a few hours.  This command
      sequence has failed, clean up and go back to ask the user what next.
    If 5yz, tell the user this will never work.  This command sequence has
      failed, clean up and go back to ask the user what next.

  Has the data connection been established?
    No, warn user the data transfer is easily spoofed
    Yes, is the remote end the expected IP address?
      No, warn the user the data transfer is probably spoofed
    If a warning was issued, await instructions from the user
      If the desire is to abort the transfer, clean up and go back to ask
      the user what next.

  Issue transfer command

  Await reply from server
    If 1yz
      Has the data connection been estalbished?
        No, await the data connection
        Is the remote end the expected IP address?
          No, warn the user the data transfer is probably spoofed and ask
            for instruction.  If the desire is to abort, jump off into the
            ABOR function's logic, which I'll not cover here.
      Read/write until EOF on the input file/connection.
      Close data connection.
      do ; await reply from server; ; while 1yz
    If 4yz, tell the user to try again in a few hours.  This command
      sequence has failed, clean up and go back to ask the user what next.
    If 5yz, tell the user this will never work.  This command sequence has
      failed, clean up and go back to ask the user what next.



For a PASV session:

  Issue PASV

  do ; await reply from server ; while 1yz
    If 4yz, tell the user to try again in a few hours.  This command
      sequence has failed, clean up and go back to ask the user what next.
    If not 2yz, tell the user this will never work.  This command sequence
      has failed, clean up and go back to ask the user what next.
    Scan reply for IP/port numbers

[ You will note PASV is a special beast.  It is one of the very few FTP
  replies the client *must* parse.  As such, it makes more sense that the
  presumption for unknown replies to PASV is failure rather than success.
  Otherwise the client would be attempting to parse replies it has no
  reason to expect would have the required information. ]

  Establish data connection to server.
    If error, tell the user the data connection failed, and why.  Clean up
      and go back for another command from the user.

  Issue transfer command

  Await reply from server
    If 1yz
      Read/write until EOF on the input file/connection.
      Close data connection.
      do ; await reply from server; ; while 1yz
    If 4yz, tell the user to try again in a few hours.  This command
      sequence has failed, clean up and go back to ask the user what next.
    If 5yz, tell the user this will never work.  This command sequence has
      failed, clean up and go back to ask the user what next.

> [...]
> > If you try against ftp.wu-ftpd.org and see a directory listing, I'd say
> > you're doing well.  If you can actually view a README file, you're fixed.
> 
> It seems for file transfers lynx was already doing the data close
> immediately after the EOF (in the normal successful case at least).
> That probably means my patch won't create any problems, since it
> just does the same for directory requests.

That's what we found with the other client, too.  Slightly different
failure (the other client closed, but last synchronization), but only on
the LIST command.

-- 

Gregory A Lundberg              WU-FTPD Development Group
1441 Elmdale Drive              address@hidden
Kettering, OH 45409-1615 USA    1-800-809-2195

pgpBKqkeKe9mW.pgp
Description: PGP signature