Re: LYNX-DEV lost packets - www.altavista.digital.com implicated

[David Woolley]

P Webb wrote:
> 
> 970701 David Woolley wrote:
> > However, I've noticed that, in circumstances where packet loss is probable,
> > Lynx will often hang after sending the HTTP request.

Some further investigation has failed to show a problem with Lynx.
It does however show a problem with www.altavista.digital.com.  They are
using path MTU discovery to try and work out the largest possible
TCP segment size, that can be used without IP fragmentation occuring.
Unfortunately, it would seem that they are failing to detect that their 
MTU is too large, and the don't fragment flag, set as part of MTU discovery,
is preventing the network delivering the packet.

For reasons I've not yet fully resolved, my Linux system is setting an
MTU and presumably MRU of 1524, which is larger than the 1500 that all PPP
implementations must support.  I am getting the SYN frame from altavista,
and the final frame.  the final frame starts at offset 1485 which makes
the total of the previous frames (presumed 1) the absolute maximum
given the IP and TCP minimum overheads of 20 bytes each.  altavista,
themselves, offer a maximum segment size which is huge by the standards
of normal internet MTUs

My guess is that either path MTU discovery is failing to reveal a lower
limit on the path, or something is declaring an MRU that it is unable to
honour.  Note that TCPDUMP on Linux is not reporting the first segment, nor
is it reporting an ICMP for it, so the latest place the frame could be dropped
is in the PPP receive handling.

One possible reason for MTU discovery failure might be that the ICMP
message is being dropped, as low priority, by a congested router.  Another
might be that their software implements path MTU discover, but they are 
running a firewall which blocks the relevant ICMP frames.

It's still possible that other sites are hanging for other reasons.

My workarounds are probably to go through a proxy (it tends to be overloaded,
and gives no benefit for dynamic pages, like those from search engines) or try
forcing a lower MRU.

> however, if it is a weakness in Lynx itself,
> perhaps there should be some discussion on lynx-dev
> & a bit of code manipulation to get Lynx to repeat requests for itself.

The only thing that Lynx could do in this particular case is to request
and explicit maximum segment size; however, this really isn't the right
place to attack the problem.
 
> BTW is there any method of tracking where packets are in fact being routed?
> it would be nice is Lynx could do that, if it can be added easily.

The sending IP implementation would have to add a record route option to
the packets.  As you can't control the this aspect of the sender, it would
require cooperation from both parties.  Record route might also be considered
intrusive by some firewall policies (reveals internal structure).  It might
also confuse the path MTU discovery process.

You can use traceroute type tools, but they probe with protocols other than
TCP, and some routers are set to route these differently; i.e. there are
some places where the result of traceroute will always be different from
the path that real data takes.

(Note I've copied this to the original questioner, but I think it will be
too technical for him, although it might help his ISP.)

MTU maximum transmission unit
MRU maximum receive unit
PPP point to point protocol (re-badged as RAS by Microsoft)
ICMP internet control message protocol, used for error feedback on low level
    internet processes (and also diagnostics)
MSS maximum segment size.

Typical tcpdump of altavista (DF is don't fragment, an indicator that path
MTU discovery is in use - DF without MTU discovery is certainly going
to cause problems with segment sizes above 512, P is push, S is SYN):

July 22nd, timezone +0100
07:45:19.997811 158.152.19.5.11870 > 204.123.2.75.80: S 
3984029018:3984029018(0) win 512 <mss 1484>
07:45:20.327752 204.123.2.75.80 > 158.152.19.5.11870: S 
1884672000:1884672000(0) ack 3984029019 win 34132 <mss 4312> (DF)
07:45:20.327752 158.152.19.5.11870 > 204.123.2.75.80: . ack 1 win 15360 (DF)
07:45:20.507720 158.152.19.5.11870 > 204.123.2.75.80: P 1:282(281) ack 1 win 
15360 (DF)
07:45:21.117612 204.123.2.75.80 > 158.152.19.5.11870: P 1485:2048(563) ack 282 
win 34132 (DF)
07:45:21.117612 158.152.19.5.11870 > 204.123.2.75.80: . ack 1 win 15360 (DF)
07:45:23.847126 158.152.19.5.11870 > 204.123.2.75.80: . ack 1 win 15360 (DF)

PS. The original question was about using hotmail and similar.  Our office ISP
has blocked them on occasions because of excessive spam, and I'm seriously 
considering doing this myself.
;
; To UNSUBSCRIBE:  Send a mail message to address@hidden
;                  with "unsubscribe lynx-dev" (without the
;                  quotation marks) on a line by itself.
;