Re: [Bug-wget] GSoC project proposals, speed up

[Tim Ruehsen]

On Wednesday 11 March 2015 01:05:38 Darshit Shah wrote:
> > *TCP Fast Open*
> > 
> > On the other hand, TFO has a wider application; of course it lives in a
> > lower level. TFO allows servers to start sending their responses directly
> > after the SYN/ACK message, without waiting for the third handshake. It is
> > based on the exchange of a secure token/cookie during the first connection
> > and saving one RTT per request after.
> > 
> > Particularly for HTTP requests, with short data flows, the overall impact
> > can be very high (the RFC estimates it up to 40%, which sounds like
> > forcing
> > a bit too much the best case). Taking some measures after the
> > implementation to verify it will complement the project nicely.
> > 
> > Linux contains the full implementation of TFO and, since 3.13, it is
> > enabled by default. The rest of most common OS don't support it (Windows,
> > Mac OS); but others are considering it (FreeBSD), maybe by summer...
> > 
> > For this task, I see I should introduce the MSG_FASTOPEN flag to the
> > calls,
> > moving from connect() to sendmsg()/sendto().Should this become a default
> > or
> > should it be configurable? It sounds like the kind of thing that could
> > leave in your .wgetrc, but I honestly don't find any reason to force the
> > conventional TCP. It should just happen automatically if the remote server
> > doesn't support TFO.

Hi Laura,

looks like you already have a good base knowledge.

TFO makes a huge difference on some corner cases. E.g. when you pick up small 
files from different servers (or the server forces one file only per 
connection). Or if you have long RTTs (long distance, congestion, etc.).

While implementing TFO for HTTP (plain text) is straight forward, doing it 
with HTTPS is a bit more challenging. In this case it is up to the underlying 
TLS library to handle it correctly. At least GnuTLS doesn't support it 
currently... so if you want, the project leads you into new terrain. But this 
is optional.

> I think we would like TFO to be the default option when it is
> supported by the server. The first thing Wget needs to do is identify
> if the server supports TFO and then calibrate the remaining requests
> accordingly. You're right though, there's no reason to force
> conventional TCP if both the ends support TFO. You must also try to
> account for the situation where Wget is interacting with a proxy that
> supports TFO, but the actual end of the connection doesn't. These are
> corner cases and shouldn't be your major focus right now, but it helps
> to keep them in mind.

If the server does not support TFO, nothing has to be done. The fallback is 
done transparently / automatically (within the kernel).

An option to switch TFO on/off is nice. People want to make speed comparisons, 
e.g. for academical purposes or just to see if their server (or their client 
kernel) is correctly configured.

AFAICS proxies are not the problem. torify/torsocks has been a problem, it was 
not ready to handle sendto. The effect was a direct connection, circumventing 
the tor network (without letting the user know)... I wrote a patch some weeks 
ago and has been accepted.

Tim

signature.asc
Description: This is a digitally signed message part.