Re: lynx-dev http://wire.ap.org

[Klaus Weide]

On Thu, 19 Nov 1998, Philip Webb wrote:

> 981118 Marcus wrote: 
> > the associated press site use to work with lynx 2.7.2 and 2.7.1
> > but doesn't work with 2.8 why is this?
> > is it because lynx 2.8 doesn't support refuring servers anymore?  
>  
> no: the problem seems to be that  wire.ap.org/  is largely Javascript;
> if you add  APnews/main.html , you get what you probably want.
> 
> note to lynx-dev generally: entering the former URL got me the JS page,
> but when i entered  \  then  \  again to recover the rendered page,
> i got a page consisting of 2 frame URLs, incl the latter above,
> whence i was able to goto some real information.
> i've never before encountered a situation
> in which 2  \'s  take me to a different rendered page.

When you first goto the page, Lynx doesn't send a Referer: header.
After that, when you do '\' (SOURCE), then it does.  Check your -trace log
to verify.

The server is sending a different page depending on that.  In other words
it does negotiation on the Referer header.  Of course it also claims to be
HTTP/1.1 compliant, but doesn't have a "Vary: referer" anywhere to
indicate this, nor does it make the page explicitly uncacheable.  (Not
that lynx would currently care about the vary, but other clients and
caches would.)

If you were accessing the page through a cache, the '\' trick wouldn't
work, at least if the cache is configured to cache responses without
Last-Modified and other caching-related headers (but with a valid Date
header.)  Note that it also wouldn't work if the caching ideas currently
discussed for lynx were implemented.  It is at least questionable whether
lynx should send a Referer header in this situation at all.  Apparently
it doesn't when RELOAD is used.  The behavior may have changed from
previous versions, or the configuration file settings may be different.

These peoples seem to be relatively clueless.  They introduce some
non-interoperable behavior, then then try to blame problems on
"Older versions of the AOL browser".  Using an AOL browser may not
be the only reason why a Refere header is not being sent.

This is from RFC 2068:

   The Referer[sic] request-header field allows the client to specify,
   for the server's benefit, the address (URI) of the resource from
   which the Request-URI was obtained (the "referrer", although the
   header field is misspelled.) The Referer request-header allows a
   server to generate lists of back-links to resources for interest,
   logging, optimized caching, etc. It also allows obsolete or mistyped
   links to be traced for maintenance. The Referer field MUST NOT be
   sent if the Request-URI was obtained from a source that does not have
   its own URI, such as input from the user keyboard.

   [ ... ]

     Note: Because the source of a link may be private information or
     may reveal an otherwise private information source, it is strongly
     recommended that the user be able to select whether or not the
     Referer field is sent. For example, a browser client could have a
     toggle switch for browsing openly/anonymously, which would
     respectively enable/disable the sending of Referer and From
     information.

See also additional remarks in the section "Security Considerations".


    Klaus