Re: lynx-dev Lynx Referer: problem

[David Woolley]

> 
> > The problem is Lynx doesn't send a Referer header at all.
> > As a webmaster, it's useful for me to know
> > what search terms people are using to get to my website.

This is  a sensitive area and there are proxy servers specifically
designed to break Referer headers because they provide information to site
owner that is not obvious to the user.  (See http://www.junkbuster.com/,
for the obvious example, but squid, the most popular free caching proxy,
also has an anonymising option.) I think some GUI browsers allow Referer
to be disabled for these reasons; lynx does.

Although it can be used to prevent external links to anything except your
home page (which in itself reduces the web like nature of the web), or
as a rather poor defense against the obvious attack on security
by obscurity logon forms (i.e.  when you login you get a link to an
internal page), both of these can be easily bypassed with only trivial
programming skills.

Where it really worries people is that it can be used for creating click
trails, both within a site and across sites.  As such, I believe it is
considered by many to be a feature that ceased to be desirable once the 
net was commercialised.  Generally, although commercial site owners want
to collect market research data, many users don't want them to do that.
(Some sites may be based on a pure market research business model, in which
case it may be reasonable to block free loaders, but they will still
risk negative publicity as a result.)

One particularly bad effect of link protection is that it requires that
pages be pre-expired, so that caches must always revalidate.  This is OK
if the pages are forms responses and every request is different, but
is very bad for the scaleability of the net if you try to protect static
pages this way.  In particular, the most obvious approach to link protection
naturally produces uncacheable pages - a proper approach requires that
the CGI script (etc.) handle conditional GETs properly.

This last week there have been people in the Balkans and South Asia,
asking on the squid support group how to get extreme levels of cacheing,
presumably because they have poor and expensive connectivity.  If you
consider these unattractive markets, may be you should be honest about it
on your home page, otherwise you are giving the local ISPs the impossible
job of explaining to their customers why your pages are so slow, and/or
encouraging them to take counter counter measures in a sort of web
protection arms race.  Note these requests have been to minimise even
conditional gets, although there was probably some lack of knowledge about
conditional gets involved, and their real problem may be uncacheable pages.

If one of the people on that list gets their way, you will only get the
true Referer from them once in every 150 days for any page they consider
static, in particular images.

If you want general market research, the most friendly approach is to 
frustrate cacheing for short sampling periods and extrapolote.  If you want
to track individual, be aware that most individuals wouldn't want to be
tracked if they knew it was happening.

I must admit, though, that although you will almost never get cookies back
from me, I don't deliberately inhibit Referer.