Re: lynx-dev The traversal limitation

[David Woolley]

> 
> I'm wondering if this idea of lynx not being allowed to access the 
> site and robots.txt is what I encounter in TV Guide's site, tvgen.com. 
> I still have problems with that site, as it continually re-forwards and 
> finally lynx tells me that the redirection limit of 10 URLs has been 
> reached. 

A deliberate forwarding loop would be a very aggressive response to 
unwelcome robots, and not as effective as generating a very large page
with links back to the same page with varying data after the script name.

Forwarding loops are almost always the results of broken server configurations,
and the only one I've had was on microsoft.com, when I was refusing cookies
manually in IE4.

Lynx traversal probably is not that common that many sites would bother
to block Lynx as a result, although I know that the policy of IMDB is to
actively block unfriendly robots and it would not be difficult for
someone to provoke such countermeasures.

I'd expect a 403, or other 4xx response, not a redirect, although I 
suppose a redirect loop could happen if someone got carried away with
customised error pages.