Re: lynx-dev Web robots

[David Woolley]

>    Please does anyone know about web robots and how to write/implement
> them?

Lynx can act as a web robot.

You need to be aware that many sites have found the web robot features in
MSIE 4 and third party products to be highly undesirable, by hitting large
numbers of pages, costing them bandwidth, that are never then read.
A number of sites are taking counter measures, although several crawlers
take counter counter measures by failing to provide a distinguishable user
agent string.  Lynx could be at risk of being blacklist if it were implicated
in significant numbers of crawling incidents with its standard user agent
string.