lynx-dev
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: lynx-dev Changing timeout length


From: David Woolley
Subject: Re: lynx-dev Changing timeout length
Date: Sat, 15 Jan 2000 13:39:13 +0000 (GMT)

> 
> But can it be done? I think that it would be a good thing if one can 
> set up a timeout value so that if one tries to connect to a Internet 
> site that could have moved, or is no longer around, Lynx would 
> timeout after a given number of seconds.

Sites that have moved should produce an ICMP host unreachable or
TCP Reset within a couple of seconds.  The reasons for going to
a full timeout are:

- firewalls which think that ignoring rejected traffic is
more secure than explicitly rejecting it;

- broken firewalls and NAT software which lose host unreachable
messages;

- broken routers;

- extreme network congestion causing massive packet loss.

NB If you are going to use Lynx for this purpose, please obtain written
permission from all the sites involved first, or manually read robots.txt
and any robots meta tags on the pages to ensure permission, then make sure
that you delay at least 30 seconds between every request to any one site,
at least one site doesn't permit this tactic for mirroring pages and insists
on 1 minute for search engine indexers, otherwise they will blacklist and
block the site.  I think the owner of this site is Lynx aware, but he does
simply block whole user agents who abuse these rules and will block IP
addresses that abuse them.

reply via email to

[Prev in Thread] Current Thread [Next in Thread]