lynx-dev
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: LYNX-DEV lynx-dev list status/problems


From: David Woolley
Subject: Re: LYNX-DEV lynx-dev list status/problems
Date: Mon, 14 Jul 1997 19:17:58 +0100 (BST)

> 
> Any sweep that comes across my FAQ gets into the message pages of
> the archive, and could crawl from there to the indices and all
> messages.  Putting a form at the front door doesn't mean that
> there aren't back doors.

Except that I think the archives should be visible to the crawlers from
the search engines, you should be using robots.txt to deter them, and
if you have enough control, you could try an infinitely recursive page,
generated by CGI, to inconvenience anyone who ignores robots.txt.
;
; To UNSUBSCRIBE:  Send a mail message to address@hidden
;                  with "unsubscribe lynx-dev" (without the
;                  quotation marks) on a line by itself.
;

reply via email to

[Prev in Thread] Current Thread [Next in Thread]