lynx-dev
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: lynx-dev Non-interactive lynx


From: Patrick
Subject: Re: lynx-dev Non-interactive lynx
Date: Sun, 18 Mar 2001 04:07:40 -0800

In "lynx-dev Non-interactive lynx"
[17/Mar/2001Sat 13:13:00]
Ilya Zakharevich wrote:

> Re prohibiting lynx from visiting sites due to misusage of
> unattended-operation mode.  What about prefixing "Non-interactive " to
> the default user-agent string for non-interractive robot-like runs of
> lynx?

I think it would still provoke those who spend time and consideration
on which of their files have;
        <META NAME="robots" CONTENT="all/none/nofollow/noindex">
and so forth.  Also bear in mind that no robot can read copyright
notices in the body of a page.

Just wondered: how easy/hard would it be to make Lynx obey robot
exclusion protocols in non-interactive mode?  This is also done
with HTTP headers?


                             Patrick
                   <mailto:address@hidden>
                <http://www.island.net/~pboylan/>

; To UNSUBSCRIBE: Send "unsubscribe lynx-dev" to address@hidden

reply via email to

[Prev in Thread] Current Thread [Next in Thread]