[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: lynx-dev Deny access for non standard browsers
|
From: |
David Woolley |
|
Subject: |
Re: lynx-dev Deny access for non standard browsers |
|
Date: |
Sat, 31 Jul 1999 00:09:31 +0100 (BST) |
>
> From: David Woolley <address@hidden>
> > (I pointed out that he should use robots.txt), so I suspect that
>
> Does lynx's crawl mode now respect rebots.txt ? I thought that it still
> didn't, which was a complaint the list had gotten in the past.
I don't think it does, but he also blacklisted wget, which does.