|
| From: | Ángel González |
| Subject: | Re: [Bug-wget] wget produces erroneous robots.txt |
| Date: | Thu, 19 Feb 2015 22:42:36 +0100 |
| User-agent: | Thunderbird |
leoh Jones wrote:
I can buy mis-configured server. I only tried it one the one server, maybe that server has an issue with http 200 requests.
Run wget -S http://<server>/robots.txt and see what was returned by the server.
| [Prev in Thread] | Current Thread | [Next in Thread] |