[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Bug-wget] Does wget check if specified user agent is allowed in robots.
|
From: |
György Chityil |
|
Subject: |
[Bug-wget] Does wget check if specified user agent is allowed in robots.txt? |
|
Date: |
Thu, 19 Jun 2014 23:17:50 +0200 |
If I specify a custom user agent for wget, eg "MyBot 1.0 (address@hidden)"
Will wget check this in robots.txt as well, if the bot was banned, or only
the general robot exclusions? Does wget check if "MyBot" is allowed to
crawl?
If not, this would be a nice feature. If yes, it would be great to include
this info in the robots overview here https://www.gnu.org/software/wget
I originally posted this question here , but then I found this list
http://stackoverflow.com/questions/24316018/does-wget-check-if-specified-user-agent-is-allowed-in-robots-txt
--
Gyuri
274 44 98
06 30 5888 744