[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Bug-wget] Does wget check if specified user agent is allowed in robots.
[Bug-wget] Does wget check if specified user agent is allowed in robots.txt?
Thu, 19 Jun 2014 23:17:50 +0200
If I specify a custom user agent for wget, eg "MyBot 1.0 (address@hidden)"
Will wget check this in robots.txt as well, if the bot was banned, or only
the general robot exclusions? Does wget check if "MyBot" is allowed to
If not, this would be a nice feature. If yes, it would be great to include
this info in the robots overview here https://www.gnu.org/software/wget
I originally posted this question here , but then I found this list
274 44 98
06 30 5888 744