[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Bug-wget] Does wget check if specified user agent is allowed in rob
Re: [Bug-wget] Does wget check if specified user agent is allowed in robots.txt?
Sun, 22 Jun 2014 01:14:30 +0530
On Sun, Jun 22, 2014 at 1:10 AM, György Chityil
> Thank you so much! This is a perfect reply. Couldn't have asked for more.
> While not a bug, an additional idea came to my mind while reading your
> reply. If this robots checking feature will be fixed, it would be great to
> be able to enable robots checking for simple, one off requests as w
That does not make sense. The robots.txt file is expected to be
checked and adhered to by robots / spiders trying to crawl the
website, not by clients trying to access a single / few pages. Wget
should NOT check the robots file for single downloads. It should be
following the rules in robots.txt only when acting like a spider, i.e.
when in recursive mode.