bug-wget
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Bug-wget] Does wget check if specified user agent is allowed in rob


From: Darshit Shah
Subject: Re: [Bug-wget] Does wget check if specified user agent is allowed in robots.txt?
Date: Sun, 22 Jun 2014 01:14:30 +0530

On Sun, Jun 22, 2014 at 1:10 AM, György Chityil
<address@hidden> wrote:
> Thank you so much! This is a perfect reply. Couldn't have asked for more.
>
> While not a bug, an additional idea came to my mind while reading your
> reply. If this robots checking feature will be fixed, it would be great to
> be able to enable robots checking for simple, one off requests as w

That does not make sense. The robots.txt file is expected to be
checked and adhered to by robots / spiders trying to crawl the
website, not by clients trying to access a single / few pages. Wget
should NOT check the robots file for single downloads. It should be
following the rules in robots.txt only when acting like a spider, i.e.
when in recursive mode.


-- 
Thanking You,
Darshit Shah



reply via email to

[Prev in Thread] Current Thread [Next in Thread]