Re: [Bug-wget] Does wget check if specified user agent is allowed in robots.txt?

[Ángel González]

On 21/06/14 21:59, György Chityil wrote:
> Thanks for your kind reply.
>
> My line of thinking was that a robot client should obey the robots.txt of a
> site regardless of single or multiple requests. Even if it shouldn't as you
> say, it could be great to have a separate flag for it instead of having it
> buried in the recursive option.

What is your rationale for having wget refuse to download the single url 
you requested?
I can't think on a case where you "want this url" and at the same time 
don't want it
if it shouldn't be crawled (eg. a calendar).