bug-wget
[Top][All Lists]
Advanced
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Bug-wget] Question regarding robots option

From: Ángel González
Subject: Re: [Bug-wget] Question regarding robots option
Date: Mon, 16 Sep 2013 23:47:38 +0200
User-agent: Thunderbird 
On 16/09/13 13:35, Tim Ruehsen wrote:

How comes that there is no --robots / --no-robots option ?
Instead we have to use -e robots=on/off.
This is ok, but a little bit inconsistent and hard to find (wget --help won't
show it).

I would create a patch if there is any interest...

Tim
I think that's on purpose for making hard to ignore the robots.txt specification.
reply via email to
[Prev in Thread] Current Thread [Next in Thread]