Re: [Bug-wget] wget produces erroneous robots.txt

[Tim Ruehsen]

On Wednesday 18 February 2015 07:45:53 leoh Jones wrote:
> Pardon me, if this email reaches you in error.
> email addresses taken from wget source.
> I was mirroring a webserver with wget -m <address>
> when it was done I went in to look at the files, and noticed that there is
> a robots.txt file. This was interesting, because the site mirrored  doesn't
> have a robots.txt file.
> so then, I looked at the robots.txt file contents, which was that of the
> site 404 page.

First of all, I can't reproduce it here with the latest version from git.

Looks like the new feature --content-on-error is enabled. Did you use it ? 
What do /etc/wgetrc and ~./wgetrc look like ? And very important: what is the 
output of 'wget --version' ?

> Is this a bug? I signed up for the mailing list, for wget bug reports but
> never heard back. Or is this expected behavior?

When you sign up for the mailing list, you should get an email very soon with 
further instructions. Just try it again.

Tim

signature.asc
Description: This is a digitally signed message part.