Re: [Bug-wget] wget "referer" development idea

[Micah Cowan]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

hysterix wrote:
> Now, if I manually change the referer to a certain directory within the
> domain, I can see the page instead of a log-in page, but when I try to
> follow those links and save them, it throws me back to the log in
> screen.  After many hours of tedious and careful study, I realized that
> when I changed the referrer manually, I was able to see the page I
> couldn't see before, but only in that directory, the second I tried to
> traverse one directory deep, it would kick me out because referrer was
> then wrong.  I studied the headers with live http headers and sure
> enough the referrer variable is changing around so I assume their vb
> software is programmed to pick it up and check it with every page load!
> 
> So, my question, or comment or statement is, how hard would it be to
> implement a switch, say for example --recursive-referrer and when this
> switch is used, wget will actively change the 'referer' value to
> whatever page it just previously came from whilst traversing through all
> directories, enabling full mirroring of sites that check the referrer
> variable and if it is wrong kicks you out (in this case vb forums).

What you describe actually is the current behavior for wget: real
referers (the pages in which the links were originally found) override
the user-specified one (you can verify this yourself by adding -d and
looking at the headers wget sends: if your results differ, please attach
the log resulting from the command with -d). This appears to have been
the case since around version 1.8. What version of Wget are you using?

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer.
GNU Maintainer: wget, screen, teseq
http://micah.cowan.name/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkl5aQcACgkQ7M8hyUobTrGxmwCdFtiIQa7Q5hruuJjtgBeVkmuZ
VScAnjO7jirs8rGmcMPGS3O31HhA6uz2
=penM
-----END PGP SIGNATURE-----