bug-wget
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Bug-wget] How do I tell wget not to follow links in a file?


From: David Skalinder
Subject: Re: [Bug-wget] How do I tell wget not to follow links in a file?
Date: Mon, 11 Apr 2011 04:13:22 +0100
User-agent: SquirrelMail/1.4.21

It just occurred to me that since wget will perform this task properly if
it gets the rule from robots.txt, maybe this issue could be worked around
by proxying or spoofing the remote site's robots.txt file locally?  That
is, I write

User-agent: *
Disallow: wgettest/links2.html

into a file, save it in my home directory, and then somehow tell wget that
davidskalinder.com/robots.txt is actually located at
/home/user/robots.txt?

Does anybody know a convenient way of doing this?  Or is there an easier
workaround I'm overlooking?




reply via email to

[Prev in Thread] Current Thread [Next in Thread]