bug-wget
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Bug-wget] does wget honor robots meta tag?


From: Allan Spiegel
Subject: [Bug-wget] does wget honor robots meta tag?
Date: Fri, 14 Nov 2008 17:00:34 -0500
User-agent: Thunderbird 2.0.0.17 (Windows/20080914)

I have some pages with

           <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

in the <head> section and when I crawl my site to make sure I have this tag in all the right pages, wget gets these pages. does wget support this tag?

here's the command I used:

wget --delete-after --cache=off --no-verbose --recursive --level=8 --no-directories --no-host-directories --no-check-certificate --no-parent --page-requisites -U "Safari/41-wget_link_testing" http://my.domain.com/folder/

wget -V
GNU Wget 1.11.4

If wget doesn't support this can someone suggest another way to verify these tags are working properly?

Thanks
Allan




reply via email to

[Prev in Thread] Current Thread [Next in Thread]