Re: [Bug-wget] --page-requisites and robot exclusion issue

[Giuseppe Scrivano]

Paul Wratt <address@hidden> writes:

> if it does not obey - server admins will ban it
>
> the work around:
> 1) get single html file first - edit out meta tag - re-get with
> --no-clobber (usually only in landing pages)
> 2) empty robots.txt (or allow all - search net)
>
> possible solutions:
> A) command line option
> B) ./configure --disable-robots-check

you can specify -e robots=off to wget at runtime.

Giuseppe