[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Bug-wget] Mirror a website but no sites with special chars like "?"
|
From: |
Tobias Krais |
|
Subject: |
Re: [Bug-wget] Mirror a website but no sites with special chars like "?" |
|
Date: |
Sun, 25 Mar 2012 22:13:59 +0200 |
|
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:11.0) Gecko/20120310 Thunderbird/11.0 |
Hi Paul,
sorry for my late answer.
> this will build the folder structure and should get the servers robots.txt
> file
> edit the file to exclude the other unwanted urls
> then do your actual recursive mirror
> just google example robots.txt to hack what you need
I never thought about this idea. I'll search for it. Thanks for the hint!
Greetings,
Tobias