[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Bug-wget] FW: Wget export URL list
From: |
The PowerTool |
Subject: |
[Bug-wget] FW: Wget export URL list |
Date: |
Wed, 3 Sep 2014 16:13:36 -0400 |
From: address@hidden
To: address@hidden
Subject: RE: [Bug-wget] Wget export URL list
Date: Wed, 3 Sep 2014 16:12:07 -0400
> Can anyone tell me how to do this with wget ?
> I want it to spider a given website and return the list of full urls in
> that website.
You haven't provided any details of your requirements beyond obtaining the
result and using wget.
wget alone will not do what you want. If it were me, I would:
1. wget the html for the site (only download the html)
2. run the html through a simple bash script to rip out all of the URLs
- and grep -in and -out, as appropriate
The simple script can be found at
http://www.comp.eonworks.com/scripts/isolate_url_link-20020716.html
This assumes a real OS (not win*) and have a small working knowledge of wget,
CL (bash), and can dl and extract the script.