Re: [Bug-wget] grab complete download link

[Darshit Shah]

I misread the original email. As Yousong suggests, you want the
recursive option, not page requisites.

Also the page requires you to login. In a web browser it is easy. But
on a command line client it isn't that straighforward. The wget man
page explains how you can work around this. Simply setting --user and
--password doesn't help. Those are used only for basic / digest
authorization. This page uses a HTML Form based authentication. To do
this, you'll first need to:

wget --save-session-cookies --keep-cookies=cookies.txt --method=POST
--body-data="username=test+&password=dummy&rememberMe=0&submit="
https://earthexplorer.usgs.gov/login/

Then run the following command:
wget --load-cookies=cookies.txt -r -l 1 <URL>

If it still doesn't work, there's probably some difference on you the
website tries to authenticate you. I'd suggest you add the --debug
parameter to both the commands and share the output logs so that we
can try and understand what's wrong

On Mon, Jul 21, 2014 at 11:24 AM, Yousong Zhou <address@hidden> wrote:
> Hi,
>
> On 21 July 2014 09:38, bas smit <address@hidden> wrote:
>> Dear Darshit Shah
>> Thanks for your response.
>>
>> I tried with the following command:
>> subprocess.call([wget,'--user',user,'--password',passw,'-P',download_dir,'--page-requisites',url,'-o',logfile,\
>>     '--no-check-certificate'])
>>
>
> The URL you provided needs login to access.  But I guess recursive
> download is what you want.  Try options `--recursive --level=1` , or
> `-r -l 1` for the short equivalent.
>
>> However, still unsuccessful to download the required file.
>>
>> I also obtained the following in the log file:
>>
>> WARNING: Certificate verification error: unable to get local issuer
>> certificate
>>
>>
>> I hope you can help me.
>>
>> Bas
>>
>>
>> WARNING: Certificate verification error: unable to get local issuer
>> certificate
>>
>>
>> On Thu, Jul 17, 2014 at 9:34 PM, Darshit Shah <address@hidden> wrote:
>>
>>> You want to use the --page-requisites option
>>>
>>> On Thu, Jul 17, 2014 at 2:22 PM, bas smit <address@hidden> wrote:
>>> > I am looking for command line option to use the same functionality as the
>>> > "Download All with Free Download Manager" does. It grabs the complete
>>> > download links though only partial links are shown in the source html.  I
>>> > tried the following code, but but could not figure out which particular
>>> > parameter is necessary for that. The url provided below is the only known
>>> > one.
>>> >
>>> > import subprocess
>>> >
>>> > user, passw = 'user', 'passw'
>>> >
>>> > url = '
>>> http://earthexplorer.usgs.gov/download/3120/LM10300301974324GDS05/STANDARD/BulkDownload
>>> '
>>> >
>>> > wget = "C:\\Users\\bas\\Downloads\\wget-1.10.2.exe"
>>> > subprocess.call([wget, '--user', user, '--password', passw, url])
>>>
>>>
>>>
>>> --
>>> Thanking You,
>>> Darshit Shah
>>>



-- 
Thanking You,
Darshit Shah