Re: [Bug-wget] wget doesnt work but curl works !

[Micah Cowan]

Also, I think Tony Lewis's updated NTLM support went in sometime after
1.11.4.

(To my knowledge, which may have gaps, the only Redhat-modified wgets
that were seriously different from GNU's, was the 1.10.2 series, which
included many incomplete/untested bits from 1.11.4, and maybe a couple
of their own changes.)

-mjc

(2011年11月13日 02:15), Giuseppe Scrivano wrote:
> hi Vishwanath,
>
> is it possible to use the last released version of wget?  You can find
> it here: ftp://ftp.gnu.org/gnu/wget/wget-1.13.4.tar.xz
>
> I have no clue what changes are contained in the so called "Red Hat
> modified" version of wget, I highly suggest to use the last upstream
> version in order to report bug instead of a modified version.
>
> What system are you using?
>
> Thanks,
> Giuseppe
>
>
> Vishwanath Reddy Beemidi <address@hidden> writes:
>
>> Hi,
>>
>> I have trouble getting wget to work when downloading a file using http.curl
>> works fine for the same URL.
>>
>> Both the commands are being run from the same server at the command line, OS
>> : RH Linux mdc1pr012 2.6.18-238.9.1.el5
>>
>> Following are the commands and the debug messages, any insights into what
>> the problem could be are appreciated.
>>
>> address@hidden wget -d -S http://www.preprod.abc.com/tools/90067660.csv
>>
>> Setting --server-response (serverresponse) to 1
>>
>> DEBUG output created by Wget 1.11.4 Red Hat modified on linux-gnu.
>>
>>
>> --2011-10-14 18:00:58--  http://www.preprod.abc.com/tools/90067660.csv
>>
>> Resolving www.preprod.abc.com... 184.31.131.61
>>
>> Caching www.preprod.abc.com => 184.31.131.61
>>
>> Connecting to www.preprod.abc.com|184.31.131.61|:80... connected.
>>
>> Created socket 3.
>>
>> Releasing 0x0000000007683bc0 (new refcount 1).
>>
>>
>> ---request begin---
>>
>> GET /tools/90067660.csv HTTP/1.0
>>
>> User-Agent: Wget/1.11.4 Red Hat modified
>>
>> Accept: */*
>>
>> Host: www.preprod.abc.com
>>
>> Connection: Keep-Alive
>>
>>
>> ---request end---
>>
>> HTTP request sent, awaiting response...
>>
>> ---response begin---
>>
>> HTTP/1.1 302 Found
>>
>> Location:
>> http://fd000xnchegrn02/?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y=
>>
>> Cache-Control: no-cache
>>
>> Pragma: no-cache
>>
>> Content-Type: text/html; charset=utf-8
>>
>> Connection: close
>>
>> Content-Length: 925
>>
>>
>> ---response end---
>>
>>
>>   HTTP/1.1 302 Found
>>
>>   Location:
>> http://fd000xnchegrn02/?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y=
>>
>>   Cache-Control: no-cache
>>
>>   Pragma: no-cache
>>
>>   Content-Type: text/html; charset=utf-8
>>
>>   Connection: close
>>
>>   Content-Length: 925
>>
>> Location:
>> http://fd000xnchegrn02/?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y=[following]
>>
>> Closed fd 3
>>
>> --2011-10-14 18:00:58--
>> http://fd000xnchegrn02/?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y=
>>
>> Resolving fd000xnchegrn02... 11.48.43.72
>>
>> Caching fd000xnchegrn02 => 11.48.43.72
>>
>> Connecting to fd000xnchegrn02|11.48.43.72|:80... connected.
>>
>> Created socket 3.
>>
>> Releasing 0x00000000076806e0 (new refcount 1).
>>
>>
>> ---request begin---
>>
>> GET
>> /?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y=
>> HTTP/1.0
>>
>> User-Agent: Wget/1.11.4 Red Hat modified
>>
>> Accept: */*
>>
>> Host: fd000xnchegrn02
>>
>> Connection: Keep-Alive
>>
>>
>> ---request end---
>>
>> HTTP request sent, awaiting response...
>>
>> ---response begin---
>>
>> HTTP/1.1 401 Unauthorized
>>
>> Cache-Control: no-cache
>>
>> Pragma: no-cache
>>
>> WWW-Authenticate: NTLM
>>
>> WWW-Authenticate: BASIC realm="Federated_Department_Stores"
>>
>> Content-Type: text/html; charset=utf-8
>>
>> Proxy-Connection: close
>>
>> Set-Cookie: BCSI-CS-3d1fe99b15515258=2; Path=/
>>
>> Connection: close
>>
>> Content-Length: 1114
>>
>>
>> ---response end---
>>
>>
>>   HTTP/1.1 401 Unauthorized
>>
>>   Cache-Control: no-cache
>>
>>   Pragma: no-cache
>>
>>   WWW-Authenticate: NTLM
>>
>>   WWW-Authenticate: BASIC realm="Federated_Department_Stores"
>>
>>   Content-Type: text/html; charset=utf-8
>>
>>   Proxy-Connection: close
>>
>>   Set-Cookie: BCSI-CS-3d1fe99b15515258=2; Path=/
>>
>>   Connection: close
>>
>>   Content-Length: 1114
>>
>> Closed fd 3
>>
>> Authorization failed.
>>
>>
>>>>>>>>>>>>>>>>> Following is the curl trace info for the same URL
>> address@hidden ~]$ curl --trace-ascii tr.out -o out.dat
>> http://www.preprodmacys.fds.com/tools/90067660.csv
>>   % Total    % Received % Xferd  Average Speed   Time    Time     Time
>>  Current
>>                                  Dload  Upload   Total   Spent    Left
>>  Speed
>> 100  155k  100  155k    0     0   449k      0 --:--:-- --:--:-- --:--:--
>>  550k
>>
>> address@hidden ~]$ more tr.out
>>
>> == Info: About to connect() to www.preprod.abc.com port 80
>>
>> == Info:   Trying 184.31.131.61... == Info: connected
>>
>> == Info: Connected to www.preprod.abc.com (184.31.131.61) port 80
>>
>> => Send header, 186 bytes (0xba)
>>
>> 0000: GET /tools/90067660.csv HTTP/1.1
>>
>> 0022: User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5
>>
>> 0062:  OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
>>
>> 008b: Host: www.preprod.abc.com
>>
>> 00ab: Accept: */*
>>
>> 00b8:
>>
>> <= Recv header, 17 bytes (0x11)
>>
>> 0000: HTTP/1.1 200 OK
>>
>> <= Recv header, 25 bytes (0x19)
>>
>> 0000: Server: IBM_HTTP_Server
>>
>> <= Recv header, 46 bytes (0x2e)
>>
>> 0000: Last-Modified: Fri, 18 Mar 2011 22:42:46 GMT
>>
>> <= Recv header, 30 bytes (0x1e)
>>
>> 0000: ETag: "b19b9-26f41-7f2b4580"
>>
>> <= Recv header, 22 bytes (0x16)
>>
>> 0000: Accept-Ranges: bytes
>>
>> <= Recv header, 26 bytes (0x1a)
>>
>> 0000: Content-Type: text/plain
>>
>> <= Recv header, 37 bytes (0x25)
>>
>> 0000: Date: Fri, 14 Oct 2011 22:06:15 GMT
>>
>> <= Recv header, 24 bytes (0x18)
>>
>> 0000: Content-Length: 159553
>>
>> <= Recv header, 24 bytes (0x18)
>>
>> 0000: Connection: Keep-Alive
>>
>> <= Recv header, 127 bytes (0x7f)
>>
>> 0000: Set-Cookie: BC_HA_7039BD33D34B38D1_70737B63=4117DB; Domain=.prep
>>
>> 0040: rodmacys.fds.com; expires=Fri, 14-Oct-11 22:36:15 GMT; Path=/
>>
>> <= Recv header, 70 bytes (0x46)
>>
>> 0000: Set-Cookie: shippingCountry=US; path=/; domain=.preprod.abc
>>
>> 0040: .com
>>
>> <= Recv header, 104 bytes (0x68)
>>
>> 0000: Set-Cookie: stop_mobi=yes; expires=Fri, 14-Oct-2011 22:36:15 GMT
>>
>> 0040: ; path=/; domain=.preprod.abc.com
>>
>> <= Recv data, 5078 bytes (0x13d6)
>>
>> 0000: 90067660,po-xx-xx-xx,popup,
>>
>> 001d: 90067660,ex-xx-xx-xx,external sites,
>>
>> 0043: 90067660,im-xx-xx-xx,imports,
>> .........MORE LINES..............
>>
>> <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
>>
>>
>> Also when we run wget from another server against the same URL it works
>> fine...the version on the other server is 1.10.4. So we got that version and
>> installed it on the server mentioned in the above example and tried but we
>> still got the same error message.
>>
>> Following is the debug log from the successful run on the another server :
>> address@hidden /home/scripts(0)$ wget -d -S
>> http://www.preprod.abc.com/tools/90067660.csv
>>
>> Setting --server-response (serverresponse) to 1
>>
>> DEBUG output created by Wget 1.10.2 (Red Hat modified) on linux-gnu.
>>
>>
>> --15:03:42--  http://www.preprod.abc.com/tools/90067660.csv
>>
>>            => `90067660.csv.3'
>>
>> Resolving www.preprod.abc.com... 184.31.131.61
>>
>> Caching www.preprod.abc.com => 184.31.131.61
>>
>> Connecting to www.preprod.abc.com|184.31.131.61|:80... connected.
>>
>> Created socket 3.
>>
>> Releasing 0x000000000063b390 (new refcount 1).
>>
>>
>> ---request begin---
>>
>> GET /tools/90067660.csv HTTP/1.0
>>
>> User-Agent: Wget/1.10.2 (Red Hat modified)
>>
>> Accept: */*
>>
>> Host: www.preprod.abc.com
>>
>> Connection: Keep-Alive
>>
>>
>> ---request end---
>>
>> HTTP request sent, awaiting response...
>>
>> ---response begin---
>>
>> HTTP/1.0 200 OK
>>
>> Server: IBM_HTTP_Server
>>
>> Last-Modified: Fri, 18 Mar 2011 22:42:46 GMT
>>
>> ETag: "b19b9-26f41-7f2b4580"
>>
>> Accept-Ranges: bytes
>>
>> Content-Type: text/plain
>>
>> Content-Length: 159553
>>
>> Date: Fri, 14 Oct 2011 22:03:42 GMT
>>
>> Connection: keep-alive
>>
>> Set-Cookie: BC_HA_7039BD33D34B38D1_70737B63=41174D; Domain=.preprod.abc.com;
>> expires=Fri, 14-Oct-11 22:33:42 GMT; Path=/
>>
>> Set-Cookie: shippingCountry=US; path=/; domain=.preprod.abc.com
>>
>> Set-Cookie: stop_mobi=yes; expires=Fri, 14-Oct-2011 22:33:42 GMT; path=/;
>> domain=.preprod.abc.com
>>
>>
>> ---response end---
>>
>>
>>   HTTP/1.0 200 OK
>>
>>   Server: IBM_HTTP_Server
>>
>>   Last-Modified: Fri, 18 Mar 2011 22:42:46 GMT
>>
>>   ETag: "b19b9-26f41-7f2b4580"
>>
>>   Accept-Ranges: bytes
>>
>>   Content-Type: text/plain
>>
>>   Content-Length: 159553
>>
>>   Date: Fri, 14 Oct 2011 22:03:42 GMT
>>
>>   Connection: keep-alive
>>
>>   Set-Cookie: BC_HA_7039BD33D34B38D1_70737B63=41174D; Domain=.
>> preprod.abc.com; expires=Fri, 14-Oct-11 22:33:42 GMT; Path=/
>>
>>   Set-Cookie: shippingCountry=US; path=/; domain=.preprod.abc.com
>>
>>   Set-Cookie: stop_mobi=yes; expires=Fri, 14-Oct-2011 22:33:42 GMT; path=/;
>> domain=.preprod.abc.com
>>
>> Registered socket 3 for persistent reuse.
>>
>> cdm: 1 2 3 4 5 6 7 8
>>
>> Stored cookie preprod.abc.com -1 (ANY) / <permanent> <insecure> [expiry
>> 2011-10-14 15:33:42] BC_HA_7039BD33D34B38D1_70737B63 41174D
>>
>> cdm: 1 2 3 4 5 6 7 8
>>
>> Stored cookie preprod.abc.com -1 (ANY) / <session> <insecure> [expiry none]
>> shippingCountry US
>>
>> cdm: 1 2 3 4 5 6 7 8
>>
>> Stored cookie preprod.abc.com -1 (ANY) / <permanent> <insecure> [expiry
>> 2011-10-14 15:33:42] stop_mobi yes
>>
>> Length: 159,553 (156K) [text/plain]
>>
>>
>> 100%[=================================================================================================================================================================================================>]
>> 159,553      105.31K/s
>>
>>
>> 15:03:44 (105.15 KB/s) - `90067660.csv.3' saved [159553/159553]
>>
>>
>>
>> PS : I changed the domain names mentioned in the logs above
>>
>> Thanks,
>> Vishwanath