Re: [Bug-wget] wget doesnt work but curl works !

[Giuseppe Scrivano]

hi Vishwanath,

is it possible to use the last released version of wget?  You can find
it here: ftp://ftp.gnu.org/gnu/wget/wget-1.13.4.tar.xz

I have no clue what changes are contained in the so called "Red Hat
modified" version of wget, I highly suggest to use the last upstream
version in order to report bug instead of a modified version.

What system are you using?

Thanks,
Giuseppe


Vishwanath Reddy Beemidi <address@hidden> writes:

> Hi,
>
> I have trouble getting wget to work when downloading a file using http.curl
> works fine for the same URL.
>
> Both the commands are being run from the same server at the command line, OS
> : RH Linux mdc1pr012 2.6.18-238.9.1.el5
>
> Following are the commands and the debug messages, any insights into what
> the problem could be are appreciated.
>
> address@hidden wget -d -S http://www.preprod.abc.com/tools/90067660.csv
>
> Setting --server-response (serverresponse) to 1
>
> DEBUG output created by Wget 1.11.4 Red Hat modified on linux-gnu.
>
>
> --2011-10-14 18:00:58--  http://www.preprod.abc.com/tools/90067660.csv
>
> Resolving www.preprod.abc.com... 184.31.131.61
>
> Caching www.preprod.abc.com => 184.31.131.61
>
> Connecting to www.preprod.abc.com|184.31.131.61|:80... connected.
>
> Created socket 3.
>
> Releasing 0x0000000007683bc0 (new refcount 1).
>
>
> ---request begin---
>
> GET /tools/90067660.csv HTTP/1.0
>
> User-Agent: Wget/1.11.4 Red Hat modified
>
> Accept: */*
>
> Host: www.preprod.abc.com
>
> Connection: Keep-Alive
>
>
> ---request end---
>
> HTTP request sent, awaiting response...
>
> ---response begin---
>
> HTTP/1.1 302 Found
>
> Location:
> http://fd000xnchegrn02/?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y=
>
> Cache-Control: no-cache
>
> Pragma: no-cache
>
> Content-Type: text/html; charset=utf-8
>
> Connection: close
>
> Content-Length: 925
>
>
> ---response end---
>
>
>   HTTP/1.1 302 Found
>
>   Location:
> http://fd000xnchegrn02/?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y=
>
>   Cache-Control: no-cache
>
>   Pragma: no-cache
>
>   Content-Type: text/html; charset=utf-8
>
>   Connection: close
>
>   Content-Length: 925
>
> Location:
> http://fd000xnchegrn02/?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y=[following]
>
> Closed fd 3
>
> --2011-10-14 18:00:58--
> http://fd000xnchegrn02/?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y=
>
> Resolving fd000xnchegrn02... 11.48.43.72
>
> Caching fd000xnchegrn02 => 11.48.43.72
>
> Connecting to fd000xnchegrn02|11.48.43.72|:80... connected.
>
> Created socket 3.
>
> Releasing 0x00000000076806e0 (new refcount 1).
>
>
> ---request begin---
>
> GET
> /?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y=
> HTTP/1.0
>
> User-Agent: Wget/1.11.4 Red Hat modified
>
> Accept: */*
>
> Host: fd000xnchegrn02
>
> Connection: Keep-Alive
>
>
> ---request end---
>
> HTTP request sent, awaiting response...
>
> ---response begin---
>
> HTTP/1.1 401 Unauthorized
>
> Cache-Control: no-cache
>
> Pragma: no-cache
>
> WWW-Authenticate: NTLM
>
> WWW-Authenticate: BASIC realm="Federated_Department_Stores"
>
> Content-Type: text/html; charset=utf-8
>
> Proxy-Connection: close
>
> Set-Cookie: BCSI-CS-3d1fe99b15515258=2; Path=/
>
> Connection: close
>
> Content-Length: 1114
>
>
> ---response end---
>
>
>   HTTP/1.1 401 Unauthorized
>
>   Cache-Control: no-cache
>
>   Pragma: no-cache
>
>   WWW-Authenticate: NTLM
>
>   WWW-Authenticate: BASIC realm="Federated_Department_Stores"
>
>   Content-Type: text/html; charset=utf-8
>
>   Proxy-Connection: close
>
>   Set-Cookie: BCSI-CS-3d1fe99b15515258=2; Path=/
>
>   Connection: close
>
>   Content-Length: 1114
>
> Closed fd 3
>
> Authorization failed.
>
>
>>>>>>>>>>>>>>>> Following is the curl trace info for the same URL
>
> address@hidden ~]$ curl --trace-ascii tr.out -o out.dat
> http://www.preprodmacys.fds.com/tools/90067660.csv
>   % Total    % Received % Xferd  Average Speed   Time    Time     Time
>  Current
>                                  Dload  Upload   Total   Spent    Left
>  Speed
> 100  155k  100  155k    0     0   449k      0 --:--:-- --:--:-- --:--:--
>  550k
>
> address@hidden ~]$ more tr.out
>
> == Info: About to connect() to www.preprod.abc.com port 80
>
> == Info:   Trying 184.31.131.61... == Info: connected
>
> == Info: Connected to www.preprod.abc.com (184.31.131.61) port 80
>
> => Send header, 186 bytes (0xba)
>
> 0000: GET /tools/90067660.csv HTTP/1.1
>
> 0022: User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5
>
> 0062:  OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
>
> 008b: Host: www.preprod.abc.com
>
> 00ab: Accept: */*
>
> 00b8:
>
> <= Recv header, 17 bytes (0x11)
>
> 0000: HTTP/1.1 200 OK
>
> <= Recv header, 25 bytes (0x19)
>
> 0000: Server: IBM_HTTP_Server
>
> <= Recv header, 46 bytes (0x2e)
>
> 0000: Last-Modified: Fri, 18 Mar 2011 22:42:46 GMT
>
> <= Recv header, 30 bytes (0x1e)
>
> 0000: ETag: "b19b9-26f41-7f2b4580"
>
> <= Recv header, 22 bytes (0x16)
>
> 0000: Accept-Ranges: bytes
>
> <= Recv header, 26 bytes (0x1a)
>
> 0000: Content-Type: text/plain
>
> <= Recv header, 37 bytes (0x25)
>
> 0000: Date: Fri, 14 Oct 2011 22:06:15 GMT
>
> <= Recv header, 24 bytes (0x18)
>
> 0000: Content-Length: 159553
>
> <= Recv header, 24 bytes (0x18)
>
> 0000: Connection: Keep-Alive
>
> <= Recv header, 127 bytes (0x7f)
>
> 0000: Set-Cookie: BC_HA_7039BD33D34B38D1_70737B63=4117DB; Domain=.prep
>
> 0040: rodmacys.fds.com; expires=Fri, 14-Oct-11 22:36:15 GMT; Path=/
>
> <= Recv header, 70 bytes (0x46)
>
> 0000: Set-Cookie: shippingCountry=US; path=/; domain=.preprod.abc
>
> 0040: .com
>
> <= Recv header, 104 bytes (0x68)
>
> 0000: Set-Cookie: stop_mobi=yes; expires=Fri, 14-Oct-2011 22:36:15 GMT
>
> 0040: ; path=/; domain=.preprod.abc.com
>
> <= Recv data, 5078 bytes (0x13d6)
>
> 0000: 90067660,po-xx-xx-xx,popup,
>
> 001d: 90067660,ex-xx-xx-xx,external sites,
>
> 0043: 90067660,im-xx-xx-xx,imports,
> .........MORE LINES..............
>
> <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
>
>
> Also when we run wget from another server against the same URL it works
> fine...the version on the other server is 1.10.4. So we got that version and
> installed it on the server mentioned in the above example and tried but we
> still got the same error message.
>
> Following is the debug log from the successful run on the another server :
> address@hidden /home/scripts(0)$ wget -d -S
> http://www.preprod.abc.com/tools/90067660.csv
>
> Setting --server-response (serverresponse) to 1
>
> DEBUG output created by Wget 1.10.2 (Red Hat modified) on linux-gnu.
>
>
> --15:03:42--  http://www.preprod.abc.com/tools/90067660.csv
>
>            => `90067660.csv.3'
>
> Resolving www.preprod.abc.com... 184.31.131.61
>
> Caching www.preprod.abc.com => 184.31.131.61
>
> Connecting to www.preprod.abc.com|184.31.131.61|:80... connected.
>
> Created socket 3.
>
> Releasing 0x000000000063b390 (new refcount 1).
>
>
> ---request begin---
>
> GET /tools/90067660.csv HTTP/1.0
>
> User-Agent: Wget/1.10.2 (Red Hat modified)
>
> Accept: */*
>
> Host: www.preprod.abc.com
>
> Connection: Keep-Alive
>
>
> ---request end---
>
> HTTP request sent, awaiting response...
>
> ---response begin---
>
> HTTP/1.0 200 OK
>
> Server: IBM_HTTP_Server
>
> Last-Modified: Fri, 18 Mar 2011 22:42:46 GMT
>
> ETag: "b19b9-26f41-7f2b4580"
>
> Accept-Ranges: bytes
>
> Content-Type: text/plain
>
> Content-Length: 159553
>
> Date: Fri, 14 Oct 2011 22:03:42 GMT
>
> Connection: keep-alive
>
> Set-Cookie: BC_HA_7039BD33D34B38D1_70737B63=41174D; Domain=.preprod.abc.com;
> expires=Fri, 14-Oct-11 22:33:42 GMT; Path=/
>
> Set-Cookie: shippingCountry=US; path=/; domain=.preprod.abc.com
>
> Set-Cookie: stop_mobi=yes; expires=Fri, 14-Oct-2011 22:33:42 GMT; path=/;
> domain=.preprod.abc.com
>
>
> ---response end---
>
>
>   HTTP/1.0 200 OK
>
>   Server: IBM_HTTP_Server
>
>   Last-Modified: Fri, 18 Mar 2011 22:42:46 GMT
>
>   ETag: "b19b9-26f41-7f2b4580"
>
>   Accept-Ranges: bytes
>
>   Content-Type: text/plain
>
>   Content-Length: 159553
>
>   Date: Fri, 14 Oct 2011 22:03:42 GMT
>
>   Connection: keep-alive
>
>   Set-Cookie: BC_HA_7039BD33D34B38D1_70737B63=41174D; Domain=.
> preprod.abc.com; expires=Fri, 14-Oct-11 22:33:42 GMT; Path=/
>
>   Set-Cookie: shippingCountry=US; path=/; domain=.preprod.abc.com
>
>   Set-Cookie: stop_mobi=yes; expires=Fri, 14-Oct-2011 22:33:42 GMT; path=/;
> domain=.preprod.abc.com
>
> Registered socket 3 for persistent reuse.
>
> cdm: 1 2 3 4 5 6 7 8
>
> Stored cookie preprod.abc.com -1 (ANY) / <permanent> <insecure> [expiry
> 2011-10-14 15:33:42] BC_HA_7039BD33D34B38D1_70737B63 41174D
>
> cdm: 1 2 3 4 5 6 7 8
>
> Stored cookie preprod.abc.com -1 (ANY) / <session> <insecure> [expiry none]
> shippingCountry US
>
> cdm: 1 2 3 4 5 6 7 8
>
> Stored cookie preprod.abc.com -1 (ANY) / <permanent> <insecure> [expiry
> 2011-10-14 15:33:42] stop_mobi yes
>
> Length: 159,553 (156K) [text/plain]
>
>
> 100%[=================================================================================================================================================================================================>]
> 159,553      105.31K/s
>
>
> 15:03:44 (105.15 KB/s) - `90067660.csv.3' saved [159553/159553]
>
>
>
> PS : I changed the domain names mentioned in the logs above
>
> Thanks,
> Vishwanath