bug-wget
[Top][All Lists]
Advanced
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Bug-wget] wget doesnt work but curl works !

From: Vishwanath Reddy Beemidi
Subject: [Bug-wget] wget doesnt work but curl works !
Date: Fri, 14 Oct 2011 15:24:39 -0700 
Hi,

I have trouble getting wget to work when downloading a file using http.curl
works fine for the same URL.

Both the commands are being run from the same server at the command line, OS
: RH Linux mdc1pr012 2.6.18-238.9.1.el5

Following are the commands and the debug messages, any insights into what
the problem could be are appreciated.

address@hidden wget -d -S http://www.preprod.abc.com/tools/90067660.csv

Setting --server-response (serverresponse) to 1

DEBUG output created by Wget 1.11.4 Red Hat modified on linux-gnu.


--2011-10-14 18:00:58--  http://www.preprod.abc.com/tools/90067660.csv

Resolving www.preprod.abc.com... 184.31.131.61

Caching www.preprod.abc.com => 184.31.131.61

Connecting to www.preprod.abc.com|184.31.131.61|:80... connected.

Created socket 3.

Releasing 0x0000000007683bc0 (new refcount 1).


---request begin---

GET /tools/90067660.csv HTTP/1.0

User-Agent: Wget/1.11.4 Red Hat modified

Accept: */*

Host: www.preprod.abc.com

Connection: Keep-Alive


---request end---

HTTP request sent, awaiting response...

---response begin---

HTTP/1.1 302 Found

Location:
http://fd000xnchegrn02/?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y=

Cache-Control: no-cache

Pragma: no-cache

Content-Type: text/html; charset=utf-8

Connection: close

Content-Length: 925


---response end---


  HTTP/1.1 302 Found

  Location:
http://fd000xnchegrn02/?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y=

  Cache-Control: no-cache

  Pragma: no-cache

  Content-Type: text/html; charset=utf-8

  Connection: close

  Content-Length: 925

Location:
http://fd000xnchegrn02/?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y=[following]

Closed fd 3

--2011-10-14 18:00:58--
http://fd000xnchegrn02/?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y=

Resolving fd000xnchegrn02... 11.48.43.72

Caching fd000xnchegrn02 => 11.48.43.72

Connecting to fd000xnchegrn02|11.48.43.72|:80... connected.

Created socket 3.

Releasing 0x00000000076806e0 (new refcount 1).


---request begin---

GET
/?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y=
HTTP/1.0

User-Agent: Wget/1.11.4 Red Hat modified

Accept: */*

Host: fd000xnchegrn02

Connection: Keep-Alive


---request end---

HTTP request sent, awaiting response...

---response begin---

HTTP/1.1 401 Unauthorized

Cache-Control: no-cache

Pragma: no-cache

WWW-Authenticate: NTLM

WWW-Authenticate: BASIC realm="Federated_Department_Stores"

Content-Type: text/html; charset=utf-8

Proxy-Connection: close

Set-Cookie: BCSI-CS-3d1fe99b15515258=2; Path=/

Connection: close

Content-Length: 1114


---response end---


  HTTP/1.1 401 Unauthorized

  Cache-Control: no-cache

  Pragma: no-cache

  WWW-Authenticate: NTLM

  WWW-Authenticate: BASIC realm="Federated_Department_Stores"

  Content-Type: text/html; charset=utf-8

  Proxy-Connection: close

  Set-Cookie: BCSI-CS-3d1fe99b15515258=2; Path=/

  Connection: close

  Content-Length: 1114

Closed fd 3

Authorization failed.


>>>>>>>>>>>>>>> Following is the curl trace info for the same URL

address@hidden ~]$ curl --trace-ascii tr.out -o out.dat
http://www.preprodmacys.fds.com/tools/90067660.csv
  % Total    % Received % Xferd  Average Speed   Time    Time     Time
 Current
                                 Dload  Upload   Total   Spent    Left
 Speed
100  155k  100  155k    0     0   449k      0 --:--:-- --:--:-- --:--:--
 550k

address@hidden ~]$ more tr.out

== Info: About to connect() to www.preprod.abc.com port 80

== Info:   Trying 184.31.131.61... == Info: connected

== Info: Connected to www.preprod.abc.com (184.31.131.61) port 80

=> Send header, 186 bytes (0xba)

0000: GET /tools/90067660.csv HTTP/1.1

0022: User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5

0062:  OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5

008b: Host: www.preprod.abc.com

00ab: Accept: */*

00b8:

<= Recv header, 17 bytes (0x11)

0000: HTTP/1.1 200 OK

<= Recv header, 25 bytes (0x19)

0000: Server: IBM_HTTP_Server

<= Recv header, 46 bytes (0x2e)

0000: Last-Modified: Fri, 18 Mar 2011 22:42:46 GMT

<= Recv header, 30 bytes (0x1e)

0000: ETag: "b19b9-26f41-7f2b4580"

<= Recv header, 22 bytes (0x16)

0000: Accept-Ranges: bytes

<= Recv header, 26 bytes (0x1a)

0000: Content-Type: text/plain

<= Recv header, 37 bytes (0x25)

0000: Date: Fri, 14 Oct 2011 22:06:15 GMT

<= Recv header, 24 bytes (0x18)

0000: Content-Length: 159553

<= Recv header, 24 bytes (0x18)

0000: Connection: Keep-Alive

<= Recv header, 127 bytes (0x7f)

0000: Set-Cookie: BC_HA_7039BD33D34B38D1_70737B63=4117DB; Domain=.prep

0040: rodmacys.fds.com; expires=Fri, 14-Oct-11 22:36:15 GMT; Path=/

<= Recv header, 70 bytes (0x46)

0000: Set-Cookie: shippingCountry=US; path=/; domain=.preprod.abc

0040: .com

<= Recv header, 104 bytes (0x68)

0000: Set-Cookie: stop_mobi=yes; expires=Fri, 14-Oct-2011 22:36:15 GMT

0040: ; path=/; domain=.preprod.abc.com

<= Recv data, 5078 bytes (0x13d6)

0000: 90067660,po-xx-xx-xx,popup,

001d: 90067660,ex-xx-xx-xx,external sites,

0043: 90067660,im-xx-xx-xx,imports,
.........MORE LINES..............

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<


Also when we run wget from another server against the same URL it works
fine...the version on the other server is 1.10.4. So we got that version and
installed it on the server mentioned in the above example and tried but we
still got the same error message.

Following is the debug log from the successful run on the another server :
address@hidden /home/scripts(0)$ wget -d -S
http://www.preprod.abc.com/tools/90067660.csv

Setting --server-response (serverresponse) to 1

DEBUG output created by Wget 1.10.2 (Red Hat modified) on linux-gnu.


--15:03:42--  http://www.preprod.abc.com/tools/90067660.csv

           => `90067660.csv.3'

Resolving www.preprod.abc.com... 184.31.131.61

Caching www.preprod.abc.com => 184.31.131.61

Connecting to www.preprod.abc.com|184.31.131.61|:80... connected.

Created socket 3.

Releasing 0x000000000063b390 (new refcount 1).


---request begin---

GET /tools/90067660.csv HTTP/1.0

User-Agent: Wget/1.10.2 (Red Hat modified)

Accept: */*

Host: www.preprod.abc.com

Connection: Keep-Alive


---request end---

HTTP request sent, awaiting response...

---response begin---

HTTP/1.0 200 OK

Server: IBM_HTTP_Server

Last-Modified: Fri, 18 Mar 2011 22:42:46 GMT

ETag: "b19b9-26f41-7f2b4580"

Accept-Ranges: bytes

Content-Type: text/plain

Content-Length: 159553

Date: Fri, 14 Oct 2011 22:03:42 GMT

Connection: keep-alive

Set-Cookie: BC_HA_7039BD33D34B38D1_70737B63=41174D; Domain=.preprod.abc.com;
expires=Fri, 14-Oct-11 22:33:42 GMT; Path=/

Set-Cookie: shippingCountry=US; path=/; domain=.preprod.abc.com

Set-Cookie: stop_mobi=yes; expires=Fri, 14-Oct-2011 22:33:42 GMT; path=/;
domain=.preprod.abc.com


---response end---


  HTTP/1.0 200 OK

  Server: IBM_HTTP_Server

  Last-Modified: Fri, 18 Mar 2011 22:42:46 GMT

  ETag: "b19b9-26f41-7f2b4580"

  Accept-Ranges: bytes

  Content-Type: text/plain

  Content-Length: 159553

  Date: Fri, 14 Oct 2011 22:03:42 GMT

  Connection: keep-alive

  Set-Cookie: BC_HA_7039BD33D34B38D1_70737B63=41174D; Domain=.
preprod.abc.com; expires=Fri, 14-Oct-11 22:33:42 GMT; Path=/

  Set-Cookie: shippingCountry=US; path=/; domain=.preprod.abc.com

  Set-Cookie: stop_mobi=yes; expires=Fri, 14-Oct-2011 22:33:42 GMT; path=/;
domain=.preprod.abc.com

Registered socket 3 for persistent reuse.

cdm: 1 2 3 4 5 6 7 8

Stored cookie preprod.abc.com -1 (ANY) / <permanent> <insecure> [expiry
2011-10-14 15:33:42] BC_HA_7039BD33D34B38D1_70737B63 41174D

cdm: 1 2 3 4 5 6 7 8

Stored cookie preprod.abc.com -1 (ANY) / <session> <insecure> [expiry none]
shippingCountry US

cdm: 1 2 3 4 5 6 7 8

Stored cookie preprod.abc.com -1 (ANY) / <permanent> <insecure> [expiry
2011-10-14 15:33:42] stop_mobi yes

Length: 159,553 (156K) [text/plain]


100%[=================================================================================================================================================================================================>]
159,553      105.31K/s


15:03:44 (105.15 KB/s) - `90067660.csv.3' saved [159553/159553]



PS : I changed the domain names mentioned in the logs above

Thanks,
Vishwanath
reply via email to
[Prev in Thread] Current Thread [Next in Thread]