Re: [Bug-wget] New to this, large files constraints?

[Ray Satiro]

----- Original Message -----

> From: Veronica Watson <address@hidden>
> To: address@hidden
> Cc: 
> Sent: Thursday, September 15, 2011 6:04 PM
> Subject: [Bug-wget] New to this, large files constraints?
> 
> Hello,
> 
> I am trying to download a data file, but am unsuccessful. I have downloaded
> other files just fine, but I have a feeling that this one is either too
> large or too complex. The website itself may have to do something before the
> file is generated. However, when I paste the link into firefox, the file
> downloads easily. What should I do?
> 
> Here is the file:
> 
> http://socds.huduser.org/permits/output_monthly_csv.odb?outpref=csv&geoval=state&datatype=monthlyF&varlist=1%232%233&yearlist=2000%232001%232002%232003%232004%232005%232006%232007%232008%232009%232010&statelist=13%2337%2345&msalist=+&cbsalist=+&bppllist=+&cntylist=13033%2313073%2313189%2313245%2337007%2337025%2337071%2337119%2337179%2345001%2345003%2345005%2345007%2345009%2345011%2345013%2345015%2345017%2345019%2345021%2345023%2345025%2345027%2345029%2345031%2345033%2345035%2345037%2345039%2345041%2345043%2345045%2345047%2345049%2345051%2345053%2345055%2345057%2345059%2345061%2345063%2345067%2345069%2345065%2345071%2345073%2345075%2345077%2345079%2345081%2345083%2345085%2345087%2345089%2345091&COUNTYSUM=YES&COUNTYALL=+&COUNTYGRP=+&STATESUM=+&STATEALL=+&METROSUM=+&METROALL=+&METRO=+&CBSA=+&PLACEGRP=+&CSUMNAME=&JSUMNAME=+&geo=state&chron=monthlyF
> 
> Thanks,
> Veronica W.
>


On Windows you may see older versions of wget give the error message "Result 
too large" but it means filename too long. In Linux "File name too long". And 
wget 1.13 --trust-server-names doesn't work with this site's response.. should 
it?

What you can do is something like this, with your URL quoted
wget -O BuildingPermits.csv "http://yourhudurl";

wget -O BuildingPermits.csv 
"http://socds.huduser.org/permits/output_monthly_csv.odb?outpref=csv&geoval=state&datatype=monthlyF&varlist=1%232%233&yearlist=2000%232001%232002%232003%232004%232005%232006%232007%232008%232009%232010&statelist=13%2337%2345&msalist=+&cbsalist=+&bppllist=+&cntylist=13033%2313073%2313189%2313245%2337007%2337025%2337071%2337119%2337179%2345001%2345003%2345005%2345007%2345009%2345011%2345013%2345015%2345017%2345019%2345021%2345023%2345025%2345027%2345029%2345031%2345033%2345035%2345037%2345039%2345041%2345043%2345045%2345047%2345049%2345051%2345053%2345055%2345057%2345059%2345061%2345063%2345067%2345069%2345065%2345071%2345073%2345075%2345077%2345079%2345081%2345083%2345085%2345087%2345089%2345091&COUNTYSUM=YES&COUNTYALL=+&COUNTYGRP=+&STATESUM=+&STATEALL=+&METROSUM=+&METROALL=+&METRO=+&CBSA=+&PLACEGRP=+&CSUMNAME=&JSUMNAME=+&geo=state&chron=monthlyF";


---response begin---
HTTP/1.1 200 OK
Content-Type: application/vnd.ms-excel
Server: Microsoft-IIS/6.0
Content-Disposition: attachment; filename=BuildingPermits.csv;
X-Powered-By: ASP.NET
Date: Sat, 17 Sep 2011 05:58:06 GMT
Connection: close

---response end---
200 OK
Length: unspecified [application/vnd.ms-excel]
output_monthly_csv.odb?outpref=csv&geoval=state&datatype=monthlyF&varlist=1#2#3&yearlist=2000#2001#2002#2003#2004#2005#2006#2007#2008#2009#2010&statelist=13#37#45&msalist=+&cbsalist=+&bppllist=+&cntylist=13033#13073#13189#13245#37007#37025#37071#37119#37179#45001#45003#45005#45007#45009#45011#45013#45015#45017#45019#45021#45023#45025#45027#45029#45031#45033#45035#45037#45039#45041#45043#45045#45047#45049#45051#45053#45055#45057#45059#45061#45063#45067#45069#45065#45071#45073#45075#45077#45079#45081#45083#45085#45087#45089#45091&COUNTYSUM=YES&COUNTYALL=+&COUNTYGRP=+&STATESUM=+&STATEALL=+&METROSUM=+&METROALL=+&METRO=+&CBSA=+&PLACEGRP=+&CSUMNAME=&JSUMNAME=+&geo=state&chron=monthlyF:
 File name too long
Closed fd 3