Re: [Bug-wget] downloading files in multiple computer

[Ángel González]

Rahul Prasad wrote:
> Problem problem problem :(
>
> i tried partially downloaded a file and terminated download by
> pressing Ctrl+c
> After that when I tried setting header to download rest of the file it
> kept on retrying.
> Here is an output
>
> address@hidden:~$ wget -c --header "Range: bytes=505137-12973463"
> --server-response  http://noya.co.in/download.zip
> --2010-11-27 20:29:23--  http://noya.co.in/download.zip
> Resolving noya.co.in... 67.228.53.112
> Connecting to noya.co.in|67.228.53.112|:80... connected.
> HTTP request sent, awaiting response...
>   HTTP/1.1 206 Partial Content
>   Date: Sat, 27 Nov 2010 15:00:00 GMT
>   Server: Apache/2.2.14 (Unix) mod_ssl/2.2.14
> OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4
> FrontPage/5.0.2.2635
>   Last-Modified: Tue, 13 Jul 2010 05:04:51 GMT
>   ETag: "55482ce-c5f597-48b3dcf1ea2c0"
>   Accept-Ranges: bytes
>   Content-Length: 12468326
>   Content-Range: bytes 505137-12973462/12973463
>   Keep-Alive: timeout=8
>   Connection: Keep-Alive
>   Content-Type: application/zip
> Retrying.
You must provide the beginning of the file. It is one of the reasons to
make you stop the download with Ctrl+C so that you got the beginning of
the file.

Note how for the second piece I made an empty file with the beginning,
for your case:
truncate -s505137 download.zip
truncate is provided by coreutils in Linux, any tool allowing you to
make a file with a given size, would work. -We later just discard that
piece. Depending on your filesystem, this will be a sparse file, so it
will take space only for eg. 1329 byte.


> Check the response header, its same in both the case still First
> method does not work.
>
> You may think my problem is solved if I use -c so I shouldnt try
> setting header "Range".
> But my problem is downloading same file partially in multiple
> computers and then combining it.
>
> Simply put, I want to intentionally download contents partially. is
> there any way I can do so using wget?


As I said in the first mail, it's completely hackish.
> computer1 $ wget http://noya.co.in/download.zip
> --2010-11-28 16:57:38--  http://noya.co.in/download.zip
> Resolving noya.co.in... 67.228.53.112
> Connecting to noya.co.in|67.228.53.112|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 12973463 (12M) [application/zip]
> Saving to: `download.zip'
>
>  2%
> [=>                                                                           
>                                     
> ] 294,799
> ^C

> computer1 $ ls -l download.zip  | cut -d' ' -f 5
> 335119
> computer1 $ wget -c --header "Range: bytes=335119-505137"
> http://noya.co.in/download.zip

> computer2 $ truncate -s505138 download.zip
> computer2 $ wget -c http://noya.co.in/download.zip
> --2010-11-28 17:01:19--  http://noya.co.in/download.zip
> Resolving noya.co.in... 67.228.53.112
> Connecting to noya.co.in|67.228.53.112|:80... connected.
> HTTP request sent, awaiting response... 206 Partial Content
> Length: 12973463 (12M), 12468325 (12M) remaining [application/zip]
> Saving to: `download.zip'
>
> 100%[++++=============================================================================================================>]
> 12,973,463
>
> 2010-11-28 17:02:05 - `download.zip' saved [12973463/12973463]

>  ( cat computer1/download.zip; tail -c +505139 computer2/download.zip
> )>  download.zip
> unzip -tq download.zip   
> No errors detected in compressed data of download.zip.