Re: [Bug-wget] Some URLs are saved as gzip-compressed data

[Darshit Shah]

Hi Mark,

The server in this case seems to be badly configured. Wget's request to the
server looks like this:

> ---request begin---
> GET /wcsstore/CrucialSAS/firmware/mx100/MU03/MX100_MU03_Update.zip HTTP/1.1
> User-Agent: Wget/1.20.1 (linux-gnu)
> Accept: */*
> Accept-Encoding: identity
> Host: assets.crucial.com
> Connection: Keep-Alive

If you notice here, the Encoding requested by Wget is "indentity", that is,
please don't make any changes to the file. And yet, the server replies back
with a "Content-Encoding: gzip". The server is sending it in a format that Wget
did not request. Firefox on the other hand, does have an "Accept-Encoding"
header that contains gzip. So it is able to deal with the resulting data
correctly.

One thing you can do it, use the `--compression=gzip` option. This will make
Wget request a gzip encoding, causing the resulting file to be correct.
However, remember this is only a workaround for a broken server. And the
`--compression=gzip` is not a default since it doesn't always work as expected.
Sadly, there are too many servers out there that break the spec in weird ways
and we are unable to support all those cases at this point.


* address@hidden <address@hidden> [190308 23:20]:
> I'm testing wget 1.20.1 on Lubuntu 16.04 x86-64.
> 
> I noticed an issue when trying to download a certain URL which is
> a .zip archive. The file created by wget is gzip-compressed. But
> downloading using e.g. Firefox the URL is saved as a .zip file.
> 
> Example:
> 
> $ wget 
> https://assets.crucial.com/wcsstore/CrucialSAS/firmware/mx100/MU03/MX100_MU03_Update.zip
> --2019-03-08 20:24:31--  
> https://assets.crucial.com/wcsstore/CrucialSAS/firmware/mx100/MU03/MX100_MU03_Update.zip
> Resolving assets.crucial.com... 68.232.35.24
> Connecting to assets.crucial.com|68.232.35.24|:443... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 12408650 (12M) [application/zip]
> Saving to: ‘MX100_MU03_Update.zip’
> 
> MX100_MU03_Update.zip                  
> 100%[==========================================================================>]
>   11.83M   738KB/s    in 18s     
> 
> 2019-03-08 20:24:49 (680 KB/s) - ‘MX100_MU03_Update.zip’ saved 
> [12408650/12408650]
> 
> $ file MX100_MU03_Update.zip 
> MX100_MU03_Update.zip: gzip compressed data, from Unix
> 
> 
> Mark
> 
> 

-- 
Thanking You,
Darshit Shah
PGP Fingerprint: 7845 120B 07CB D8D6 ECE5 FF2B 2A17 43ED A91A 35B6

signature.asc
Description: PGP signature