Re: Warn that --compression=gzip is not enforced

[Tim Rühsen]

On 30.07.21 03:30, 積丹尼 Dan Jacobson wrote:
> The man page should mention for --compression=gzip,
> the website can still ignore it and send the whole file uncompressed.
> And wget is not picky, and still will accept it. So better have extra
> disk space ready. Tested with
> https://www.ncc.gov.tw/chinese/files/opendata/radio.csv .

This is because the client just tells the server what it supports / 
accepts (not what it desperately wants).

The issue here is that the "identity" encoding is always implicit - 
there is no way to disable it.

So a
  Accept-Encoding: gzip
means "hey server, send me the content either not encoded or gzip 
encoded - whatever you like :-)"

If you know that the server sends uncompressed content, you can compress 
it yourself on-the-fly to avoid excessive disk space usage. At least it 
works for single files:
  wget -O- <URL> | gzip > radio.csv.gz

Printing a warning may help some users, but others will be swamped, e.g. 
when doing recursive downloads.

Regards, Tim

OpenPGP_signature
Description: OpenPGP digital signature