Re: [Bug-wget] Help: Why wget wall clock time much higher than download time?

[Tim Rühsen]

On 6/17/19 10:32 PM, David Bodin wrote:
> *wget --page-requisites --span-hosts --convert-links --adjust-extension
> --execute robots=off --user-agent Mozilla
> --random-wait 
> https://www.invisionapp.com/inside-design/essential-steps-designing-empathy/
> <https://www.invisionapp.com/inside-design/essential-steps-designing-empathy/>*
> 
> This command above provides the following stats:
> 
> Total wall clock time: 35s
> 
> Downloaded: 248 files, 39M in 4.2s (9.36 MB/s)
> 
> This website takes about 5 seconds to download and display all files on a
> hard refresh in the browser.
> 
> Why is the wall clock time *significantly longer* than the download time
> and is there a way to make it faster?

First of all, --random-wait waits 0.5 to 1.5 seconds after each
downloaded page. Don't use it - there have been times when web servers
blocked fast clients, but that shouldn't be the case today.

Wget uses just one connection for downloading, no compression by
default, no http/2.

You can try Wget2 which uses as many parallel connections as you like,
uses compression by default and http/2 if possible. Depending on the
HTTP server, Wget2 is often 10x faster then Wget just with it's default
settings.

You find the latest Wget2 tarball at
https://gnuwget.gitlab.io/wget2/wget2-latest.tar.gz.

Instructions how to build at
https://gitlab.com/gnuwget/wget2/blob/master/README.md

Regards, Tim

signature.asc
Description: OpenPGP digital signature