Re: wget crashes on a recursive download of my city's website

[Tim Rühsen]

On 06.06.22 23:07, Eric Gallager wrote:
> I was trying to use wget to download all of the calendar files on my
> city website: http://concordnh.gov/iCalendar.aspx
> Admittedly I'm kind of inexperienced at wget, so I probably didn't use
> the right command-line, but anyways it was:
> $ wget -r --tries=3 -c --random-wait -E --preserve-permissions
> --no-parent https://www.concordnh.gov/iCalendar.aspx
> Here is the tail end of the terminal output leading up to the crash:
>
> --2022-06-04 21:23:09--  http://www.concordnh.gov/1737/Concord-Municipal-Airport
> Reusing existing connection to www.concordnh.gov:80.
> HTTP request sent, awaiting response... 302 Found
> Location: http://www.concordnh.gov/344/Concord-Municipal-Airport [following]
> --2022-06-04 21:23:09--  http://www.concordnh.gov/344/Concord-Municipal-Airport
> Reusing existing connection to www.concordnh.gov:80.
> HTTP request sent, awaiting response... 200 OK
> Length: 96714 (94K) [text/html]
> Saving to: ‘www.concordnh.gov/1737/Concord-Municipal-Airport.html’
>
> www.concordnh.gov/1737/Concord-Municipal-Airport.html
> 100%[====================================================================================================================================================>]
>   94.45K   370KB/s    in 0.3s
>
> 2022-06-04 21:23:10 (370 KB/s) -
> ‘www.concordnh.gov/1737/Concord-Municipal-Airport.html’ saved
> [96714/96714]
>
> Segmentation fault: 11
> $
>
> Here is my wget --version info:
> $ wget --version
> GNU Wget 1.21.3 built on darwin20.6.0.
>
> -cares +digest -gpgme +https +ipv6 +iri +large-file -metalink +nls
> +ntlm +opie +psl +ssl/gnutls
>
> Wgetrc:
>      /Users/ericgallager/.wgetrc (user)
>      /opt/local/etc/wgetrc (system)
> Locale:
>      /opt/local/share/locale
> Compile:
>      /usr/bin/clang -DHAVE_CONFIG_H
>      -DSYSTEM_WGETRC="/opt/local/etc/wgetrc"
>      -DLOCALEDIR="/opt/local/share/locale" -I. -I../lib -I../lib
>      -I/opt/local/include
>      -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk
>      -DNDEBUG -pipe -Os -std=c99
>      -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk
>      -arch x86_64
> Link:
>      /usr/bin/clang -DNDEBUG -pipe -Os -std=c99
>      -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk
>      -arch x86_64 -L/opt/local/lib -Wl,-headerpad_max_install_names
>      -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk
>      -arch x86_64 -L/opt/local/lib -lpcre2-8 -L/opt/local/lib -lidn2
>      -L/opt/local/lib -lnettle /opt/local/lib/libgnutls.dylib
>      -L/opt/local/lib -lz -L/opt/local/lib -lpsl ../lib/libgnu.a
>      /opt/local/lib/libiconv.dylib /opt/local/lib/libintl.dylib
>      -Wl,-framework -Wl,CoreFoundation /opt/local/lib/libunistring.dylib
>
> Copyright (C) 2015 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://www.gnu.org/licenses/gpl.html>.
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
>
> Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
> Please send bug reports and questions to <bug-wget@gnu.org>.
> $
>
> I have it installed via MacPorts, and MacPorts tells me that this is
> the latest version:
> $ port -v livecheck wget
> wget seems to be up to date
> $
>
> It took several hours before the crash happened, so I don't really
> want to run the command again just to get a backtrace... any hints on
> how to reduce the command-line to get a minimized test-case are
> welcome.
>
> Thanks,
> Eric Gallager

Hi Eric,

for a backtrace you'll need the debug symbols for the wget binary. You 
can check that with the `file` command, e.g. `file /usr/bin/wget` 
(replace /usr/bin with the location of wget on your machine).

If you don't have the debug symbols, it doesn't make sense to run the 
wget command for a backtrace.

In case you have a binary with debug symbols, run the command over night 
with valgrind (valgrind wget -r ...).

You can likely drop --random-wait to speed up the download.

When the crash happened, did you start with a clean directory or did you 
continue the download ? I am asking because of the -c option - could 
help to figure out how to reproduce.

Another question: can you check whether your machine runs out of memory 
when running wget ? For a long download wget may take a lot of memory...

Regards, Tim

OpenPGP_signature
Description: OpenPGP digital signature