Re: wget crashes on a recursive download of my city's website

[Tim Rühsen]

Nvm, just reproduced the crash (or another one), took no longer than 10 
mins.

I have no time right now to dig into it.
Just in case someone else wants to chime in:

This was on branch `origin/dynamic-buf-size`, where origin is 
https://gitlab.com/gnuwget/wget.git. Built with ./configure 
--with-ssl=openssl.

$ ../src/wget --version
GNU Wget 1.21.3.1-9f93 built on linux-gnu.

-cares +digest +gpgme +https +ipv6 +iri +large-file +metalink +nls
+ntlm +opie +psl +ssl/openssl


##############
$ valgrind ../src/wget -r --tries=3 -c -E --preserve-permissions 
--no-parent https://www.concordnh.gov/iCalendar.aspx -o log
==4462== Memcheck, a memory error detector
==4462== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==4462== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==4462== Command: ../src/wget -r --tries=3 -c -E --preserve-permissions 
--no-parent https://www.concordnh.gov/iCalendar.aspx -o log
==4462==
==4462== Invalid write of size 2
==4462==    at 0x48493C3: memmove (in 
/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==4462==    by 0x4B855E1: ??? (in /usr/lib/x86_64-linux-gnu/libssl.so.1.1)
==4462==    by 0x4B8C4A4: ??? (in /usr/lib/x86_64-linux-gnu/libssl.so.1.1)
==4462==    by 0x4B973C2: SSL_read (in 
/usr/lib/x86_64-linux-gnu/libssl.so.1.1)
==4462==    by 0x149AB5: openssl_read_peek (openssl.c:643)
==4462==    by 0x149C00: openssl_read (openssl.c:662)
==4462==    by 0x115C3A: fd_read (connect.c:947)
==4462==    by 0x13B9DF: fd_read_body (retr.c:446)
==4462==    by 0x12B855: read_response_body (http.c:1734)
==4462==    by 0x12F503: gethttp (http.c:4184)
==4462==    by 0x12F909: http_loop (http.c:4423)
==4462==    by 0x13BF99: retrieve_url (retr.c:1026)
==4462==  Address 0x7507950 is 0 bytes after a block of size 8,192 alloc'd
==4462==    at 0x483F7B5: malloc (in 
/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==4462==    by 0x164C8A: xmalloc (xmalloc.c:44)
==4462==    by 0x13B34A: fd_read_body (retr.c:263)
==4462==    by 0x12B855: read_response_body (http.c:1734)
==4462==    by 0x12F503: gethttp (http.c:4184)
==4462==    by 0x12F909: http_loop (http.c:4423)
==4462==    by 0x13BF99: retrieve_url (retr.c:1026)
==4462==    by 0x139F2B: retrieve_tree (recur.c:332)
==4462==    by 0x136064: main (main.c:2167)
==4462==
==4462== Syscall param write(buf) points to unaddressable byte(s)
==4462==    at 0x5055603: write (write.c:26)
==4462==    by 0x4FE6A94: _IO_file_write@@GLIBC_2.2.5 (fileops.c:1181)
==4462==    by 0x4FE5E25: new_do_write (fileops.c:449)
==4462==    by 0x4FE718D: _IO_new_file_xsputn (fileops.c:1255)
==4462==    by 0x4FE718D: _IO_file_xsputn@@GLIBC_2.2.5 (fileops.c:1197)
==4462==    by 0x4FDBDDC: fwrite (iofwrite.c:39)
==4462==    by 0x13ADCE: write_data (retr.c:191)
==4462==    by 0x13BA5A: fd_read_body (retr.c:524)
==4462==    by 0x12B855: read_response_body (http.c:1734)
==4462==    by 0x12F503: gethttp (http.c:4184)
==4462==    by 0x12F909: http_loop (http.c:4423)
==4462==    by 0x13BF99: retrieve_url (retr.c:1026)
==4462==    by 0x139F2B: retrieve_tree (recur.c:332)
==4462==  Address 0x7507950 is 0 bytes after a block of size 8,192 alloc'd
==4462==    at 0x483F7B5: malloc (in 
/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==4462==    by 0x164C8A: xmalloc (xmalloc.c:44)
==4462==    by 0x13B34A: fd_read_body (retr.c:263)
==4462==    by 0x12B855: read_response_body (http.c:1734)
==4462==    by 0x12F503: gethttp (http.c:4184)
==4462==    by 0x12F909: http_loop (http.c:4423)
==4462==    by 0x13BF99: retrieve_url (retr.c:1026)
==4462==    by 0x139F2B: retrieve_tree (recur.c:332)
==4462==    by 0x136064: main (main.c:2167)
==4462==
==4462== Invalid read of size 1
==4462==    at 0x4849EB0: mempcpy (in 
/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==4462==    by 0x4FE9017: _IO_default_xsputn (genops.c:386)
==4462==    by 0x4FE9017: _IO_default_xsputn (genops.c:370)
==4462==    by 0x4FE7152: _IO_new_file_xsputn (fileops.c:1265)
==4462==    by 0x4FE7152: _IO_file_xsputn@@GLIBC_2.2.5 (fileops.c:1197)
==4462==    by 0x4FDBDDC: fwrite (iofwrite.c:39)
==4462==    by 0x13ADCE: write_data (retr.c:191)
==4462==    by 0x13BA5A: fd_read_body (retr.c:524)
==4462==    by 0x12B855: read_response_body (http.c:1734)
==4462==    by 0x12F503: gethttp (http.c:4184)
==4462==    by 0x12F909: http_loop (http.c:4423)
==4462==    by 0x13BF99: retrieve_url (retr.c:1026)
==4462==    by 0x139F2B: retrieve_tree (recur.c:332)
==4462==    by 0x136064: main (main.c:2167)
==4462==  Address 0x7508950 is 16 bytes before a block of size 40 free'd
==4462==    at 0x484217B: free (in 
/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==4462==    by 0x125DFB: tagstack_pop (html-parse.c:322)
==4462==    by 0x126776: map_html_tags (html-parse.c:1147)
==4462==    by 0x1275D2: get_urls_html_fm (html-url.c:833)
==4462==    by 0x1276C9: get_urls_html (html-url.c:868)
==4462==    by 0x139B69: retrieve_tree (recur.c:426)
==4462==    by 0x136064: main (main.c:2167)
==4462==  Block was alloc'd at
==4462==    at 0x483F7B5: malloc (in 
/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==4462==    by 0x164C8A: xmalloc (xmalloc.c:44)
==4462==    by 0x125A38: tagstack_push (html-parse.c:286)
==4462==    by 0x126186: map_html_tags (html-parse.c:930)
==4462==    by 0x1275D2: get_urls_html_fm (html-url.c:833)
==4462==    by 0x1276C9: get_urls_html (html-url.c:868)
==4462==    by 0x139B69: retrieve_tree (recur.c:426)
==4462==    by 0x136064: main (main.c:2167)
==4462==
==4462== Invalid read of size 1
==4462==    at 0x4849EBE: mempcpy (in 
/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==4462==    by 0x4FE9017: _IO_default_xsputn (genops.c:386)
==4462==    by 0x4FE9017: _IO_default_xsputn (genops.c:370)
==4462==    by 0x4FE7152: _IO_new_file_xsputn (fileops.c:1265)
==4462==    by 0x4FE7152: _IO_file_xsputn@@GLIBC_2.2.5 (fileops.c:1197)
==4462==    by 0x4FDBDDC: fwrite (iofwrite.c:39)
==4462==    by 0x13ADCE: write_data (retr.c:191)
==4462==    by 0x13BA5A: fd_read_body (retr.c:524)
==4462==    by 0x12B855: read_response_body (http.c:1734)
==4462==    by 0x12F503: gethttp (http.c:4184)
==4462==    by 0x12F909: http_loop (http.c:4423)
==4462==    by 0x13BF99: retrieve_url (retr.c:1026)
==4462==    by 0x139F2B: retrieve_tree (recur.c:332)
==4462==    by 0x136064: main (main.c:2167)
==4462==  Address 0x7508952 is 14 bytes before a block of size 40 free'd
==4462==    at 0x484217B: free (in 
/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==4462==    by 0x125DFB: tagstack_pop (html-parse.c:322)
==4462==    by 0x126776: map_html_tags (html-parse.c:1147)
==4462==    by 0x1275D2: get_urls_html_fm (html-url.c:833)
==4462==    by 0x1276C9: get_urls_html (html-url.c:868)
==4462==    by 0x139B69: retrieve_tree (recur.c:426)
==4462==    by 0x136064: main (main.c:2167)
==4462==  Block was alloc'd at
==4462==    at 0x483F7B5: malloc (in 
/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==4462==    by 0x164C8A: xmalloc (xmalloc.c:44)
==4462==    by 0x125A38: tagstack_push (html-parse.c:286)
==4462==    by 0x126186: map_html_tags (html-parse.c:930)
==4462==    by 0x1275D2: get_urls_html_fm (html-url.c:833)
==4462==    by 0x1276C9: get_urls_html (html-url.c:868)
==4462==    by 0x139B69: retrieve_tree (recur.c:426)
==4462==    by 0x136064: main (main.c:2167)
==4462==
==4462== Invalid write of size 1
==4462==    at 0x48493F3: memmove (in 
/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==4462==    by 0x4B855E1: ??? (in /usr/lib/x86_64-linux-gnu/libssl.so.1.1)
==4462==    by 0x4B8C4A4: ??? (in /usr/lib/x86_64-linux-gnu/libssl.so.1.1)
==4462==    by 0x4B973C2: SSL_read (in 
/usr/lib/x86_64-linux-gnu/libssl.so.1.1)
==4462==    by 0x149AB5: openssl_read_peek (openssl.c:643)
==4462==    by 0x149C00: openssl_read (openssl.c:662)
==4462==    by 0x115C3A: fd_read (connect.c:947)
==4462==    by 0x13B9DF: fd_read_body (retr.c:446)
==4462==    by 0x12B855: read_response_body (http.c:1734)
==4462==    by 0x12F503: gethttp (http.c:4184)
==4462==    by 0x12F909: http_loop (http.c:4423)
==4462==    by 0x13BF99: retrieve_url (retr.c:1026)
==4462==  Address 0x75081e6 is 70 bytes inside a block of size 96 free'd
==4462==    at 0x484217B: free (in 
/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==4462==    by 0x13E399: url_free (url.c:1238)
==4462==    by 0x13C7F4: free_urlpos (retr.c:1442)
==4462==    by 0x139D43: retrieve_tree (recur.c:494)
==4462==    by 0x136064: main (main.c:2167)
==4462==  Block was alloc'd at
==4462==    at 0x48445EF: calloc (in 
/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==4462==    by 0x164F41: xcalloc (xmalloc.c:297)
==4462==    by 0x13F2BA: url_parse (url.c:905)
==4462==    by 0x126AF6: append_url (html-url.c:335)
==4462==    by 0x12732A: tag_find_urls (html-url.c:463)
==4462==    by 0x1268CD: collect_tags_mapper (html-url.c:785)
==4462==    by 0x126714: map_html_tags (html-parse.c:1151)
==4462==    by 0x1275D2: get_urls_html_fm (html-url.c:833)
==4462==    by 0x1276C9: get_urls_html (html-url.c:868)
==4462==    by 0x139B69: retrieve_tree (recur.c:426)
==4462==    by 0x136064: main (main.c:2167)
==4462==
--4462-- VALGRIND INTERNAL ERROR: Valgrind received a signal 11 
(SIGSEGV) - exiting
--4462-- si_code=128;  Faulting address: 0x0;  sp: 0x1002e8ddc0

valgrind: the 'impossible' happened:
   Killed by fatal signal

host stacktrace:
==4462==    at 0x5804A36C: ??? (in 
/usr/libexec/valgrind/memcheck-amd64-linux)
==4462==    by 0x5804A98E: ??? (in 
/usr/libexec/valgrind/memcheck-amd64-linux)
==4462==    by 0x58004B1B: ??? (in 
/usr/libexec/valgrind/memcheck-amd64-linux)
==4462==    by 0x58004FF6: ??? (in 
/usr/libexec/valgrind/memcheck-amd64-linux)
==4462==    by 0x5800518D: ??? (in 
/usr/libexec/valgrind/memcheck-amd64-linux)
==4462==    by 0x58099E5C: ??? (in 
/usr/libexec/valgrind/memcheck-amd64-linux)
==4462==    by 0x580E1BCC: ??? (in 
/usr/libexec/valgrind/memcheck-amd64-linux)

sched status:
  running_tid=1

Thread 1: status = VgTs_Runnable (lwpid 4462)
==4462==    at 0x483F6C5: malloc (in 
/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==4462==    by 0x164CCC: xrealloc (xmalloc.c:65)
==4462==    by 0x13D65F: append_string (url.c:1361)
==4462==    by 0x13E63F: url_file_name (url.c:1826)
==4462==    by 0x12F784: http_loop (http.c:4284)
==4462==    by 0x13BF99: retrieve_url (retr.c:1026)
==4462==    by 0x139F2B: retrieve_tree (recur.c:332)
==4462==    by 0x136064: main (main.c:2167)
client stack range: [0x1FFEFFB000 0x1FFF000FFF] client SP: 0x1FFEFFF340
valgrind stack range: [0x1002D8E000 0x1002E8DFFF] top usage: 19080 of 
1048576


Note: see also the FAQ in the source distribution.
It contains workarounds to several common problems.
In particular, if Valgrind aborted or crashed after
identifying problems in your program, there's a good chance
that fixing those problems will prevent Valgrind aborting or
crashing, especially if it happened in m_mallocfree.c.

If that doesn't help, please report this bug to: www.valgrind.org

In the bug report, send all the above text, the valgrind
version, and what OS and version you are using.  Thanks.

OpenPGP_signature
Description: OpenPGP digital signature