Re: [Bug-wget] wget -crNl inf --- filenames mangled

[Tim Rühsen]

On 2/18/19 12:02 AM, Andres Valloud wrote:
> Hi, so I ran wget like this:
> 
> wget --no-check-certificate -dcrNl inf $baseUrl/root/pub/mods/2012/ -P
> $baseLocal -o wget-mods-2012.log
> 
> Looking at the log, '1f43' appears (I think) as a consequence of -l inf,
> because .../mods/2012/ has a reference to .../mods/, which leads wget to
> read the entire .../mods/ index.

Use -np / --no-parent if you don't want to ascend to the parent directory.


> According to my understanding of the log file, wget then collects all
> the possible URLs from .../mods/.  It is here that, after what seems
> like thousands of file, a single merge log entry shows '1f43' (some path
> parts elided).

'1f43' is part of a 'chunked' download. I made some tests printing out
the raw received payload of /root/pub/mods/index.html. Seeing this in a
downloaded file is clearly a bug.

But I can't reproduce it with your command sequence. The different index
files in /root/pub/mods/ all have a size of 997015 here. Even after
several retries. Maybe you can send me via PM a full working command
sequence using a fresh / clean directory. To further reduce the
downloads, try with -R '*.mp3,*.xm,*.ogg,*.mod,*.it,*.spc'.


Regards, Tim

> .../root/pub/mods/index.html?C=N;O=D:
> merge(‘.../root/pub/mods/?C=N;O=D’, ‘lizardking_-_quest.mp31f43’) ->
> .../root/pub/mods/lizardking_-_quest.mp31f43
> appending ‘.../root/pub/mods/lizardking_-_quest.mp31f43’ to urlpos.
> 
> Then I issued the command (some path parts elided)
> 
> wget --no-check-certificate .../root/pub/mods/
> 
> which resulted in a 974kb index.html file that has no occurrences of
> '1f43' (more on this request down below).
> 
> I wondered whether this could be happening because there are .html files
> that *do* have '1f43' already downloaded in the local downloading
> directory.  That is, will wget look at existing files, or will it
> download them from scratch?  But the log file seems to indicate the
> index.html was downloaded from scratch, not examined from disk.
> 
> The "bad" request looks like this (some path parts elided):
> 
> ---request begin---
> GET /root/pub/mods/?C=N;O=D HTTP/1.1^M
> Referer: .../root/pub/mods/^M
> If-Modified-Since: Sun, 10 Feb 2019 02:33:09 GMT^M
> Range: bytes=998575-^M
> User-Agent: Wget/1.20.1 (linux-gnu)^M
> Accept: */*^M
> Accept-Encoding: identity^M
> Host: saphirjd.me^M
> Connection: Keep-Alive^M
> ^M
> ---request end---
> HTTP request sent, awaiting response...
> ---response begin---
> HTTP/1.1 200 OK^M
> Date: Sat, 16 Feb 2019 21:51:21 GMT^M
> Server: Apache/2.4.23 (Win64) OpenSSL/1.0.2h^M
> Keep-Alive: timeout=2, max=18^M
> Connection: Keep-Alive^M
> Transfer-Encoding: chunked^M
> Content-Type: text/html;charset=UTF-8^M
> ^M
> ---response end---
> 200 OK
> Length: unspecified [text/html]
> Saving to: ‘.../root/pub/mods/index.html?C=N;O=D’
> 
>      0K .......... .......... .......... .......... ..........  234K
>     50K .......... .......... .......... .......... .......... 11.6M
>    100K .......... .......... .......... .......... .......... 14.4M
>    150K .......... .......... .......... .......... ..........  238K
>    200K .......... .......... .......... .......... ..........  657K
>    250K .......... .......... .......... .......... .......... 11.3M
>    300K .......... .......... .......... .......... .......... 8.44M
>    350K .......... .......... .......... .......... ..........  397K
>    400K .......... .......... .......... .......... ..........  627K
>    450K .......... .......... .......... .......... .......... 2.38M
>    500K .......... .......... .......... .......... .......... 4.47M
>    550K .......... .......... .......... .......... .......... 3.46M
>    600K .......... .......... .......... .......... ..........  477K
>    650K .......... .......... .......... .......... .......... 4.14M
>    700K .......... .......... .......... .......... ..........  717K
>    750K .......... .......... .......... .......... .......... 3.50M
>    800K .......... .......... .......... .......... .......... 3.01M
>    850K .......... .......... .......... .......... .......... 4.40M
>    900K .......... .......... .......... .......... .......... 2.69M
>    950K .......... .......... ...                              68.9K=1.4s
> 
> Last-modified header missing -- time-stamps turned off.
> 2019-02-16 13:51:25 (717 KB/s) - ‘.../root/pub/mods/index.html?C=N;O=D’
> saved [998575]
> 
> Loaded .../root/pub/mods/index.html?C=N;O=D (size 998575).
> 
> 
> The "good" request looks like this:
> 
> ---request begin---
> GET /root/pub/mods/ HTTP/1.1^M
> User-Agent: Wget/1.20.1 (linux-gnu)^M
> Accept: */*^M
> Accept-Encoding: identity^M
> Host: saphirjd.me^M
> Connection: Keep-Alive^M
> ^M
> ---request end---
> HTTP request sent, awaiting response...
> ---response begin---
> HTTP/1.1 200 OK^M
> Date: Sun, 17 Feb 2019 22:42:04 GMT^M
> Server: Apache/2.4.23 (Win64) OpenSSL/1.0.2h^M
> Keep-Alive: timeout=2, max=25^M
> Connection: Keep-Alive^M
> Transfer-Encoding: chunked^M
> Content-Type: text/html;charset=UTF-8^M
> ^M
> ---response end---
> 200 OK
> Registered socket 5 for persistent reuse.
> Length: unspecified [text/html]
> Saving to: ‘index.html.1’
> 
>      0K .......... .......... .......... .......... .......... 71.1K
>     50K .......... .......... .......... .......... ..........  221K
>    100K .......... .......... .......... .......... ..........  241K
>    150K .......... .......... .......... .......... ..........  232K
>    200K .......... .......... .......... .......... .......... 4.81M
>    250K .......... .......... .......... .......... .......... 1.64M
>    300K .......... .......... .......... .......... ..........  249K
>    350K .......... .......... .......... .......... .......... 2.49M
>    400K .......... .......... .......... .......... .......... 3.71M
>    450K .......... .......... .......... .......... ..........  258K
>    500K .......... .......... .......... .......... .......... 1.41M
>    550K .......... .......... .......... .......... .......... 1.46M
>    600K .......... .......... .......... .......... .......... 2.32M
>    650K .......... .......... .......... .......... ..........  340K
>    700K .......... .......... .......... .......... .......... 2.19M
>    750K .......... .......... .......... .......... .......... 4.10M
>    800K .......... .......... .......... .......... .......... 2.68M
>    850K .......... .......... .......... .......... .......... 3.17M
>    900K .......... .......... .......... .......... .......... 3.22M
>    950K .......... .......... ...                              2.07M=2.1s
> 
> 2019-02-17 14:42:09 (453 KB/s) - ‘index.html.1’ saved [997015]
> 
> 
> So I examined the "bad" html file.  Unlike the "good" file, the "bad"
> file starts like this (contents enclosed by ====== bars):
> 
> ======================================================================
> 13a
> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
> <html><head>
> <title>416 Requested Range Not Satisfiable</title>
> </head><body>
> <h1>Requested Range Not Satisfiable</h1>
> <p>None of the range-specifier values in the Range
> request-header field overlap the current extent
> of the selected resource.</p>
> </body></html>
> 
> 0
> 
> HTTP/1.1 200 OK
> Date: Sun, 10 Feb 2019 02:33:04 GMT
> Server: Apache/2.4.23 (Win64) OpenSSL/1.0.2h
> Keep-Alive: timeout=2, max=24
> Connection: Keep-Alive
> Transfer-Encoding: chunked
> Content-Type: text/html;charset=UTF-8
> 
> ee3
> ======================================================================
> 
> 
> The "13a" and "ee3" characters are present in the file.  This data also
> seems to explain why the file saved to disk is about 1kb larger than the
> file downloaded individually.  It looks like the index.html file saved
> to disk contains (i.e. begins with) garbage from a different request
> that ended in 416.  After that prolog of apparent junk, the file proper
> seems to begin as expected --- but it also has several occurrences of
> '1f43'.
> 
> A vimdiff run on bad.html and good.html shows some order differences,
> seemingly a table replaced with '1f43', and things of that nature.  The
> structure of the differences is not immediately obvious, as there are
> very large sections that differ seemingly because the file was served in
> different order.
> 
> Andres.
> 
> 
> On 2/17/19 12:15, Tim Rühsen wrote:
>> On 16.02.19 23:02, Andres Valloud wrote:
>>> Tim,
>>>
>>> I limited the data from 99gb to 3.3gb, and just to the directory where
>>> I've seen the problem occurs.  The strange string '1f43' appears in this
>>> limited setup.  The '1f43' substring seems to appear deterministically
>>> depending on the file name (I have not checked *every* occurrence by
>>> hand).
>>>
>>> How should I track this down?
>>
>> I'd use -d -olog and leave away -k. If 1f43 still appears, we know it's
>> not because of wget's parsing or conversion. In this case it#s from the
>> server... check in which file 1f43 appears and find the request in the
>> log file.
>>
>> Then try to download that file with a single (non-recursive) wget
>> command. Check if 1f43 appears in there. If it doesn't, compare both
>> requests to see the difference.
>>
>> Let us know the results.
>>
>> Regards, Tim
>>
>>>
>>> Andres.
>>>
>>> On 2/14/19 04:03, Tim Rühsen wrote:
>>>> On 2/14/19 12:25 PM, Andres Valloud wrote:
>>>>> Tim,
>>>>>
>>>>> On 2/14/19 02:03, Tim Rühsen wrote:
>>>>>>> I looked at the downloaded html files with grep.  They do contain
>>>>>>> the
>>>>>>> substring "1f43", seemingly after a ^M character (I did not check
>>>>>>> every
>>>>>>> single occurrence).  Sometimes, the ^M character is within a file
>>>>>>> name
>>>>>>> such as this:
>>>>>>>
>>>>>>> <tr><td valign="top"><img src="https://some.url/icons/mp3ogg.png^M
>>>>>>> 1f43^M
>>>>>>> "
>>>>>>
>>>>>> If this is contained in the HTML file, then 'mp3ogg.png1f43' seems
>>>>>> correct. ^M is a Carriage Return (Microsoft uses ^M plus linefeed for
>>>>>> End-Of-Line (EOL). In a HTML file, EOL has no meaning - parsers
>>>>>> simply
>>>>>> ignore it. This is nothing that can be addressed with
>>>>>> --restrict-file-names.
>>>>>>
>>>>>> But to make sure, look at the original file by downloading it with
>>>>>> 'wget
>>>>>> <URL>'. Does the file have the above 'lf43'/^M stuff in it as well
>>>>>> ? If
>>>>>> so, we can't do much about it.
>>>>>>
>>>>>> If all looks ok in there, please attach both files so we can compare
>>>>>> and
>>>>>> possibly reproduce.
>>>>>>
>>>>>> If you set the 'User-Agent' header to e.g. "Mozilla/5.0 (X11; Linux
>>>>>> x86_64; rv:65.0) Gecko/20100101 Firefox/65.0", the server thinks the
>>>>>> request is coming via Firefox.
>>>>>> curl and wget have both the --user-agent option for this.
>>>>>>
>>>>>> Do you get a different file when using that option ?
>>>>>
>>>>> There was one additional detail to make this work.  Instead of
>>>>> placing a
>>>>> request for index.html, I had to ask curl to get just the directory
>>>>> name
>>>>> ending with a slash.  Then the server responded with (essentially)
>>>>> index.html.
>>>>
>>>> A web server might give different content on 'dir', 'dir/' and
>>>> 'dir/index.html'. This is sometimes puzzling and as you can see, 'dir/'
>>>> can't be used as filename - so we use 'dir/index.html' for that. Which
>>>> is not correct if the server serves 'dir/index.php' when we request
>>>> 'dir/'.
>>>>
>>>>>
>>>>> Both curl and wget retrieve index.html contents without '1f43' when
>>>>> asking for just that URL.  vimdiff says the retrieved files are
>>>>> identical.
>>>>
>>>> Try to start with this URL using your original wget command line. You
>>>> could add a quota (-Q) to limit the amount of data. In the hope to
>>>> reproduce your issue with far less files/data to be downloaded.
>>>>
>>>>> I am at a loss as to how to explain how the '1f43' problem appears
>>>>> when
>>>>> asking wget to update the mirror of the site (rather than
>>>>> downloading a
>>>>> single file).  I'll look at the log file tomorrow and see if I get
>>>>> more
>>>>> ideas.
>>>>
>>>> Try to reduce the needed amount of data to reproduce it.
>>>>
>>>> Regards, Tim
>>>>
>>
>

signature.asc
Description: OpenPGP digital signature