Re: [Bug-wget] Unexpected wget -N behaviour for 1.17 onwards?

[Tim Rühsen]

You are right, --if-modified-since changes -N behavior in case a file is
incomplete. --if-modified-since can't easily be fixed since the 304
response does not include file size information.

As you suggest, we should disable this option by default or at least
discuss the options we have.

Regards, Tim

On 2/10/19 2:42 PM, Lawrence Wade wrote:
> Hi Tim,
> 
> Okay. Using the OpenSUSE-packaged wget (1.19.5) that comes with Leap 15.0:
> 
> $ wget -r -N 192.168.2.100:8080
> ...
> Reusing existing connection to 192.168.2.100:8080.
> HTTP request sent, awaiting response... 304 Not Modified
> File ‘192.168.2.100:8080/OaP6ysTyz6Y.mp
> 4’ not modified on server. Omitting download.
> 
> This file is incomplete in my local copy.
> 
> Trying again as you suggest,
> 
> $ wget -r -N --no-if-modified-since 192.168.2.100:8080
> ...
> --2019-02-10 08:35:14--  http://192.168.2.100:8080/OaP6ysTyz6Y.mp4
> Reusing existing connection to 192.168.2.100:8080.
> HTTP request sent, awaiting response... 200 OK
> Length: 38044195 (36M) [application/octet-stream]
> The sizes do not match (local 8643456) -- retrieving.
> --2019-02-10 08:35:14--  http://192.168.2.100:8080/OaP6ysTyz6Y.mp4
> Reusing existing connection to 192.168.2.100:8080.
> HTTP request sent, awaiting response... 200 OK
> Length: 38044195 (36M) [application/octet-stream]
> Saving to: ‘192.168.2.100:8080/OaP6ysTy
> z6Y.mp4
> ...
> 
> And it appears to work as expected. Won't this change to the behaviour
> of -N option subtly break a lot of scripts which rely on wget?
> 
> Thanks so much, Tim. I do have an answer and a workaround though my
> concerns remain.
> 
> Lawrence Wade
> Ottawa, Canada
> 
> On Sun, Feb 10, 2019 at 2:11 AM Lawrence Wade <address@hidden> wrote:
>>
>> Hi Everyone,
>>
>> This might be a corroboration of this
>> http://lists.gnu.org/archive/html/bug-wget/2018-10/msg00049.html
>> and this
>> https://bugs.launchpad.net/ubuntu/+source/wget/+bug/1715481
>>
>> I use wget to backup my cellphone running Palapa Web Server, and it
>> has worked well for me for years. Since upgrading to OpenSUSE Leap 15,
>> I have been having corrupted files.
>>
>> My method is
>> $ wget -r -N 192.168.2.100:8080
>> and if the connection is interrupted for any reason, the next time I
>> call wget it would complete any incomplete files. And since Leap 15, I
>> have been getting gradually corrupted backups. I was tearing my hair
>> out looking at wgetrc and other things.
>>
>> With one long file that I knew was incomplete, I got a Not Modified -
>> omitting download, even though I knew the file sizes were different
>> between the server and wget's copy - though the wget man page
>> explicitly states that if the file sizes do not match, -N will trigger
>> a download.
>>
>> I tried on OpenSUSE 42.3 (wget 1.14) and the incomplete file triggered
>> a download, even though wgetrc was identical.
>>
>> Again, on Leap 15, I compiled 1.20.1 (latest), 1.17.1, and then
>> finally with 1.16.3 the behaviour went back to what I expected (and I
>> got my corrupted phone backups fixed).
>>
>> Was a bug possibly introduced in 1.17 with the support for 
>> --if-modified-since?
>>
>> Version shipping with OpenSUSE Leap 15:
>> GNU Wget 1.19.5 built on linux-gnu.
>> +cares +digest +gpgme +https +ipv6 +iri +large-file +metalink +nls
>> +ntlm +opie +psl +ssl/openssl
>>
>> Last version I tried where "wget -r -N" works as expected:
>> GNU Wget 1.16.3 built on linux-gnu.
>> +digest +https +ipv6 -iri +large-file +nls +ntlm +opie +psl +ssl/gnutls
>>
>> I'm open to the possibility that there may be something else causing
>> this bug, I have not found many mentions of it, but then again it is
>> subtle. You get pretty confident when you just let wget do its thing,
>> so there may be a lot of incomplete files out there... :)
>>
>> Thanks so much for your help. I can provide any other info that would
>> be helpful.
>>
>> Lawrence Wade
>> Ottawa, Canada
>

signature.asc
Description: OpenPGP digital signature