Re: [Bug-wget] Unexpected wget -N behaviour for 1.17 onwards?

[Lawrence Wade]

Hi Tim,

Okay. Using the OpenSUSE-packaged wget (1.19.5) that comes with Leap 15.0:

$ wget -r -N 192.168.2.100:8080
...
Reusing existing connection to 192.168.2.100:8080.
HTTP request sent, awaiting response... 304 Not Modified
File ‘192.168.2.100:8080/OaP6ysTyz6Y.mp
4’ not modified on server. Omitting download.

This file is incomplete in my local copy.

Trying again as you suggest,

$ wget -r -N --no-if-modified-since 192.168.2.100:8080
...
--2019-02-10 08:35:14--  http://192.168.2.100:8080/OaP6ysTyz6Y.mp4
Reusing existing connection to 192.168.2.100:8080.
HTTP request sent, awaiting response... 200 OK
Length: 38044195 (36M) [application/octet-stream]
The sizes do not match (local 8643456) -- retrieving.
--2019-02-10 08:35:14--  http://192.168.2.100:8080/OaP6ysTyz6Y.mp4
Reusing existing connection to 192.168.2.100:8080.
HTTP request sent, awaiting response... 200 OK
Length: 38044195 (36M) [application/octet-stream]
Saving to: ‘192.168.2.100:8080/OaP6ysTy
z6Y.mp4
...

And it appears to work as expected. Won't this change to the behaviour
of -N option subtly break a lot of scripts which rely on wget?

Thanks so much, Tim. I do have an answer and a workaround though my
concerns remain.

Lawrence Wade
Ottawa, Canada

On Sun, Feb 10, 2019 at 2:11 AM Lawrence Wade <address@hidden> wrote:
>
> Hi Everyone,
>
> This might be a corroboration of this
> http://lists.gnu.org/archive/html/bug-wget/2018-10/msg00049.html
> and this
> https://bugs.launchpad.net/ubuntu/+source/wget/+bug/1715481
>
> I use wget to backup my cellphone running Palapa Web Server, and it
> has worked well for me for years. Since upgrading to OpenSUSE Leap 15,
> I have been having corrupted files.
>
> My method is
> $ wget -r -N 192.168.2.100:8080
> and if the connection is interrupted for any reason, the next time I
> call wget it would complete any incomplete files. And since Leap 15, I
> have been getting gradually corrupted backups. I was tearing my hair
> out looking at wgetrc and other things.
>
> With one long file that I knew was incomplete, I got a Not Modified -
> omitting download, even though I knew the file sizes were different
> between the server and wget's copy - though the wget man page
> explicitly states that if the file sizes do not match, -N will trigger
> a download.
>
> I tried on OpenSUSE 42.3 (wget 1.14) and the incomplete file triggered
> a download, even though wgetrc was identical.
>
> Again, on Leap 15, I compiled 1.20.1 (latest), 1.17.1, and then
> finally with 1.16.3 the behaviour went back to what I expected (and I
> got my corrupted phone backups fixed).
>
> Was a bug possibly introduced in 1.17 with the support for 
> --if-modified-since?
>
> Version shipping with OpenSUSE Leap 15:
> GNU Wget 1.19.5 built on linux-gnu.
> +cares +digest +gpgme +https +ipv6 +iri +large-file +metalink +nls
> +ntlm +opie +psl +ssl/openssl
>
> Last version I tried where "wget -r -N" works as expected:
> GNU Wget 1.16.3 built on linux-gnu.
> +digest +https +ipv6 -iri +large-file +nls +ntlm +opie +psl +ssl/gnutls
>
> I'm open to the possibility that there may be something else causing
> this bug, I have not found many mentions of it, but then again it is
> subtle. You get pretty confident when you just let wget do its thing,
> so there may be a lot of incomplete files out there... :)
>
> Thanks so much for your help. I can provide any other info that would
> be helpful.
>
> Lawrence Wade
> Ottawa, Canada