[Bug-wget] wget 1.16.1 on Cygwin mangles %2B in redirects

[Benjamin Gilbert]

Hi,

On Cygwin, wget 1.16.1 (and current Git master) occasionally fails to fetch
release artifacts hosted on GitHub.  Example URL:

https://github.com/openslide/openslide/releases/download/v3.4.0/openslide-3.4.0.tar.xz

GitHub redirects the request to an Amazon S3 signed URL, and that signature
sometimes includes a percent-encoded "+" character.  In this case, wget
improperly decodes the "%2B" back to "+" before processing the redirect,
causing Amazon to return 403.  Wget then repeatedly retries the request to
GitHub until either 1) the redirect URL does not include a "+", or 2) 20
failed redirects have occurred.

This does not happen if IRI is compiled out, does not happen if wget is run
from the Cygwin shell, and does not happen on Linux.  It does occur if wget
is run from the Windows command shell (or, as I originally discovered it,
from a Buildbot process running under cygrunsrv).


wget version info:

C:\cygwin\home\user\wget>src\wget.exe -V
GNU Wget 1.16.1.36-01d5 built on cygwin.

+digest +https +ipv6 +iri +large-file +nls +ntlm +opie -psl +ssl/gnutls

Wgetrc:
    /usr/local/etc/wgetrc (system)
Locale:
    /usr/local/share/locale
Compile:
    gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/usr/local/etc/wgetrc"
    -DLOCALEDIR="/usr/local/share/locale" -I. -I../lib -I../lib
    -Iyes/include -I/usr/include/p11-kit-1 -DHAVE_LIBGNUTLS -DNDEBUG
Link:
    gcc -I/usr/include/p11-kit-1 -DHAVE_LIBGNUTLS -DNDEBUG -Lyes/lib
    -liconv -lintl -lpcre -luuid -lnettle -lgnutls -lz -lintl -liconv
    -lp11-kit -lgmp -lhogweed -lgmp -lnettle -ltasn1 -lp11-kit -lz -lz
    -lidn ftp-opie.o gnutls.o http-ntlm.o ../lib/libgnu.a

Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic <address@hidden>.
Please send bug reports and questions to <address@hidden>.



Sample failure:

C:\cygwin\home\user\wget>src\wget.exe
https://github.com/openslide/openslide/releases/download/v3.4.0/openslide-3.4.0.tar.xz
--2015-01-03 06:41:09--
https://github.com/openslide/openslide/releases/download/v3.4.0/openslide-3.4.0.tar.xz
Connecting to github.com (github.com)|192.30.252.131|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location:
https://s3.amazonaws.com/github-cloud/releases/827644/96a6c9f0-8629-11e3-9b05-739b7f3aed83.xz?response-content-disposition=attachment%3B%20filename%3Dopenslide-3.4.0.tar.xz&response-content-type=application/octet-stream&AWSAccessKeyId=AKIAISTNZFOVBIJMK3TQ&Expires=1420267329&Signature=5d6RHL2oIvoDkET%2BZSiGgT4ZCY0%3D
[following]
--2015-01-03 06:41:09--
https://s3.amazonaws.com/github-cloud/releases/827644/96a6c9f0-8629-11e3-9b05-739b7f3aed83.xz?response-content-disposition=attachment;%20filename=openslide-3.4.0.tar.xz&response-content-type=application/octet-stream&AWSAccessKeyId=AKIAISTNZFOVBIJMK3TQ&Expires=1420267329&Signature=5d6RHL2oIvoDkET+ZSiGgT4ZCY0=
Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.2.56|:443...
connected.

HTTP request sent, awaiting response... 403 Forbidden
2015-01-03 06:41:09 ERROR 403: Forbidden.

--2015-01-03 06:41:09--
https://github.com/openslide/openslide/releases/download/v3.4.0/openslide-3.4.0.tar.xz
Connecting to github.com (github.com)|192.30.252.131|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location:
https://s3.amazonaws.com/github-cloud/releases/827644/96a6c9f0-8629-11e3-9b05-739b7f3aed83.xz?response-content-disposition=attachment%3B%20filename%3Dopenslide-3.4.0.tar.xz&response-content-type=application/octet-stream&AWSAccessKeyId=AKIAISTNZFOVBIJMK3TQ&Expires=1420267330&Signature=MK1DBqMuFnPJ2BF0dOTSnPxDkw0%3D
[following]
--2015-01-03 06:41:10--
https://s3.amazonaws.com/github-cloud/releases/827644/96a6c9f0-8629-11e3-9b05-739b7f3aed83.xz?response-content-disposition=attachment;%20filename=openslide-3.4.0.tar.xz&response-content-type=application/octet-stream&AWSAccessKeyId=AKIAISTNZFOVBIJMK3TQ&Expires=1420267330&Signature=MK1DBqMuFnPJ2BF0dOTSnPxDkw0=
Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.2.56|:443...
connected.

HTTP request sent, awaiting response... 200 OK
Length: 401488 (392K) [application/octet-stream]
Saving to: 'openslide-3.4.0.tar.xz.2'

openslide-3.4.0.tar 100%[=====================>] 392.08K  --.-KB/s   in
0.08s

2015-01-03 06:41:10 (4.73 MB/s) - 'openslide-3.4.0.tar.xz.2' saved
[401488/40148
8]



Example run of the same binary from the Cygwin shell, with %2B properly
handled:

$ src/wget
https://github.com/openslide/openslide/releases/download/v3.4.0/openslide-3.4.0.tar.xz
--2015-01-03 06:42:26--
https://github.com/openslide/openslide/releases/download/v3.4.0/openslide-3.4.0.tar.xz
Resolving github.com (github.com)... 192.30.252.131
Connecting to github.com (github.com)|192.30.252.131|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location:
https://s3.amazonaws.com/github-cloud/releases/827644/96a6c9f0-8629-11e3-9b05-739b7f3aed83.xz?response-content-disposition=attachment%3B%20filename%3Dopenslide-3.4.0.tar.xz&response-content-type=application/octet-stream&AWSAccessKeyId=AKIAISTNZFOVBIJMK3TQ&Expires=1420267407&Signature=YB9b5SgW7eyUX%2BSWK5L4A0CxzdA%3D
[following]
--2015-01-03 06:42:27--
https://s3.amazonaws.com/github-cloud/releases/827644/96a6c9f0-8629-11e3-9b05-739b7f3aed83.xz?response-content-disposition=attachment%3B%20filename%3Dopenslide-3.4.0.tar.xz&response-content-type=application/octet-stream&AWSAccessKeyId=AKIAISTNZFOVBIJMK3TQ&Expires=1420267407&Signature=YB9b5SgW7eyUX%2BSWK5L4A0CxzdA%3D
Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.13.0
Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.13.0|:443...
connected.
HTTP request sent, awaiting response... 200 OK
Length: 401488 (392K) [application/octet-stream]
Saving to: ‘openslide-3.4.0.tar.xz.9’

openslide-3.4.0.tar 100%[=====================>] 392.08K  --.-KB/s   in 0.1s

2015-01-03 06:42:27 (3.66 MB/s) - ‘openslide-3.4.0.tar.xz.9’ saved
[401488/401488]



Thanks,
--Benjamin Gilbert