Re: [Bug-wget] wget 1.16.1 on Cygwin mangles %2B in redirects

[Tim Rühsen]

A short test here on Linux can reproduce it:

LC_ALL="C" ../src/wget -d --restrict-file-names=windows 
"http://www.example.com/bla?Signature=xxx%2Bxxx";
...
URI encoding = 'ANSI_X3.4-1968'
converted 'http://www.example.com/bla?Signature=xxx%2Bxxx' (ANSI_X3.4-1968) -> 
'http://www.example.com/bla?Signature=xxx+xxx' (UTF-8)
...
GET /bla?Signature=xxx+xxx HTTP/1.1

But after a 4xx you can also see:
[IRI fallbacking to non-utf8 for 
'http://www.example.com/bla?Signature=xxx%2Bxxx'
...
GET /bla?Signature=xxx%2Bxxx HTTP/1.1
...

With this second try (also seen in your example output) the file is correctly 
downloaded and saved. Looks like an expected (or wanted) behavior of Wget, but 
can't say any more about it.

Tim


Am Samstag, 3. Januar 2015, 02:28:22 schrieb Benjamin Gilbert:
> Hi,
> 
> On Cygwin, wget 1.16.1 (and current Git master) occasionally fails to fetch
> release artifacts hosted on GitHub.  Example URL:
> 
> https://github.com/openslide/openslide/releases/download/v3.4.0/openslide-3.
> 4.0.tar.xz
> 
> GitHub redirects the request to an Amazon S3 signed URL, and that signature
> sometimes includes a percent-encoded "+" character.  In this case, wget
> improperly decodes the "%2B" back to "+" before processing the redirect,
> causing Amazon to return 403.  Wget then repeatedly retries the request to
> GitHub until either 1) the redirect URL does not include a "+", or 2) 20
> failed redirects have occurred.
> 
> This does not happen if IRI is compiled out, does not happen if wget is run
> from the Cygwin shell, and does not happen on Linux.  It does occur if wget
> is run from the Windows command shell (or, as I originally discovered it,
> from a Buildbot process running under cygrunsrv).
> 
> 
> wget version info:
> 
> C:\cygwin\home\user\wget>src\wget.exe -V
> GNU Wget 1.16.1.36-01d5 built on cygwin.
> 
> +digest +https +ipv6 +iri +large-file +nls +ntlm +opie -psl +ssl/gnutls
> 
> Wgetrc:
>     /usr/local/etc/wgetrc (system)
> Locale:
>     /usr/local/share/locale
> Compile:
>     gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/usr/local/etc/wgetrc"
>     -DLOCALEDIR="/usr/local/share/locale" -I. -I../lib -I../lib
>     -Iyes/include -I/usr/include/p11-kit-1 -DHAVE_LIBGNUTLS -DNDEBUG
> Link:
>     gcc -I/usr/include/p11-kit-1 -DHAVE_LIBGNUTLS -DNDEBUG -Lyes/lib
>     -liconv -lintl -lpcre -luuid -lnettle -lgnutls -lz -lintl -liconv
>     -lp11-kit -lgmp -lhogweed -lgmp -lnettle -ltasn1 -lp11-kit -lz -lz
>     -lidn ftp-opie.o gnutls.o http-ntlm.o ../lib/libgnu.a
> 
> Copyright (C) 2014 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://www.gnu.org/licenses/gpl.html>.
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> 
> Originally written by Hrvoje Niksic <address@hidden>.
> Please send bug reports and questions to <address@hidden>.
> 
> 
> 
> Sample failure:
> 
> C:\cygwin\home\user\wget>src\wget.exe
> https://github.com/openslide/openslide/releases/download/v3.4.0/openslide-3.
> 4.0.tar.xz --2015-01-03 06:41:09--
> https://github.com/openslide/openslide/releases/download/v3.4.0/openslide-3.
> 4.0.tar.xz Connecting to github.com (github.com)|192.30.252.131|:443...
> connected. HTTP request sent, awaiting response... 302 Found
> Location:
> https://s3.amazonaws.com/github-cloud/releases/827644/96a6c9f0-8629-11e3-9b0
> 5-739b7f3aed83.xz?response-content-disposition=attachment%3B%20filename%3Dop
> enslide-3.4.0.tar.xz&response-content-type=application/octet-stream&AWSAcces
> sKeyId=AKIAISTNZFOVBIJMK3TQ&Expires=1420267329&Signature=5d6RHL2oIvoDkET%2BZ
> SiGgT4ZCY0%3D [following]
> --2015-01-03 06:41:09--
> https://s3.amazonaws.com/github-cloud/releases/827644/96a6c9f0-8629-11e3-9b0
> 5-739b7f3aed83.xz?response-content-disposition=attachment;%20filename=opensl
> ide-3.4.0.tar.xz&response-content-type=application/octet-stream&AWSAccessKey
> Id=AKIAISTNZFOVBIJMK3TQ&Expires=1420267329&Signature=5d6RHL2oIvoDkET+ZSiGgT4
> ZCY0= Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.2.56|:443...
> connected.
> 
> HTTP request sent, awaiting response... 403 Forbidden
> 2015-01-03 06:41:09 ERROR 403: Forbidden.
> 
> --2015-01-03 06:41:09--
> https://github.com/openslide/openslide/releases/download/v3.4.0/openslide-3.
> 4.0.tar.xz Connecting to github.com (github.com)|192.30.252.131|:443...
> connected. HTTP request sent, awaiting response... 302 Found
> Location:
> https://s3.amazonaws.com/github-cloud/releases/827644/96a6c9f0-8629-11e3-9b0
> 5-739b7f3aed83.xz?response-content-disposition=attachment%3B%20filename%3Dop
> enslide-3.4.0.tar.xz&response-content-type=application/octet-stream&AWSAcces
> sKeyId=AKIAISTNZFOVBIJMK3TQ&Expires=1420267330&Signature=MK1DBqMuFnPJ2BF0dOT
> SnPxDkw0%3D [following]
> --2015-01-03 06:41:10--
> https://s3.amazonaws.com/github-cloud/releases/827644/96a6c9f0-8629-11e3-9b0
> 5-739b7f3aed83.xz?response-content-disposition=attachment;%20filename=opensl
> ide-3.4.0.tar.xz&response-content-type=application/octet-stream&AWSAccessKey
> Id=AKIAISTNZFOVBIJMK3TQ&Expires=1420267330&Signature=MK1DBqMuFnPJ2BF0dOTSnPx
> Dkw0= Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.2.56|:443...
> connected.
> 
> HTTP request sent, awaiting response... 200 OK
> Length: 401488 (392K) [application/octet-stream]
> Saving to: 'openslide-3.4.0.tar.xz.2'
> 
> openslide-3.4.0.tar 100%[=====================>] 392.08K  --.-KB/s   in
> 0.08s
> 
> 2015-01-03 06:41:10 (4.73 MB/s) - 'openslide-3.4.0.tar.xz.2' saved
> [401488/40148
> 8]
> 
> 
> 
> Example run of the same binary from the Cygwin shell, with %2B properly
> handled:
> 
> $ src/wget
> https://github.com/openslide/openslide/releases/download/v3.4.0/openslide-3.
> 4.0.tar.xz --2015-01-03 06:42:26--
> https://github.com/openslide/openslide/releases/download/v3.4.0/openslide-3.
> 4.0.tar.xz Resolving github.com (github.com)... 192.30.252.131
> Connecting to github.com (github.com)|192.30.252.131|:443... connected.
> HTTP request sent, awaiting response... 302 Found
> Location:
> https://s3.amazonaws.com/github-cloud/releases/827644/96a6c9f0-8629-11e3-9b0
> 5-739b7f3aed83.xz?response-content-disposition=attachment%3B%20filename%3Dop
> enslide-3.4.0.tar.xz&response-content-type=application/octet-stream&AWSAcces
> sKeyId=AKIAISTNZFOVBIJMK3TQ&Expires=1420267407&Signature=YB9b5SgW7eyUX%2BSWK
> 5L4A0CxzdA%3D [following]
> --2015-01-03 06:42:27--
> https://s3.amazonaws.com/github-cloud/releases/827644/96a6c9f0-8629-11e3-9b0
> 5-739b7f3aed83.xz?response-content-disposition=attachment%3B%20filename%3Dop
> enslide-3.4.0.tar.xz&response-content-type=application/octet-stream&AWSAcces
> sKeyId=AKIAISTNZFOVBIJMK3TQ&Expires=1420267407&Signature=YB9b5SgW7eyUX%2BSWK
> 5L4A0CxzdA%3D Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.13.0
> Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.13.0|:443...
> connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 401488 (392K) [application/octet-stream]
> Saving to: ‘openslide-3.4.0.tar.xz.9’
> 
> openslide-3.4.0.tar 100%[=====================>] 392.08K  --.-KB/s   in 0.1s
> 
> 2015-01-03 06:42:27 (3.66 MB/s) - ‘openslide-3.4.0.tar.xz.9’ saved
> [401488/401488]
> 
> 
> 
> Thanks,
> --Benjamin Gilbert

signature.asc
Description: This is a digitally signed message part.