[bug #62869] if retry hits a 302 FOUND wget forgets to send the Range header thus appending the whole file to what's downloaded alrdy

[Emanuel Czirai]

URL:
  <https://savannah.gnu.org/bugs/?62869>

                 Summary: if retry hits a 302 FOUND wget forgets to send the
Range header thus appending the whole file to what's downloaded alrdy
                 Project: GNU Wget
               Submitter: correabuscar
               Submitted: Sat 06 Aug 2022 05:49:12 AM UTC
                Category: Program Logic
                Severity: 3 - Normal
                Priority: 5 - Normal
                  Status: None
                 Privacy: Public
             Assigned to: None
         Originator Name: 
        Originator Email: 
             Open/Closed: Open
                 Release: trunk
         Discussion Lock: Any
        Operating System: GNU/Linux
         Reproducibility: Every Time
           Fixed Release: None
         Planned Release: None
              Regression: None
           Work Required: None
          Patch Included: Yes


    _______________________________________________________

Follow-up Comments:


-------------------------------------------------------
Date: Sat 06 Aug 2022 05:49:12 AM UTC By: Emanuel Czirai <correabuscar>
Hello.
I've encountered this append bug on Gentoo with wget-1.21.3-r1 while portage
is downloading the file android-studio-2022.1.1.9-linux.tar.gz for Android
Studio Canary (a 1G file, which on disk was 1.6G and thus corrupt due to this
bug)

I've (not yet) attached file *problem_on_real_url.log* if you want to see wget
output the second time I've reproduced the above which yielded a file that was
24 MiB larger. I haven't redacted anything(like my IP address). I haven't
attached this yet, because only 4 files can be attached, if you really want to
see this let me know, I will attach in the next comment, but only if you need
to see it.

I couldn't reproduce it all the time because those google servers don't always
yield a 302 FOUND after a timeout and they don't always timeout either.

So I've come up with a test that always reproduces this issue (unfortunately,
I couldn't figure out how to make it a test case - test suite doesn't seem to
have the needed functionality): A server that pretends to timeout in the
middle of the transfer then when wget retries, it will give a 302 FOUND
<https://www.rfc-editor.org/rfc/rfc7231.html#section-6.4.3> and redirect to
another server and this is when wget forgets to send the Range header which
specifies from where should the server continue sending the file, thus the
server sends the full file from the beginning, and wget still acts as if the
file is being sent from the continue point, thus appending the full file to
whatever it already downloaded until the timeout(and the 302) occurred.

I've attached files:

a.py
go
tst
wget_no_append_on302_uponretry.patch


to run the test and check that the bug exists just first *chmod a+x go tst*
then run(as normal user, always):

./go

or
to see wget --debug output:

./go --debug

or

./go bug --debug



The last line should be a red color: "Bug still present!"

To see how wget acts when the server doesn't do a 302 redirect after a timeout
(ie. it never hits this bug) then run:

./go nobug --debug


This will always say as last line: "Bug is fixed."

To test both:

./tst


For this test script, if the bug is not fixed you get a yellow/brown last
line:
"ok, bug test is fine ie. wget isn't fixed (but it should eventually be, hence
why this is yellow)"

but if the bug is fixed, you get:
"Failed to reveal the bug, was the wget bug fixed?! (assume this is green if
you know that wget got fixed)"



The test wants to wget the file with contents "Hello World.\r\n" but the
server induces a timeout after "Hello " and this causes wget to retry, but the
server then gives a 302 which wget follows and then wget doesn't send a Range
header anymore causing the server to reply with 200 OK instead of 206 Partial
Content, thus the final file contents are "Hello Hello World.\r\n" when the
bug is present, thus showcasing the fact that the whole file(which is "Hello
World.\r\n") just got appended to whatever it already downloaded(which is the
first "Hello ")

Apply that attached patch to wget to see a proof of concept hacky fix which
makes wget do send a Range header after the 302 happens by pretending that
wget was ran with --start-pos=X arg, where X is the file offset it should've
continued from. It's a hack, not the actual fix.








    _______________________________________________________
File Attachments:


-------------------------------------------------------
Date: Sat 06 Aug 2022 05:49:12 AM UTC  Name:
wget_no_append_on302_uponretry.patch  Size: 1KiB   By: correabuscar
test for the bug presence and hacky poc patch
<http://savannah.gnu.org/bugs/download.php?file_id=53534>
-------------------------------------------------------
Date: Sat 06 Aug 2022 05:49:12 AM UTC  Name: go  Size: 993B   By: correabuscar
test for the bug presence and hacky poc patch
<http://savannah.gnu.org/bugs/download.php?file_id=53533>
-------------------------------------------------------
Date: Sat 06 Aug 2022 05:49:12 AM UTC  Name: tst  Size: 2KiB   By:
correabuscar
test for the bug presence and hacky poc patch
<http://savannah.gnu.org/bugs/download.php?file_id=53532>
-------------------------------------------------------
Date: Sat 06 Aug 2022 05:49:12 AM UTC  Name: a.py  Size: 8KiB   By:
correabuscar
test for the bug presence and hacky poc patch
<http://savannah.gnu.org/bugs/download.php?file_id=53531>

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?62869>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/