[bug #63308] wget uses http_proxy variable when I download HSTS http:// link

[Askar Safin]

URL:
  <https://savannah.gnu.org/bugs/?63308>

                 Summary: wget uses http_proxy variable when I download HSTS
http:// link
                 Project: GNU Wget
               Submitter: safinaskar
               Submitted: Wed 02 Nov 2022 06:10:57 PM UTC
                Category: None
                Severity: 3 - Normal
                Priority: 5 - Normal
                  Status: None
                 Privacy: Public
             Assigned to: None
         Originator Name: 
        Originator Email: 
             Open/Closed: Open
                 Release: None
         Discussion Lock: Any
        Operating System: GNU/Linux
         Reproducibility: Every Time
           Fixed Release: None
         Planned Release: None
              Regression: No
           Work Required: None
          Patch Included: No


    _______________________________________________________

Follow-up Comments:


-------------------------------------------------------
Date: Wed 02 Nov 2022 06:10:57 PM UTC By: Askar Safin <safinaskar>
When I try to download http:// link using wget and wget tries to download
https:// instead (because of HSTS policy), then wget tries to use proxy from
http_proxy variable, which is (I think) wrong.

Steps to reproduce:

1. Run "wget http://d-i.debian.org";. $http_proxy should not be set. This will
store d-i.debian.org to HSTS database
2. Verify that d-i.debian.org got to HSTS database ( "$HOME/.wget-hsts" )
3. Install "apt-cacher-ng". Don't modify its default configuration
4. Set http_proxy to "http://localhost:3142/"; (this is port used by
apt-cacher-ng)
5. Run "wget http://d-i.debian.org";. You will see this:

%%% begin %%%
root@f521fb232cf5:/# wget http://d-i.debian.org
URL transformed to HTTPS due to an HSTS policy
--2022-11-02 16:50:19--  https://d-i.debian.org/
Resolving localhost (localhost)... 127.0.0.1, ::1
Connecting to localhost (localhost)|127.0.0.1|:3142... connected.
Proxy tunneling failed: CONNECT denied (ask the admin to allow HTTPS
tunnels)Unable to establish SSL connection.
%%% end %%%

Wget version is 1.21.3.

As you can see URL transforms to https://d-i.debian.org/ and wget tries to
download this new URL using content of $http_proxy variable. But I think this
is wrong, because this new URL is https:// URL, and thus $http_proxy should
not be used.

If I run "wget https://d-i.debian.org";, then $http_proxy is (I think
correctly) ignored. But if I run "wget http://d-i.debian.org";, then URL
transforms to https:// one, and $http_proxy is used for that https:// URL,
which is (I think) wrong.

You can see full steps of reproducing with output (including output of
--debug) here: https://builds.sr.ht/~safinaskar/job/874219 . Click "view
manifest »" to see script (with comments).

Now let me say why I need all this.

I have a script, which downloads (using wget) files from various Debian
domains using various URLs, some http and some https. For speed I use
apt-cacher-ng and set $http_proxy when I run this script. I use apt-cacher-ng
in its default configuration. In the default configuration apt-cacher-ng
doesn't have any kind of https support.

So I set $http_proxy var, but don't set $https_proxy var. And this works. When
I download (using wget) files from http:// URLs, they are downloaded through
apt-cacher-ng. And when I download files from https:// URLs, then are
downloaded directly (and this is good, because apt-cacher-ng in its default
configuration doesn't support https). And this is exactly what I want.

But when I use "wget http://d-i.debian.org/...";, it fails (let's assume that
we already run "wget http://d-i.debian.org/..."; at least one time in the past,
i. e. let's assume the domain is already in HSTS database). Wget tries to
download https:// URL, not http:// one, and it tries to do this using proxy!
But the proxy doesn't support https, so the downloading fails. I. e. wget
tries to download https:// link using $http_proxy variable, which is (in my
opinion) wrong.

Of course, I can use some workaround, i. e. to configure apt-cacher-ng
differently or just to use "wget https://d-i.debian.org/..."; instead of "wget
http://d-i.debian.org/...";, but I still think there is a bug in wget







    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?63308>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/