bug#53941: 27.2; socks + tor dont work with https

[J.P.]

In light of recent related activity on the development mailing list [1],
I think it's worth summarizing where this bug stands, even though that
discussion seems to favor other approaches that I personally have little
interest in. As a reminder, this bug is about integrating the built-in
`socks' and `url' libraries to make proxying specific connections over
Tor mostly transparent and hassle free.

As it stands, the two biggest blockers I can see are both heavy on the
legwork side of things. The first and more daunting involves surveying
how the `url' library is used in the wild when it comes to proxies: what
interfaces people use and how that compares with what's prescribed in
the manual and what was promised in the original design discussions. The
goal would be to use these findings to enhance or overhaul the existing
interface and behavior and to affirm the result as canon. This will
likely involve reworking parts of the existing documentation and
saturating affected parts of the library in comprehensive test coverage.
It may also require central coordination to ensure interoperability
among consuming libraries and continuity of purpose among future
contributors.

The second and IMO easier (but still labor-intensive) task is fleshing
out and fortifying the actual networking implementation. What I propose
is that we start by leveraging the built-in logging facility of torsocks
itself [2]. By default, the verbose-level output is noisy but tolerable
after minimal modifications (e.g., by commenting out the "close" and
"fclose" statements). We'd need volunteers with access to the various
platforms Emacs supports (capable of running a Tor daemon) to run
through some contrived recipes, such as loading a site with `eww' and
fetching a package from an ELPA endpoint.

  $ LD_PRELOAD=/home/me/torsocks/src/lib/.libs/libtorsocks.so \
    TORSOCKS_LOG_FILE_PATH=/tmp/torsocks.log \
    TORSOCKS_LOG_LEVEL=5 ./src/emacs -Q

What we'd be looking for in the output is activity in those libc "GAI"
functions shadowed by the program.

While this methodology seems hokey, I think it's actually preferable (at
least as a starting point) over more traditional tracing solutions
because with the latter we'd still have to isolate the set of
torsocks-shadowed functions in whatever recording is produced, and then
filter out all the false positives from things like connect(2) calls for
AF_LOCAL/AF_UNIX, which we don't care about. Moreover, these
hand-crafted logs show us other niceties, like parameters of interest
(socket type, hint flags, etc.), which makes sense because this program
comes from the Tor Project itself and is well written. Anyway, once we
have a solid idea of what needs intercepting and/or inhibiting [3],
we'll need volunteers yet again, this time to help run packet traces
against a prototype both for the proxied connection and for DNS leakage.
Obviously, help from those familiar with the Emacs network stack would
go a long way here.

Anyway, the attached POC implementation is basically just a slightly
cleaned up version of the same patch set I've been fiddling with this
entire bug thread. WRT to the `url' side, I've done basically zero
additional research, and the interface you see is just a "best guess"
based on the current documentation and a cursory stroll around the
library. As for the `socks' side, everything is based on observations
from exactly one Emacs installation on one GNU/Linux machine. That it
doesn't leak DNS (for me) should just be a data point.

If others want to work on this or take over, please let me know. I
encourage anyone with a similar setup (GNU/Linux, x86_64, modern
libraries) to try them out. You'll need a Tor service running locally,
preferably with the default configuration. The API is pretty basic and
could be simplified even further. For now, it's

  (require 'socks)
  (require 'url)
  (setopt url-proxy-services '(("https" . "socks5h://localhost:9050")
                               ("http" . "socks5h://localhost:9050"))
          socks-server '("tor" "localhost" 9050 5)
          socks-username ""
          socks-password "")

then something like

  M-x eww RET https://check.torproject.org RET

or

  M-x list-packages RET.


[1] https://lists.gnu.org/archive/html/emacs-devel/2023-12/msg00570.html

[2] https://gitlab.torproject.org/tpo/core/torsocks.git

[3] Just to clarify a couple things I've personally found confusing,
    in case others may find them beneficial:

    If we only decide to support local proxies, we don't have to worry
    about DNS leakage for the proxy service itself. I've heard folks
    express concern about calls to getaddrinfo and, in particular, the
    async `:nowait' variant getaddrinfo_a (because torsocks AFAIK
    doesn't shadow it). But those won't come into play since our only
    connections through `make-network-process' are to the local proxy
    service. IOW, the actual remote lookup is tunneled, so the only
    calls we'll need to care about intercepting WRT DNS are those from
    `nsm-check' to `network-lookup-address-info'. (In the attached POC
    implementation, I simply reroute these through a tor-specific
    resolver.)

    The "onion cookie" bookkeeping business performed by torsocks is of
    no interest to us because we control the means of connection. IOW,
    at no point should an onion address (or that of a normal remote
    endpoint) be exposed to the underlying networking machinery. It's
    all just part of the opaque application data unit (payload).

0000-v5-v6.diff
Description: Text Data

0001-POC-Support-SOCKS-resolve-extension.patch
Description: Text Data

0002-POC-Simplify-network-stream-openers-in-socks.el.patch
Description: Text Data

0003-POC-Integrate-the-socks-and-url-libraries.patch
Description: Text Data