Re: Could the Go importer use the Go toolchain? (was Re: Go importer and packages with version flags)

[François]

Hello,

After a hiatus I am trying to package several softwares written on Golang
(I would like to have terraform and go-jsonnet for example) and I have
some problems with the current implementation so I am resurrecting this
old mail from my Draft folder.

Looking at it I think it is mostly reformulations of what Katherine and
Sarah already wrote on the thread but several months later it could be
a useful reminder.

I have not really the energy at the moment to work on this alone but I
would be interested for team work.

On Thu, Sep 30, 2021 at 10:31:08AM -0500, Katherine Cox-Buday wrote:
> Sarah Morgensen <iskarian@mgsn.dev> writes:
> 
> >> IMO, module resolution and graph dependencies in Go are complicated enough
> >> that I'd much rather take the effort we put in trying to replicate and keep
> >> up with the Go toolchain's behavior, and spend that effort elsewhere.
> >>
> >> Does anyone have opinions on this?
> >
> > Hmmm.  Setting aside whether or not we want to use external tools in
> > general...
> >
> > If we use the Go tool, we shift the problem domain into "translating between
> > `go' and Guix."

I felt uneasy when we reimplemented some inherently Go-specific pieces
of software like go.mod file parsing and dependency resolution. It seems
so brittle. So I think that for importer specifically we should be able
to use external programs like the Go toolchain to be in $PATH. And,
given the Go tool would have proper interfaces, we would indeed just
have to do some sort of translation.

But as to the exact "how" it is not easy.

Ideally we could split the tasks in severals steps.
1. identify all dependencies on form of go modules paths
2. identify source repositories (generally identified by a git-ref) for each of 
those modules
3. download content of source repository
4. populate a ready-to-use source code local cache
5. build

As I see it in Guix we want :
- 1 and 2 in the importer;
- 3 is the "origin" method;
- 4 and 5 in the build system.

`go mod download` does all of 1, 2 and 3. The translations problems are
how to extract data from this output to isolate dependency management on
the importer and having the necessary info for the "origin" method. Nix
bypass completely the problem by using the complete go mod cache as its
"origin" (or more recently "vendored" dependencies see this Nix change
to go mod vendor[4]) for each Go package. This solution is not viable
for us as we want to be able to reuse Go modules.

For the build system part I think we want to use the GOMODCACHE format as
output of Guix package and to be able to do union-dir of all dependencies
when we want to build a leaf package (e.g. using GOPROXY=file:/// or
something like that).

> > For example: when Go downloads a module, it only saves the module, not the
> > whole repository*, so we can't use that to generate the hash.  (* Except it
> > does if you used GOPROXY=direct, but that's in the module cache, and only a
> > bare repository.)
>
> We could use the ~GOMODCACHE~ environment variable[0] to specify where these 
> files are placed.

Yes we want to do that but it leaves open as to how we create this
directory structure. I think it is good to use Go tooling at "guix import"
time but not at build time. In consequence the source must be downloaded
either from upstream git repo (bare or not I do not know) or as zips
from proxy.golang.org. The zip solution could just parse the directory
contents of GOMODCACHE as described in [5]. I would rather import git
repositories closer to real source but it seems more difficult to do.

Even with zip, it leaves open the problem of mapping source zips with
Guix packages and to handle the dependencies correctly.

Any interest on the matter ?

François

[4]: 
https://github.com/NixOS/nixpkgs/pull/86376/commits/9761128d2da7bf4d878982918242e43ae28f9b94

[5]: https://github.com/golang/go/issues/35922#issuecomment-560464243

    cd go/src/golang.org/x/tools/gopls   # or any other module
    export GOPATH=$(mktemp -d)
    go mod download
    find $GOPATH/pkg/mod/cache/download -type f | \
        grep '\.\(mod\|info\|zip\)$' | \
        sed -e "s,$GOPATH/pkg/mod/cache/download,https://proxy.golang.org,";