Re: Could the Go importer use the Go toolchain? (was Re: Go importer and packages with version flags)

[Maxim Cournoyer]

Hi François,

Good to see interest being picked up on the Go front.

François <francois-oss@avalenn.eu> writes:

> Hello,
>
> After a hiatus I am trying to package several softwares written on Golang
> (I would like to have terraform and go-jsonnet for example) and I have
> some problems with the current implementation so I am resurrecting this
> old mail from my Draft folder.
>
> Looking at it I think it is mostly reformulations of what Katherine and
> Sarah already wrote on the thread but several months later it could be
> a useful reminder.
>
> I have not really the energy at the moment to work on this alone but I
> would be interested for team work.
>
> On Thu, Sep 30, 2021 at 10:31:08AM -0500, Katherine Cox-Buday wrote:
>> Sarah Morgensen <iskarian@mgsn.dev> writes:
>> 
>> >> IMO, module resolution and graph dependencies in Go are complicated enough
>> >> that I'd much rather take the effort we put in trying to replicate and 
>> >> keep
>> >> up with the Go toolchain's behavior, and spend that effort elsewhere.
>> >>
>> >> Does anyone have opinions on this?
>> >
>> > Hmmm.  Setting aside whether or not we want to use external tools in
>> > general...
>> >
>> > If we use the Go tool, we shift the problem domain into "translating 
>> > between
>> > `go' and Guix."
>
> I felt uneasy when we reimplemented some inherently Go-specific pieces
> of software like go.mod file parsing and dependency resolution. It seems
> so brittle. So I think that for importer specifically we should be able
> to use external programs like the Go toolchain to be in $PATH. And,
> given the Go tool would have proper interfaces, we would indeed just
> have to do some sort of translation.

It's true, that we could perhaps make use of Go for the importer, if
that simplify things and make them more robust in the long term.  I do
not expect these file formats (go.mod) to change too much though, so the
brittleness concern may be exaggerated.  In general, there's a tendency
in Guix to do everything in Guile, for better and worst.

> But as to the exact "how" it is not easy.
>
> Ideally we could split the tasks in severals steps.
> 1. identify all dependencies on form of go modules paths
> 2. identify source repositories (generally identified by a git-ref) for each 
> of those modules
> 3. download content of source repository
> 4. populate a ready-to-use source code local cache
> 5. build
>
> As I see it in Guix we want :
> - 1 and 2 in the importer;
> - 3 is the "origin" method;
> - 4 and 5 in the build system.
>
> `go mod download` does all of 1, 2 and 3. The translations problems are
> how to extract data from this output to isolate dependency management on
> the importer and having the necessary info for the "origin" method. Nix
> bypass completely the problem by using the complete go mod cache as its
> "origin" (or more recently "vendored" dependencies see this Nix change
> to go mod vendor[4]) for each Go package. This solution is not viable
> for us as we want to be able to reuse Go modules.
>
> For the build system part I think we want to use the GOMODCACHE format as
> output of Guix package and to be able to do union-dir of all dependencies
> when we want to build a leaf package (e.g. using GOPROXY=file:/// or
> something like that).

I agree this seems the way to go.

[...]

> Any interest on the matter ?

I think supporting GO modules properly is going to be a must have as Go
moves forward in that direction.  I am interested to have that, but I
don't think I have the bandwidth to volunteer much on the coding front.

I'd be happy to review things or brainstorm though.

-- 
Thanks,
Maxim