Re: Racket packages / build system

[Christopher Lemmer Webber]

Bonface M. K. writes:

> Christopher Lemmer Webber <cwebber@dustycloud.org>
> writes:
>
>> Bonface M. K. writes:
>>
>>> Christopher Lemmer Webber <cwebber@dustycloud.org> writes:
>>>
>>>> Just a heads up that right now you *can* run Racket-on-Chez, but soon
>>>> Racket-on-Chez will be the "default"... maybe it's a good idea to see
>>>> how hard it will be to make a racket-on-chez package variant.
>>>>
>>>> I'm interested in looking at that but it'll probably have to be a while
>>>> before I can do so... if someone does so before me, I'll be interested
>>>> in beta testing at least.
>>>>
>>>> But also leaving this here as a self-reminder so we aren't surprised
>>>> when it becomes a more important thing to deal with :)
>>>>
>>>>
>>>
>>> Also, since we are talking about Racket, what
>>> happened to having a racket build system?
>>
>> There's interest in it, and Dimos made interesting progress towards
>> figuring out some of the key problems... there's been interest beyond
>> that but sadly it seems like organizing the energy to work on it hasn't
>> happened for whatever reason yet.
>>
>
> Can I volunteer on this task? There's some work in
> my team done in Racket; and it would be of great
> interest to us to have a working Racket build
> system. I can set apart some time to work on
> this. I'd ask for alot of guidance/ review on this
> though :)

YES!  Please do.  Let's talk.  You can ping me on IRC also, dustyweb on
freenode.

>> I've offered to throw money at the problem if someone's willing to take
>> it on btw... not much, but some money.  I've talked to a couple of
>> people about that but it stalled in each iteration.... I don't think
>> it's impossible but I guess it's one of those tasks that for whatever
>> reason seems difficult to get going on because there are some
>> complexities around the way Racket builds "collections" that eg don't
>> seem to show up in Python.  Anyway that's not a judgement because
>> despite wanting it fairly badly clearly I haven't made progress on it.
>>
>> I have the notes that Dimos wrote up not long ago in case anyone is
>> interested.  Dimos, do you mind if I post them to the list?
>>
> Please do share the notes! I can try to hack
> something up \m/\m/.

I looked at the email that Dimos sent to me (also the email I had was
apparently not the most recent email that they are using, corrected in
the addressing this time), and they had asked me if they should post it
to the mailing list, so I think that's consent to post it since they
expressed consideration in doing so in our exchange... so I'm attaching
it.

* Basics of Racket packaging system

Racket provides three abstractions of how to reuse and move around modules of 
code. These are:
** Libraries

A library in Racket is a single file module that can be used in other files. 
Libraries that are
serving some higher purpose are organised together in Collections.

Documentation can be found 
[[https://docs.racket-lang.org/reference/collects.html][here]].

** Collections

Collections are a number of libraries bundled together. They can be added to 
the system through
packages. Racket gets informed for presence of collections through collection 
link files.

In the filesystem collections are directories that include library files. The 
default path they are
stored is #<path:/usr/share/racket/collects>. The corresponding link file is 
located at
#<path:/usr/share/racket/links.rktd> and includes a list as illustrated below:

#+begin_src racket
((root "pkgs/racket-lib")
 (root "pkgs/main-distribution")
 (root "pkgs/2d"))
#+end_src

More specifics about link files and their structure can be found 
[[https://docs.racket-lang.org/reference/collects.html#%28tech._collection._links._file%29][here]].
 More links can be added to
arbitary directories through 'raco link'. This file informs racket where a 
collection resides in the
#<path:/usr/share/racket> directory.

** Packages

Packages in Racket are the abstractions used to share and move around modules 
of code. The include a
number of libraries in a collection, or more collections. They are the means 
through which
dependencies are defined. The main entry for managing them is 'raco pkg'.

The tool for installing packages is 'raco pkg install' that takes as an 
argument a package source
(name of package in a catalog, directory, tar or zip file etc). How it will act 
is defined by the
"info.rkt" file included in the package source. 

Documentation for 
[[https://docs.racket-lang.org/pkg/cmdline.html#%28part._raco-pkg-install%29]['raco
 pkg install']] and 
[[https://docs.racket-lang.org/pkg/metadata.html]["info.rkt";]].

Racket can be configured for where to install and search for packages through a 
confing file
#<path:/etc/usr/racket/config.rktd>. The documentation for the configurations 
can be found 
[[https://docs.racket-lang.org/raco/config-file.html?q=raco][here]].

* Racket build system

I will get into the thoughts that I have for the various phases of the build 
procedure:

** unpack phase
Here normal procedures form 'gnu-build-system' work just fine. The only 
exception is the handling of
.plt files, which can be done as the .gem files are handled in 
'ruby-build-system'.

** bootstrap/build/install phase
Since building and installing is done with 'raco pkg install', it makes sense 
to have a single phase
dealing with this. The problems that exist here are:
1. <#path:/etc/racket/config.rktd> needs to be updated with all the the places 
it needs to look for
   the packages. These include the following:
   1. lib-search-dirs
   2. lib-dir
   3. bin-dir
   4. links-search-files
   5. pkgs-search-dirs
   6. collects-search-dirs
   7. doc-search-dirs
      
   This needs to occur two times. First time to build the package, including 
just its inputs.  Then
   after/while installing the package, we need to create a new config.rktd that 
includes the proper
   places for all formerly and newly installed packages.

2. Racket packages have circular dependencies.

3. Racket tries to rebuild the dependencies for the package it installs, even 
when not needed, based
   on timestamps. There exists an environmental variable 
"PLT_COMPILED_FILE_CHECK", that in
   documentation says if it's set on exist this won't happen. This doesn't seem 
to work as intended.

4. Documentation is created by mutating the docs structure and adding new links 
to new nodes of documentation.

For the *first issue*, Claes Wallin that worked on racket2nix, in fact 
recreates the racket structure
in the outputs folder by copying in the /usr/share/collects, and /lib/racket 
folders, and creating a
symlink of /usr/share/racket/include. He then creates uses the store provided 
binaries of
racket(racket, raco, gracket) with -G flag to set the config directory in the 
outputs and -X flat to
set the collects directory. This way the packages are built and installed in 
the racket
configuration we created at the outputs. Then the config.rktd needs to be 
generated. Claes Wallin
does this with a racket script, which we could use, but could also just use a 
guile function.
I have implemented this, and works as expected, in most cases that don't meet 
the following issues.

The *second issue* is resolved by the importer in racket2nix. The derivations 
are expanded to
include circular dependencies and while building the offending packages, dummy 
inputs are
created. We can do the same using some DFS guile package. Are there any 
standard graph libraries in
guile? 

The *third issue* is more complicated. Racket will try to to recompile 
dependencies based on
timestamps. This creates issues during the build. In racket2nix there is 
created a separate
environment where all dependencies are writable. It is an ugly workaround, but 
seems to work.

As for the *fourth issue* I didn't research the racket documentation system a 
lot, especially since
it's packages are the of the main offenders in circulars dependencies.

Binaries of racket packages are placed in "launchers" in the racket 
configuration folder, so
symlinks to '/bin' shall be created.

** testing phase

It's mostly trivial with 'raco test'.

* Racket importer
Racket packages are defined with "info.rkt" files. Since it's sexp-based it's 
trivial to parse them
and create package-definitons in guile. There exists some edge-cases like 
'implies' that change the
way the packages are upgraded, but in general it pretty simple.

Here is also the place where maybe we could solve the thing with circular 
dependencies. Is it
possible to create custom fields for inputs in guix derivations?