Re: Update on the ‘wip-gash’ branch

[Janneke Nieuwenhuizen]

Timothy Sample writes:

Hi Timothy!

> I just pushed a major update to the ‘wip-gash’ branch.  It is rebased on
> top of ‘v0.24.2’, and includes 123 commits.  The first 66 commits
> (‘v0.24.2..538b0f5b1’) add support for Guile-style modules (these have
> been discussed before).  The next 57 add system interfaces and fix bugs
> to get Gash working.  In conjunction with Gash's ‘wip-modular-mes’
> branch, Gash running on Mes can now stumble through an Autoconf
> configure script!  In general, it’s working quite well.

As I wrote in a private reply: This is a really an amazing piece of
work, wow!

> Many thanks to NLnet for funding this work!

+1

Also, thanks for your rebasing effort.  I am working towards a Mes 0.25
release that should include these branches:

   wip:              M2-Planet 1.10.0 support (removal of --bootstrap mode)
     -> wip-x86_64:  Support for x86_64 M2-Planet build (kaem -f kaem.x86_64)
       -> wip-riscv: Initial RISC-V support (possibly kaem -f kaem.riscv64??)

the 0.25 release will probably wait for M2-Planet 1.11.0, as it has an
essential fix for 64bit.  It would be also be nice to get RISC-V working
with M2-Planet, but we'll see...

on top of which I rebased:
      
         -> wip-module  First 66 commits, and
           -> wip-gash  Remaining Gash work

I have added a couple of squash! commits to ressurrect M2-Planet build,
the debugging simple.make and simple.sh builds, and made some minor
changes (prefix work in src/ with "core:" instead of "mes:", add
some Copyright lines, mention you in AUTHORS an from the repl, etc.).

Planning now is to release this as Mes 0.26, we'll have quite some time
to change or add things.

> The bulk of the changes are new system interfaces to handle the minutiae
> of shell scripts: ‘scandir’, ‘environ’, ‘execle’, ‘pipe’, etc.,
> etc. (there are quite a few).  Nine of these are hidden from M2 builds
> of Mes, and only apply when Mes is built with a more complete C
> toolchain (e.g., MesCC and its C library).

Very nice!

> There are downsides to this, but knowing exactly which interfaces
> would need to be implemented in M2 lets us decide if it’s worth the
> effort.  We could also move the goal posts and decide that we don’t
> need file globbing and skip porting ‘readdir’ to M2, say.

Yeah, I'm quite happy with your choice here.  There has been some
interest from #bootstrappable for Gash on Mes because it could reduce if
not eliminate use of ancient packages eearly in the bootstrap.  So that
might just lead to a wish for more functionality in M2libc, we'll see.

> Besides system interfaces, there are many additions to the string and
> character set handling procedures (SRFIs 13 and 14).  There’s a copy of
> SRFI 11 from Guile.  I added a Scheme-level implementation of
> ‘dynamic-wind’ that actually winds.  In general, there are little Scheme
> interface improvements here and there.

Yay!

> I also fixed (or worked around) a few gnarly bugs:
>
>   • Mes’s ‘syntax-rules’ is not hygienic, so I replaced parts of the
>   ‘match’ code with ‘define-macro’ and ‘gensym’ so that it is a little
>   better behaved.  To make it work perfectly, the macro system would
>   need some really deep fixes.

Very nice, and ...ouch!  It seems this got us much better support for
`(ice-9 match'), I had to revert to (system base pmatch) in MesCC
because of "match" brokenness...

>   • When Autoconf tries to find out how to print, it does a test that
>   involves many hundreds of backslashes.  While trying to process all
>   these escapes using the usual list processing procedures (‘map’ and
>   friends) the Mes Scheme stack would overflow.  To fix this, I rewrote
>   many list processing procedures to be iterative instead of recursive.

Yeah, I've been wondering how that works.  "loop" is implemented as a
lamdda that also recurses...

--8<---------------cut here---------------start------------->8---
(define-macro (simple-let bindings . rest)
  (cons (cons 'lambda (cons (map1 car bindings) rest))
        (map1 cadr bindings)))

(define-macro (xnamed-let name bindings rest)
  `(simple-let ((,name *unspecified*))
     (set! ,name (lambda ,(map1 car bindings) ,@rest))
     (,name ,@(map1 cadr bindings))))
--8<---------------cut here---------------end--------------->8---

...so while it's great that you found this solution, I'm a bit puzzled.
Of course, Mes should get tail call elimination support some time ;-)
Until then, I guess these changes are fine.

>   • Gash assumes that file descriptors will be managed by the garbage
>   collector, but Mes does not do this.  An Autoconf script opens /a lot/
>   of files.  More than Mes expects.  So much more, in fact, that Mes
>   starts clobbering the beginning of managed memory (the Scheme symbol
>   table) as part of it’s I/O buffer handling.  It turns out Mes behaves
>   really badly when you overwrite the ‘core:apply’ symbol with junk!  :)

hehe :)

>   After a lot of consternation, I added a check to raise ‘EMFILE’ if Mes
>   opens too many files.  I worked around how many files the scripts open
>   by bumping the limit to 4096 (which is the maximum that my system will
>   let me have with ‘ulimit’ by default).  The configure script I was
>   testing with (Bash’s) opened more than 1K but less than 2K.

Ah right.  Hmm.  Well, let's see if that's enough.  How difficult (or
costly) would it be for the garbage collector to go through the dead
cells to close ports?

>   • I rewrote ‘catch’ to unwind the stack before calling the error
>   handler.  This is how Guile behaves, and it prevents infinite loops if
>   the error handler itself raises an error.

Again, very nice.

> I think the branch is in good shape.  There are some shims and
> workarounds on the Gash side that could be addressed in Mes, but that
> can happen later.  There are two commits that I need help with:
>
>   • fc4a5c972 * DRAFT core: Use exceptions instead of asserts.
>   • 31071dbb3 * xxx! core: Use 'throw' in 'error' unconditionally.
>
> The first one is from Jan, and I don’t really know what it means by
> “DRAFT”.  It’s three years old and part of the original module work.

Yeah, I'm using "DRAFT" usually to mean: Almost finished, needs some
final touches, sometimes just the ChangeLog.  I'm using "WIP" to mean:
work in progress, use at your own risk.

> The second one is from me.  I made it so that Mes would call ‘throw’ in
> ‘error’ even when compiled with M2 or MesCC.  It seems fine, but I don’t
> know why the check was there in the first place.  Jan, I’m hoping you
> can bail me out on these two!  :)

Sure, I've pushed down your xxx commit and renamed it to into squash!
right above the cleaned-up DRAFT.  When the Guix bootstrap builds with
this I will collapse all the squash! commits.

I have also added

    519a690b * squash! DRAFT core: Add a C-only module lookup fast path.
    4e7d290c * squash! mes: guile: Remove 'read-line'.

that remove code that you commented-out.  Any reason for keeping that
commented code, otherwise I'll squash them too.

> If it makes it easier, we could start by merging the first 66 commits.
> That is, merge everything up to:
>
>   538b0f5b1 * core: Add a C-only module lookup fast path.

Yes, so I've split-up your branch so that we can test them separately.
Probably "wip-gash" will see some more work, but it's looking pretty OK.

> That includes just the module system work.  I tested that the other day,
> and was able to build everything “--with-bootstrap” using “boot-5.scm”
> as the default.  I had to mark “psyntax.test” as an expected failure,
> but all the other tests (the full “make check”) pass.  To be clear, the
> ‘wip-gash’ branch also passes all tests, but I haven’t tried using
> “boot-5.scm” (I see no reason why it wouldn’t work).

That's awesome!  Looking at wip-modular-mes, please feel free to move
any/more compatibility, support or stubs to Mes if you like (usleep,
fold-right, string-for-each, ...).

> That’s all for now.  There are a lot of commits here, and I’m happy to
> answer any questions about them.  Let me know how I can help with
> review/merging.

I'm thankful and amazed how you picked-up, cleaned-up, fixed and
finished this module work.  It's one of the tricksiest parts of the Mes
code base for sure!

I'm wondering now that boot-5.scm works *), should we remove the unused and
flaky psyntax support and default to boot-5.  Possibly we could even
remove the mes-use-module "feature" altogether,  WDYT?

Greetings,
Janneke

*) The only thing that I found not to work is using (use-modules ...)
   from the repl, so we'd need some fix or workaround for that possibly
   a new ,use command?

-- 
Janneke Nieuwenhuizen <janneke@gnu.org>  | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com