[MIT-Scheme-devel] definition environments in syntactic closures

[Taylor Campbell]

When I first browsed through the implementation of syntactic closures
in MIT Scheme, I thought the definition environment mechanism was an
unnecessary complication in the system, but, after fiddling with a
couple of hirsute macros I recently wrote, it occurred to me that the
presence of a definition environment, separate from the closing
environment, might be exploited in syntactic closures macros to avoid
needing to fall to the clumsiness of explicit renaming (or reverse
syntactic closures, which equates to the same thing, pretty much).

Specifically, macros that generate definitions currently need to
generate output closed entirely in the usage environment and must
refer to identifiers in the transformer's closing environment
explicitly with RSC or ER.  This is a bit of a bother, since it
defeats much of the convenience of SC macros in the first place over
ER macros.  Instead, if means of distinction were provided between the
regular syntactic environment and the definition environment, these
definition-generating macros could be simplified considerably, perhaps
with a WITH-DEFINITION-ENVIRONMENT procedure:

  (define (with-definition-environment definition-environment form)
    (classifier->form
     (lambda (*form environment *definition-environment history)
       *form *definition-environment    ; ignore
       (classify/form form environment definition-environment
                      (history/replace-reduction form
                                                 environment
                                                 history)))))

Using this, one can write macros like so:

  (define-syntax define-foo
    (sc-macro-transformer
     (lambda (form environment)
       (with-definition-environment environment
         `(DEFINE ,(cadr form) 'FOO)))))

DEFINE-FOO expands to a form closed in the transformer's own closing
environment, but it can introduce definitions in the environment of
the macro's usage.  Does this sound like a good idea to add to the
syntactic closure abstraction?  It does complicate the abstraction
somewhat beyond the original incarnations of syntactic closures, but I
think it would be extremely useful in writing definition-generating
macros, assuming I'm not overlooking any fundamental problems with it.
(Some cursory testing seemed to indicate workage of the above
implementation of WITH-DEFINITION-ENVIRONMENT.)

-- (Skip the rest if you're uninterested in the dissection of some
    examples.)

For some more comprehensive (and real-world) examples of where this
extension would be useful, the two macros I referred to above are a
variant record type definition macro and a macro implementing the join
calculus model of concurrency:

  <http://mumble.net/~campbell/scheme/mit-variant-type.scm>
  <http://mumble.net/~campbell/scheme/mit-join.scm>

(Unfortunately, both of these are slightly broken in small ways: the
variant type definition macro produces output that cannot be fasdumped
because it contains references to environments; and the join calculus
macro has an obscure bug whose gruesome details are uninteresting in
this mail.)

In the case of the variant type, I had to jump through some hoops with
reversed syntactic closures to close every form in the correct
environment, since there is no distinction between the definition
environment and the regular closing environment; the hair here is
apparent in the DEFINE-VARIANT-CASE-MACRO macro.  (There is a lot of
hair elsewhere in the macro as well, it being a hygienic macro that
generates hygienic macros itself, but that's another story.)

In my join calculus macro, the problem is not immediately apparent.
The code managing hygiene is a minuscule portion of the rest of the
hair in the macro, which has more to do with analyzing what channels
are associated with what entry points to processes.  And the macro
WITH-PROCESS works as it is.  But I struck a bit of difficulty trying
to implement a DEFINE-PROCESS macro that was suggested to me: I'd have
to dramatically restructure the code so that the definitions would
actually happen in the usage environment of the DEFINE-PROCESS macro,
while all identifiers inserted by the DEFINE-PROCESS macro must remain
closed in the transformer's environment.  This dramatic difference
between WITH-PROCESS & DEFINE-PROCESS is due to the fact that
WITH-PROCESS can control the scope of subsequent forms (i.e. the forms
in its body) and add the defined names to the list of free identifiers
in the body, while DEFINE-PROCESS cannot, so it must introduce the
definitions into the correct environment in the first place.