Re: bug#67677: 30.0.50; ERC 5.6: Use templates for formatting chat messages

[J.P.]

"J.P." <jp@neverwas.me> writes:

> Earlier work for this bug extended ERC's existing template catalog
> system with an internal framework for dictating (hopefully more
> explicitly and resolutely) how messages materialize in chat buffers.
> Modules can make use of this framework to shape most aspects of message
> formatting, which opens the door to radically contrasting styles, such
> as "multi-part" messages with header/body/footer sections and messages
> with integrated time stamps (meaning you can finally ditch that blessed
> `stamp' module for good).
>
> However, this approach remains awkward when it comes to
>
>   1. minor modifications
>
>   2. modularity itself (keeping modules loosely coupled)
>
> At present, making use of the framework involves defining an entire
> format catalog (although inheritance helps a bit in this regard). But
> the boilerplate issue really begins to compound when trying to integrate
> with other modules because the process is somewhat dependent on defining
> yet more catalogs to serve the various combinations, and more still if
> trying to keep things abstract.

Perhaps this bears some clarifying because a traditional reason for
defining "behavior protocols" is to promote modularity and code reuse.
That'd still be the case here, but not so much in the short term, where
catalog proliferation would be unavoidable. For example, let's say a new
module for spoofing bridge nicknames (call it `masquerade') provides an
option for showing select envelope info on its own line in a
"headline-style" message format:

  <{DisplayName}> {Handle{Affiliation}}\n{Body}

Now suppose another module, `away-indicator', normally displays
"<speaker>" tags in italics for nicks currently away. And suppose this
module wants to integrate with templates supplying headline-style
formatting because there's more room to show a dedicated "away" lighter.
It would first need to detect when such formatting was in play (or
arrange for doing so later) during module init, perhaps by checking
whether the `masquerade' option's variable is bound and true. It would
then replace the current `erc-message-speaker-catalog', possibly set to
something like `masqueraade-speaker', with its own `away-masq-speaker'
(or whatever). It could not, though, elect to detect such behavior on
the fly in a hook member (likely by searching for behavioral clues, like
a newline following a "<speaker>" label) because by that time, the
message has already been formatted.

A superior approach to such an explicit, cross-contaminating integration
(and in-hook heuristics-based detection, for that matter) would be for
ERC to provide a base protocol for headline-style catalogs. This would
likely mean a higher order "meta template" sporting text properties for
identifying intervals relevant to insertion-time formatting. Such
properties need not be exposed directly to users and would likely be
removed after running insertion hooks. Implementers of compatible
catalogs would then declare support by invoking a constructor, perhaps
at top level, that "pre-renders" their actual user-facing templates
(although a runtime version may also work via the function variant of
the symbol-name-based `erc-message-{catalog}-{key}' interface).

For example, a headline-style meta template might look something like

  {Pre}{Headline}{Sep}{Body} ; in reality #("%P%H%S%B" 0 2 (...) ...)

with predefined text properties turning each specifier's argument into a
quasi field of sorts. This would allow the rendered version to do things
like have "{Pre}" and "{Sep}" contain (even multiple) newlines, possibly
as `display' property replacement text. But the main point would be to
make it trivial for "consumers," like `fill', to isolate and operate on
various components and implement consistent provider-agnostic behavior
in hook members without being thrown off by specifics, like the length
or visibility of "{Sep}". Of course, the cost of all this is complexity
and verbosity in the form of additional catalogs, even if some are just
stubs consisting mostly of glue. (That is unless we add some kind of
hook-based solution as proposed by this bug.)

> To alleviate some of this awkwardness and cut down on the verbosity, I'm
> proposing we introduce a more practical extension to this framework.
> It'll be reserved for internal use at first, but with an eye toward
> eventual export (likely in 5.7). The basic idea is that we define a
> single abnormal hook per speaker catalog that runs just prior to
> insertion, and we allow its members to influence the parameters passed
> to `format-spec'. And, we do so in a convenient and structured (and
> reusable) way, so members don't have to twiddle plists in search of a
> single ingredient to manually splice into format strings based on the
> verdict of some flimsy heuristic.
>
> I'm tentatively calling this hook system "msgfspc" (one word), short for
> "message format-spec."

This name is pretty ugly, but we can't really drop the leading "msg" for
just "fspec" or "fmtspec" because ERC already uses `format-spec' in
differing contexts, such as for creating its prompt and its mode line.
IOW, it must be qualified as message-related somehow. Secondly, the name
probably shouldn't contain hyphens because that hampers readability once
a reader has been conditioned to expect it. For example, I'm guessing
most would agree that `erc--msg-fmt-spec-foo-bar' is initially
preferable to `erc--msgfspec-foo-bar'. However, after repeated exposure,
the latter starts to appear less cluttered (IMO), with the elision from
`msgfmtspec' to `msgfspec' perhaps helping in this regard (hand wave).

> It works by defining a struct to accompany each
> hook, with slots based on the catalog's common set of template
> specifiers. Subscribing code then has the freedom to modify the template
> itself and add or subtract specifiers as needed by mutating the struct
> instance for that particular formatting pass. The client API looks like
> this:
>
>   ;; Module activation body
>
>   (add-hook 'erc-msgfspec-speaker-hook
>             #my-maybe-transform-on-msgfspec-speaker nil t)
>   (setq my-state "foo")
>   
>   ;; Top level of package
>
>   (defun my-maybe-transform-on-msgfspec-speaker (spec)
>     (pcase spec
>
>       ;; Modify an outgoing message template.
>
>       ((cl-struct erc-msgfspec-speaker
>                   (key (or 'input-chan-privmsg 'input-query-privmsg)))
>        (erc-msgfspec-insert-spec-after
>         spec ?n ?i (propertize "%i" 'font-lock-face 'my-face))
>        (push `(?i . ,my-state) (erc-msgfspec-alist spec)))
>   
>       ;; Modify an incoming message body.
>
>       ((cl-struct erc-msgfspec-speaker
>                   (key (or 'chan-privmsg 'query-privmsg))
>                   (\?m msg))
>        (setf (erc-msgfspec-speaker-?m spec)
>              (decode-coding-string (my-transform-message msg)
>                                    'utf-8)))))
>
>   ;; Note that at present, all the "erc-foo" symbols above are actually
>   ;; "erc--foo" (internal)
>
> There's at least one unfortunate aspect to the API scheme above: the
> buffer where the working version of a template resides isn't current
> when hooks run. This happens because members still need access to local
> state in the ERC buffer where the actual insertion takes place. I've
> experimented a bit with using the virtual buffer facility (via
> `buffer-swap-text') to get around this, and it appears to work great
> (even seemingly shaving a second or two off runs of ERC's extended test
> suite). However, I'm quite reticent to introduce something I've never
> used before and almost never see in the wild. Thus, this approach will
> have to wait pending further investigation.

An alternative approach would be to use positional arguments of the
hook's member functions to provide access to the various parameters,
instead of passing around a struct instance. This trades the burden of
having to learn the struct's composition with learning the hook's
calling convention. Such an implementation somewhat resembles args- or
return-filtering advice, except it'd almost certainly be implemented via
`run-hook-wrapped'.

Such an approach might be friendlier to more casual users who aren't
comfortable manipulating cl-struct objects. However, one possible
downside is a slightly elevated maintenance cost. IMO, evolving the
schema of a struct is pretty painless and transparent to the user as
long as they're able to take sessions offline to recompile dependencies.
However, updating function signatures, as would be required by the
wrapped-hook method, likely involves adding a `condition-case' to the
wrapper, wherein we'd trap v1 calls and issue a warning before adapting
them to handle v2's format.

I may implement a demo version with a simulated version migration so we
have something concrete to compare.

> The current version of the proposed implementation can be found in the
> second of the attached patches. The first is from bug#68265 but included
> here because the associated demo addressing a real-world use case
> requires both. Please see that bug's recent posts for links and
> instructions.
>
> Thanks.

In the second patch,

> +                              (char (car spec)))
> +                    (cl-assert (stringp (cdr spec)))
> +                    (push `(,(intern (format "?%c" char)) ""
> +                            :type 'string :documentation ,(cdr spec))

                                     ^ this should not be quoted
> +                          out)
> +                    (push char chars))

> +
> +       (cl-defstruct (erc-msgfspec-foo (:include erc--msgfspec))
> +         "Shared object for `foo' catalog message-format hook."
> +         (\?a "" :type 'string :documentation "Ay.")
> +         (\?b "" :type 'string :documentation "Bee.")
> +         (\?c "" :type 'string :documentation "See.")
> +         (my-slot nil :type list :documentation "OK."))

And likewise in the test's assertion for the macro expansion.