bug#45117: 28.0.50; process-send-string mysteriously exiting non-locally when called from timer

[João Távora]

Hi Maintainers, Eli

When using SLY Common Lisp IDE package for Emacs, a user has recently
reported a strange intermittent error in SLY's asynchronous ElDoc
function.  That function produces documenttion by querying a network
process, a common technique in many such IDEs.  The user found that when
reducing eldoc-idle-delay to 0.1 he could trigger the problem more
often.

The original report lives at
https://github.com/joaotavora/sly/issues/385.

It was triggered with Emacs 27.1, but I have also reproduced it with a
recent master build.

After analysing the problem, I came to the conclusion that given certain
mysterious conditions, process-send-string, which is called from SLY's
`eldoc-documentation-function` will abruptly return non-locally even
though no error or quit seems to have been signalled.

For now, reproducing this means installing SLY, which is easily done
with a git clone of its source at github.com/joaotavora/sly.git.  One
also needs a `lisp` executable pointing to something like SBCL (South
Bank Common Lisp).  Then

   emacs -Q -L . -l sly-autoloads -f sly

should bring up Emacs with SLY and nothing else.  To trigger the error
it's easier to

  M-: (setq eldoc-idle-delay 0.05)
  C-x C-f ~/path/to/sly/slynk/slynk.lisp ;; or some other lisp file

Now, the user should navigate around, triggering ElDoc and seeing the
documentation in the echo area, until one gets an "unexpected reply"
error in the minibufer.

Interestingly, this unexpected reply comes from the fact that the
network process where process-send-string is writing to has already
processed the request (presumably fully), and has answered. Thus the
process's filter function has been run with that output.

Unfortunately, because process-send-string exited prematurely and
non-locally (under line 2 in the sly.el snippet below), the so-called
continuation lambda hasn't been registered (line 3). Thus when the
filter function runs (code not shown) it will fail to find the
continuation and report an unexpected reply.

1  (let ((id (cl-incf (sly-continuation-counter))))
2     (sly-send `(:emacs-rex ,form ,package ,thread ,id ,@extra-options))
3     (push (cons id continuation) (sly-rex-continuations))
4     (sly--refresh-mode-line))

Reading process-send-string's docstring I see no mention of this
possibility, only the fact that output can be accepted between bunches
of strings.  While the latter could happen, I'm pretty sure that the
Lisp network process wouldn't have enough information to work with,
prematurely, so I don't think that's the reason for the unexpected
reply.

Moreover, reading the C source for process.c I do see some worries with
timers in code comments.  However, I cannot correlate those worries with
this use case.

I'll be soon patching this in SLY with an unwind-protect that only adds
the continuation if process-send-string has exited locally.  That should
hide this hard-to-trigger problem, but the underlying problem remains.

Thanks for your attention,
João

PS: Recently, I've also seen similar "silent" non-local-exits with
accept-process-output when called from filter functions, but I will
report these separately once I have better reproduction recipes.