Re: call-process should not block process filters from running

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: app-emacs-dev@janestreet.com
> Date: Tue, 27 Jun 2023 17:55:00 -0400
> 
> 
> When Lisp code calls call-process, then while call-process is running,
> all Lisp is blocked from running, including process filters and timers
> created beforehand by other Lisp.  This call-process behavior is
> harmful, but we can fix call-process to not behave this way.
> 
> This call-process behavior is harmful:
> Many packages rely on their process filters and timers being able to
> run:
> - Packages which communicate over the network (such as ERC)
>   rely on being able to respond to heartbeats and prevent timeouts.
> - Packages which respond to requests from other programs (such as EXWM)
>   rely on being able to respond to those requests.
> A simple (shell-command "sleep 60") will cause most such packages to
> break or behave poorly.

Packages that rely on the above should be defensive in the face of
delays, because Emacs calls blocking APIs all over the place;
call-process is just one of them.  Moreover, a long-running Lisp
program will also block timers and async communications, because Emacs
can only be relied upon to run timers and read subprocess output when
it is idle; otherwise we need some special calls or signals to arrive
to trigger reading input that waits.

Therefore, I think the above exaggerates the problem, and also
"blames" a single API for something that is ubiquitous in Emacs Lisp
programming and should be accounted for by any package that is
sensitive to delays.

> My suggestion is that we should create a new helper function in Lisp,
> perhaps called "process-run", which has an identical interface as
> call-process, except that while it is running, other process filters and
> other Lisp are still able to run.  Then we can move users of
> call-process over to this new function.  Most (but perhaps not all) Lisp
> using call-process should be using process-run, since most Lisp doesn't
> actually want to block process filters from running.

I have no objections to adding new functionality that could make
call-process less blocking (but rewriting it in Lisp based on
make-process and accept-process-output is not the only possible
implementation, see below).  But I am firmly against any massive
moving of the current users of call-process to this new functionality,
for several reasons:

  . some Lisp programs might rely on the fact that no other Lisp runs
    while the sub-process is running, and no input from any source but
    that subprocess can arrive;
  . using async subprocesses has its downsides:
    - it is hard to read only from a given subprocess without causing
      issues to other filters
    - reading from async subprocess has known problems when you need
      to decode its output
    - the timing and synchronization is problematic when using
      accept-process-output, and has its pitfalls
    - on Windows the number of simultaneous async subprocesses is
      limited by a relatively small number
  . reimplementing this in Lisp will increase consing, which will
    trigger more GCs, which will slow down callers of this new API wrt
    call-process

So I think we must consider each caller of call-process separately, on
a case by case basis, and only switch where (a) the process can indeed
take a long time, and (b) after careful auditing of the code and its
expectations from the subprocess call.

> - This does not require changing the C core of Emacs at all, nor
>   changing the call-process implementation.  The existing
>   accept-process-output API is sufficient for creating a synchronous
>   process-running API which does not block Lisp from running in process
>   filters and timers and so on.

I would actually suggest to consider a different approach, which does
need changes in C (but they are relatively simple changes).
Reimplementing all the complexities of the C code (don't forget
call-process-region) in Lisp will definitely cause bugs and
destabilize features that worked flawlessly for decades, so if a way
exists that allows us to have a "less blocking" call-process, without
requiring such massive reimplementation, I think we should seriously
consider it.

And AFAIU, such a way does exist.  The implementation of call-process
actually forks a subprocess, then reads from its pipe until EOF, and
then waits in waitpid for the subprocess to exit.  Both the reading
and the waiting are done in a loop.  So one way of making call-process
less blocking is by adding to these loops calls to
wait_reading_process_output, like we do, for example, in sleep-for,
but conditioned by some new variable exposed to Lisp.  Lisp programs
which want this behavior could then activate it by binding that new
variable.  This could give us most or all of the advantages of
non-blocking API without most disadvantages.  (We'd still need to move
to this functionality on a case by case basis, because some of the
considerations against that could still be valid in individual cases.)