bug-bash
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Examples of concurrent coproc usage?


From: Carl Edquist
Subject: Re: Examples of concurrent coproc usage?
Date: Mon, 15 Apr 2024 12:01:16 -0500 (CDT)

On Sat, 13 Apr 2024, Chet Ramey wrote:

The original intent was to allow the shell to drive a long-running process that ran more-or-less in parallel with it. Look at examples/scripts/bcalc for an example of that kind of use.

Thanks for mentioning this example. As you understand, this model use case does not require closing the coproc fds when finished, because they will be closed implicitly when the shell exits. (As bcalc itself admits.)

And if the coproc is left open for the lifetime of the shell, the alternate behavior of deferring the coproc deallocation (until both coproc fds are closed) would not require anything extra from the user.

The bcalc example does close both coproc fds though - both at the end, and whenever it resets. And so in this example (which as you say, was the original intent), the user is already explicitly closing both coproc fds explicitly; so the alternate deferring behavior would not require anything extra from the user here either.

...

Yet another point brought to light by the bcalc example relates to the coproc pid variable. The reset() function first closes the coproc pipe fds, then sleeps for a second to give the BC coproc some time to finish.

An alternative might be to 'wait' for the coproc to finish (likely faster than sleeping for a second). But you have to make and use your $coproc_pid copy rather than $BC_PID directly, because 'wait $BC_PID' may happen before or after the coproc is reaped and BC_PID is unset. (As the bcalc author seems to understand.) So in general the coproc *_PID variable only seems usable for making a copy when starting the coproc.

The info page has the following:

The process ID of the shell spawned to execute the coprocess is available as the value of the variable 'NAME_PID'. The 'wait' builtin command may be used to wait for the coprocess to terminate.

But it seems to me that the copy is necessary, and it is never reliable to run 'wait $NAME_PID'. Because any time the shell is in a position to wait for the coproc to finish, by that time it's going to be a race whether or not NAME_PID is still set.


So this is another example for me of why it would be handy if coproc deallocation were deferred until explicit user action (closing both coproc fds, or unsetting the coproc variable). That way ${NAME[@]} and $NAME_PID could reliably be used directly without having to make copies.

Anyway, just food for thought if down the line you make a shell option for coproc deallocation behavior.


Now, if you built bash with multiple coproc support, I would have expected you could still rig this up, by doing the redirection work explicitly yourself. Something like this:

     coproc UP   { stdbuf -oL tr a-z A-Z; }
     coproc DOWN { stdbuf -oL tr A-Z a-z; }

     # make user-managed backup copies of coproc fds
     exec {up_r}<&${UP[0]} {up_w}>&${UP[1]}
     exec {down_r}<&${DOWN[0]} {down_w}>&${DOWN[1]}

     coproc THREEWAY { tee /dev/fd/$up_w  /dev/fd/$down_w; }


But the above doesn't actually work, as it seems that the coproc shell (THREEWAY) closes specifically all the pipe fds (beyond 0,1,2), even the user-managed ones explicitly copied with exec.

File descriptors the user saves with exec redirections beyond [0-2] are set to close-on-exec. POSIX makes that behavior unspecified, but bash has always done it.

Ah, ok, thanks. I believe I found where this gets set in do_redirection_internal() in redir.c. (Whew, a big function.)

As far as I can tell the close-on-exec state is "duplicated" rather than set unconditionally. That is, the new fd in a redirection is only set close-on-exec if the source is. (Good, because in general I rely on redirections to be made available to external commands.) But apparently coproc marks its pipe fds close-on-exec, so there's no way to expose manual copies of these fds to external commands.

So, that explains the behavior I was seeing ...

It's just a bit too bad for anyone that actually wants to do more elaborate coproc interconnections with manual redirections, as they're limited to shell builtins.

...

I might pose a question to ponder about this though:

With the multi-coproc support code, is it still necessary to set the coproc pipe fds close-on-exec? (If, perhaps, they're already getting explicitly closed in the right places.)

Because if the new coproc fds are _not_ set close-on-exec, in general that would allow the user to do manual redirections for external commands (eg tee(1) or paste(1)) to communicate with multiple coproc fds together.


Shells don't offer any standard way to modify the state of that flag, but there is the `fdflags' loadable builtin you can experiment with to change close-on-exec.

Thanks for the tip. It's nice to know there is a workaround to leave copies of the coproc fds open across exec; though for now I will probably continue setting up pipes in the shell by methods other than the coproc keyword.


Cheers,

Carl


reply via email to

[Prev in Thread] Current Thread [Next in Thread]