ob-shell: proposal to remove "posh"

[Matt]

Hi,

I would like people's thoughts on removing the "posh" language header.

ob-shell.el supports a "posh" shell.  What is "posh?"

* Posh is not PowerShell
"posh" was added to =ob-shell= in fb09863f with no commit message on December 
13, 2013 (Friday the 13th!).
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=fb09863fbb35bf15bcf78262b6e31b8b8b8617e7

A mail message the same day says,

> Eric Schulte writes:
> >> How about the following resolution?  We rename ob-sh.el to ob-shell.el.
> >> New "shell" code blocks could use the value of the
> >> `org-babel-sh-command' environment variable.  Then sh, bash, zsh, csh,
> >> ash, dash (am I missing any other common ones) use the specific shell
> >> specified.
>
> I've also seen ksh, mksh, posh (the latter specifically for POSIX
> compatibility checks).
https://list.orgmode.org/878uvychr1.fsf@pinto.chemeng.ucl.ac.uk/T/#mc20039f988519d409294dc604b5e0dc0f439307b

There was discussion about different shells, Eric asked for others, "posh" was 
mentioned as "specially for POSIX compatibility checks", and then a "posh" was 
added to ob-shell.el by Eric (fb09863f).
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=fb09863fbb35bf15bcf78262b6e31b8b8b8617e7

Around that time are a few stack exchange answers suggesting to use posh, the 
"Policy-compliant Ordinary SHell", to test for posix compliance.  Debian 
distributed it by saying, "using posh as your /bin/sh may reveal breakage."  It 
seems that posh was used to check for POSIX compliance.  It's still available 
on Debian.
https://unix.stackexchange.com/questions/48786/how-can-i-test-for-posix-compliance-of-shell-scripts
https://web.archive.org/web/20130930231522/https://packages.debian.org/sid/posh
https://packages.debian.org/sid/posh

Wikipedia claims PowerShell is based on POSIX.  However, it's not clear if it's 
"POSIX compliant" (and, if it were, I would be shocked if someone seriously 
suggested using it for POSIX compatibility checks).
https://en.wikipedia.org/wiki/PowerShell#Grammar

Checking the Microsoft documentation, I find no mention of "posh."  PowerShell 
7 introduced a "pwsh.exe" binary.  However, the same page has a reference for 
Windows PowerShell 5.1, corresponding to "powershell.exe".  This is also 
corroborated by Wikipedia which says "PowerShell 7 is the replacement for 
PowerShell Core 6.x products as well as Windows PowerShell 5.1, which is the 
last supported Windows PowerShell version."  "posh" was added to =ob-shell= on 
December 13, 2013.  This would correspond to Powershell 3 or 4 which, AFAIU, 
both have the binary "powershell.exe."
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_pwsh?view=powershell-7.4
https://en.wikipedia.org/wiki/PowerShell#PowerShell_7

It's possible that Eric meant to add PowerShell and abbreviated it to "posh."  
However, it appears much more likely that it refers to the "Policy-compliant 
Ordinary SHell." (https://salsa.debian.org/clint/posh)

* Now what?
So, if we're agreed that "posh" is not PowerShell, what should we do about this?

I propose dropping "posh" from =org-babel-shell-names= and removing it from 
=org-babel-shell-set-prompt-commands=.  That is, let's not explicitly support 
the "Policy-compliant Ordinary SHell" or PowerShell.

We never supported PowerShell intentionally and only came to it by accident.  I 
don't think anyone has used the "Policy-compliant Ordinary SHell" in at least 2 
years.  Die-hard adherents to either can still use them, even if we removed the 
"posh" language header.

** We didn't begin "supporting" PowerShell intentionally (AFAICT)
Up to 2020 and through most of 2022, Powershell wasn't considered supported.
https://list.orgmode.org/87pn707jdz.fsf@gnu.org/T/#u

August 26, 2022 changed that with commit a35d1636.  When 
=org-babel-shell-set-prompt-commands= was added to set the shell prompt, 
PowerShell syntax was included for completeness because people thought "posh" 
was PowerShell.  AFAIK, no one explicitly asked for PowerShell support as an 
end-user (then or after).  The inclusion of PowerShell appears based on the 
misunderstanding of "posh" as PowerShell.  Since then, the handful of list 
messages mentioning PowerShell assume it's supported.  That "support" is all 
based on commit a35d1636.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=a35d163685908386833a3d549ed110931bf3915a
https://list.orgmode.org/?q=d%3A17%2FJune%2F2022..+AND+powershell
https://list.orgmode.org/?q=d%3A17%2FJune%2F2022..+AND+posh

AFAICT, PowerShell is mainly (only?) used for tests by people on the mailing 
list to check that ob-shell can run them.  Occasionally, (once every few years) 
someone specifically asks about it or comments on how it's not (well) 
supported.  Only after commit a35d1636 did people begin saying that PowerShell 
is supported.  People do use PowerShell.  However, there are only 36 hits for 
it on the list in (since Babel was made in 2009) and only a handful of side 
comments since the change.

** No one uses the "Policy-compliant Ordinary SHell"
The change on August 26, 2022 should have caused a breaking error for someone 
using the "Policy-compliant Ordinary SHell."  The prompt for "posh" in 
"org-babel-shell-set-prompt-commands" is valid PowerShell syntax (AFAIKT) and 
invalid bash/dash syntax:

function prompt { "org_babel_sh_prompt> " }

It's not clear to me what this would do in the "Policy-compliant Ordinary 
SHell."  In Bash, functions are defined using a form like:

function fname [()] compound-command [ redirections ]

It says, "Note that for historical reasons, in the most common usage the curly 
braces that surround the body of the function must be separated from the body 
by blanks or newlines. This is because the braces are reserved words and are 
only recognized as such when they are separated from the command list by 
whitespace or another shell metacharacter. Also, when using the braces, the 
list must be terminated by a semicolon, a ‘&’, or a newline."
https://www.gnu.org/software/bash/manual/bash.html#Shell-Functions

So, for 'bash --posix', the syntax is invalid.  Dash simply errors out with 
"dash: 1: function: not found".  Whatever it does on a POSIX-y shell, it's a 
show-stopper.

I don't have "posh" available, but I've run the following using dash and bash.  
It causes Emacs to hang until you hit C-g.  Maybe the "Policy-compliant 
Ordinary SHell" is different and the command works?  Since it fails for 'bash 
--posix' and dash, I doubt it.  Here's what I ran:

(org-babel-do-load-languages 'org-babel-load-languages '((shell . t)))

(defconst org-babel-shell-set-prompt-commands
  '((t . "function prompt { \"%s\" }")))

#+begin_src dash :session *dash*
  echo "hi"
#+end_src

Canceling and switching to the shell that was created, the prompt is "> " (the 
PS2 prompt).  This swallows all input that's not an error.  
=org-babel-execute:shell= calls =org-babel-sh-initiate-session= which calls 
=org-babel-comint-wait-for-output= which waits for the =comint-prompt-regexp= 
which is "org_babel_sh_prompt> ", not "> ", so Emacs hangs.

There have been no complaints from "Policy-compliant Ordinary SHell" users 
after August 26, 2022 about "posh" being broken.  The talk of "posh" (and 
PowerShell, specifically) since August 26, 2022 is often as an aside.

I conclude that no one uses (or at least cares much about using) the 
"Policy-compliant Ordinary SHell."

** Both the "Policy-compliant Ordinary SHell" and PowerShell are usable without 
"posh"
Since the "Policy-compliant Ordinary SHell" is POSIX, it should be 
straight-forward to use with the "shell" language by replacing 
"shell-file-name."  The "shell" language header will hit the default prompt 
command which is general POSIX syntax (that is, it changes PS1 and PS2).  A 
shebang should also work.

I had access to a Windows 10 machine recently and verified the following makes 
PowerShell "work".  I use quotes because the following outputs a banner and 
often fails to remove the prompt, just like the current "posh" behavior.

(org-babel-do-load-languages 'org-babel-load-languages '((shell . t)))

;;; Setup for non-sessions
(setq shell-file-name 
"C:/Windows/System32/WindowsPowerShell/v1.0/powershell.exe")

#+begin_src shell :results output
echo "hello"
#+end_src

;;; Setup for sessions
;; required by `shell.el' in `shell' command
(setq explicit-shell-file-name
      "C:/Windows/System32/WindowsPowerShell/v1.0/powershell.exe")
(setq explicit-powershell.exe-args '("-NoExit"))

;; required by `ob-shell.el' in Org 9.6.6 (distributed with Emacs
;; 29.1 which is the latest build of Emacs for Windows)
(setq org-babel-prompt-command "function prompt { \"org_babel_sh_prompt> \" }")

#+begin_src shell :results output :session *powershell*
echo "world"
#+end_src

* My stance
The unfortunate reality is we "support" PowerShell currently.  We have code 
explicitly to handle PowerShell.  Changing that could, technically, break 
someone's workflow.

Here's how I reason it:

First, AFAIK no PowerShell user would refer to it as "posh".  At best they may 
say "pwsh" and most likely "powershell".

We could keep "posh" for the "Policy-compliant Ordinary SHell" and add 
languages for "pwsh" or "powershell".  This would need to be communicated 
PowerShell users.

I don't know what other keyword we could use for the "Policy-compliant Ordinary 
SHell".  We could keep "posh" for PowerShell, drop all "support" for the 
"Policy-compliant Ordinary SHell", and commit to PowerShell only.  This would 
need to be communicated to the "Policy-compliant Ordinary SHell" users.

In any case, we need to communicate our decision, in ORG-NEWS, through our 
mailing list communications, and definitely by updating the comment.

The cost of keeping PowerShell means maintaining it and whatever weirdo stuff 
comes with it.

I suggest we make the "t" in =org-babel-shell-set-prompt-commands= a catch-all 
for anything with PS1/PS2 (anything POSIX-y?) and remove the "posh" language 
header.  The "t" case should handle bash, dash, and (I suspect) the 
"Policy-compliant Ordinary SHell".  This has the potential to break someone 
using "posh" for PowerShell.  However, I see no indication of anyone seriously 
using it.  We've said for years we don't support it.  The only places I'm aware 
of saying we support PowerShell is the code comment and a few people on the 
mailing list.  Moreover, a "breakage" happens anyway, for either 
"Policy-compliant Ordinary SHell" users (as currently likely exists) or 
PowerShell users (if we give a new keyword).  If we're going to have to put a 
work-around or message out there anyway, I figure let's see about removing any 
semblance of supporting PowerShell until we talk it over.

Fortunately, when I wrote the Worg page, I put "the "Policy-compliant Ordinary 
SHell" for "posh."  So, the only place someone might think that PowerShell is 
supported is from the code comment or from the mailing list.

** PowerShell or cmd.exe?
I'm not claiming that people haven't wanted PowerShell support.  It's been 
discussed a few times on the list.  It always reduces to "it would be nice, 
however we lack a developer or the equipment, so sadly no."

As current ob-shell maintainer, here's how I see it:

I like the idea of supporting PowerShell.  I like the idea of supporting 
cmd.exe much, much more.

Both are associated with a non-free system.  Providing support (even partially) 
for non-free systems is good because it provides an opportunity to teach people 
about software freedom.

AFAIKT, both PowerShell and cmd are MIT licensed:
- https://github.com/microsoft/terminal
- https://github.com/PowerShell/PowerShell

The thought of compiling either for a GNU system is...ugh.  But maybe someone 
else has gotten them working?  Otherwise, it looks like Microsoft distributes a 
developer VM image of Windows.

All together, this means there's no *technical* barrier preventing us from 
running (and hence developing for) PowerShell or cmd.

It's important to note that Emacs defaults to cmdproxy.exe on Windows.  Using 
"shell" in Babel on Windows returns from cmd.exe (through cmdproxy.exe).  I 
believe it's best to support the defaults first.

Because it's the default, I've always used cmd when using Emacs on Windows.  I 
would personally much prefer support for cmd.exe over PowerShell.  That is, I'd 
be happy to work on PowerShell stuff after cmd stuff and both after cleaning up 
ob-shell, ob-comint, and ob-eval.

Thoughts overall?

--
Matt Trzcinski
Emacs Org maintainer (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode