CreateConsoleScreenBuffer problem on Win10

[Hannes Domani]

Hello

Starting with Win10, there is a problem when creating a process while ncurses
is active, and you just get an error dialog that says:

  The application was unable to start correctly (0xc0000142)
  Click OK to close the application.

It's mainly visible when trying to start a process when using gdb -tui.
See [1] [2] [3] for some of the problem reports.

I finally figured out that the root of this problem is
CreateConsoleScreenBuffer, and created a small reproducer:

#include <windows.h>

int main()
{
  HANDLE con = GetStdHandle(STD_OUTPUT_HANDLE);

  HANDLE out = CreateConsoleScreenBuffer(
      GENERIC_READ | GENERIC_WRITE, 0,
      NULL, CONSOLE_TEXTMODE_BUFFER, NULL);
  SetConsoleActiveScreenBuffer(out);

  STARTUPINFO si;
  PROCESS_INFORMATION pi;
  memset(&si, 0, sizeof(si));
  si.cb = sizeof(si);
  char cmd[] = "cmd.exe /c echo text";
  if (CreateProcess(NULL, cmd,
        NULL, NULL, FALSE, 0, NULL, NULL, &si, &pi))
  {
    WaitForSingleObject(pi.hProcess, INFINITE);
    CloseHandle(pi.hProcess);
    CloseHandle(pi.hThread);
  }

  SetConsoleActiveScreenBuffer(con);

  return 0;
}


The second argument of CreateConsoleScreenBuffer is dwShareMode, and because
it is 0, it's apparently not possible to access the created screen buffer
from the process created afterwards, and it aborts immediately.

If you allow sharing like this:

  HANDLE out = CreateConsoleScreenBuffer(
      GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE,
      NULL, CONSOLE_TEXTMODE_BUFFER, NULL);

The following process starts without problem.

The difference between Win10 and earlier versions seems to be that on Win10
the new process gets the currently active screen buffer as stdout, but on Win7
it still got the original screen buffer as stdout (where sharing is not
restricted).

In all, I suggest this patch:

--- ncurses/tinfo/lib_win32con.c    2021-09-04 12:54:35.000000000 +0200
+++ ncurses/tinfo/lib_win32con.c    2023-02-06 14:37:16.149588200 +0100
@@ -1213,7 +1213,7 @@
                     T(("... creating console buffer"));
                     WINCONSOLE.hdl =
                         CreateConsoleScreenBuffer(GENERIC_READ | GENERIC_WRITE,
-                                                  0,
+                                                  FILE_SHARE_READ | 
FILE_SHARE_WRITE,
                                                   NULL,
                                                   CONSOLE_TEXTMODE_BUFFER,
                                                   NULL);
--- ncurses/win32con/win_driver.c    2021-09-04 12:54:35.000000000 +0200
+++ ncurses/win32con/win_driver.c    2023-02-06 14:38:59.175347600 +0100
@@ -2228,7 +2228,7 @@
     } else {
         T(("... creating console buffer"));
         CON.hdl = CreateConsoleScreenBuffer(GENERIC_READ | GENERIC_WRITE,
-                        0,
+                        FILE_SHARE_READ | FILE_SHARE_WRITE,
                         NULL,
                         CONSOLE_TEXTMODE_BUFFER,
                         NULL);


[1] https://sourceware.org/pipermail/gdb-patches/2020-July/170011.html
[2] https://github.com/msys2/MINGW-packages/issues/10434
[3] https://stackoverflow.com/q/71432412/1983398


Regards
Hannes