[PATCH] Improve Windows tty/pty detection

[Pavel Fedin]

 Hello again! Please consider my second patch to fix the second problem.

 The problem is: MinGW uses MinTTY as its primary console. MinTTY is 
capable of full terminal emulation; but unfortunately terminfo driver 
(using traditional control codes) doesn't work on Windows. I can't tell 
the exact reason for that; i remember it worked back in Windows 8 times, 
but now attempt to use it simply ends up in a garbled display. Probably 
due to some conflict with some Windows 10's own processing.

 The way to go is to prefer using API driver whenever possible. Since 
Aug 2019 ConPTY support has been integrated into Cygwin, and MinGW as a 
consequence; see 
https://github.com/mintty/mintty/issues/56#issuecomment-526170062 . 
Therefore there's no more need to use control codes in MinTTY; we can 
always use API driver and it will work. MinTTY will get console events 
and react accordingly.

 Why setting $TERM to "#win32console" is not a good solution:

1. MinGW consists of two environments: MinGW and MSYS. MSYS is full 
UNIX-alike; in fact it's a tailored version of Cygwin. MSYS build of 
ncurses only relies on control codes; it does not use API driver. 
Somehow it works fine in this configuration; again, i don't know why 
MinGW configuration doesn't work. Both environments share the same 
MinTTY with the same config. Setting TERM to #win32console works fine in 
MinGW shell, but totally kills TUI in MSYS shell with the message

Error opening terminal: #win32con

2. There are some small tools, like "clear", which solely rely on 
control codes. And they need $TERM to be set to real terminal name. 
Again, setting "#win32con" totally kills those tools:

'#win32con': unknown terminal type.

 The suggested patch relies on SysISATTY() macro to detect whether our 
fd is a console or not. _isatty() successfully detects pseudo-consoles 
using ConPTY. If the console is detected, WinAPI driver accepts the 
terminal regardless of $TERM variable. Some more notes on the 
implementation:

1. CON.isMinTTY removed, because i found out that it does nothing.

2. I decided not to include _ismintty() test because frankly speaking i 
don't understand why it's there. See the preamble above, MinTTY makes 
use of ConPTY support, so it's correctly detected as console. There's no 
need to perform an explicit check. Perhaps it's obsolete.

3. Again, i only touched old, production driver. It's up to you to 
decide whether this change also fits the new experimental driver.

diff -ru ncurses-6.4-20230708.orig/ncurses/tinfo/lib_setup.c 
ncurses-6.4-20230708/ncurses/tinfo/lib_setup.c
--- ncurses-6.4-20230708.orig/ncurses/tinfo/lib_setup.c 2023-06-24 
16:25:14 +0000
+++ ncurses-6.4-20230708/ncurses/tinfo/lib_setup.c    2023-07-15 
01:19:57 +0000
@@ -763,10 +763,14 @@

 #ifdef USE_TERM_DRIVER
     INIT_TERM_DRIVER();
+    /*
+     * _nc_get_driver() will call td_CanHandle() for each driver, and 
win_driver
+     * needs file descriptor to do the test, so set it before calling.
+     */
+    termp->Filedes = (short) Filedes;
     TCB = (TERMINAL_CONTROL_BLOCK *) termp;
     code = _nc_globals.term_driver(TCB, myname, errret);
     if (code == OK) {
-        termp->Filedes = (short) Filedes;
         termp->_termname = strdup(myname);
     } else {
         ret_error1(errret ? *errret : TGETENT_ERR,
diff -ru ncurses-6.4-20230708.orig/ncurses/win32con/win_driver.c 
ncurses-6.4-20230708/ncurses/win32con/win_driver.c
--- ncurses-6.4-20230708.orig/ncurses/win32con/win_driver.c 2023-07-08 
22:53:51 +0000
+++ ncurses-6.4-20230708/ncurses/win32con/win_driver.c 2023-07-15 
00:39:17 +0000
@@ -126,7 +126,6 @@
     BOOL buffered;
     BOOL window_only;
     BOOL progMode;
-    BOOL isMinTTY;
     BOOL isTermInfoConsole;
     HANDLE out;
     HANDLE inp;
@@ -613,6 +612,12 @@
     returnCode(result);
 }

+#ifdef __MING32__
+#define SysISATTY(fd) _isatty(fd)
+#else
+#define SysISATTY(fd) isatty(fd)
+#endif
+
 static bool
 wcon_CanHandle(TERMINAL_CONTROL_BLOCK * TCB,
            const char *tname,
@@ -644,7 +649,9 @@
     }
     } else if (tname != 0 && stricmp(tname, "unknown") == 0) {
     code = TRUE;
-    }
+    } else if (SysISATTY(TCB->term.Filedes)) {
+        code = TRUE;
+    }

     /*
      * This is intentional, to avoid unnecessary breakage of applications
@@ -1960,11 +1967,6 @@
 {
     int result = 0;

-#ifdef __MING32__
-#define SysISATTY(fd) _isatty(fd)
-#else
-#define SysISATTY(fd) isatty(fd)
-#endif
     if (SysISATTY(fd)) {
     result = 1;
     } else {
@@ -2180,9 +2182,6 @@
     BOOL b;

     START_TRACE();
-    if (_nc_mingw_isatty(0)) {
-        CON.isMinTTY = TRUE;
-    }

     for (i = 0; i < (N_INI + FKEYS); i++) {
         if (i < N_INI) {