Re: testsuite under wine

[Jacek Caban]

Hi Jacob,

Sorry for the delay.

On 12/24/22 06:33, Jacob Bachmeyer wrote:
> Jacek Caban wrote:
> > On 12/23/22 04:51, Jacob Bachmeyer wrote:
> > > Eric Pouech wrote:
> > > > Le 22/12/2022 à 05:16, Jacob Bachmeyer a écrit :
> > > > > > I think that it would not be enough. The way Windows consoles 
> > > > > > work is that we manage complete internal screen buffer and emit 
> > > > > > output that synchronizes the buffer with Unix terminal inside 
> > > > > > conhost.exe process. It means that its output heavily processed 
> > > > > > and may be very different from what application writes to its 
> > > > > > console handle. While escape codes discussed in this thread are 
> > > > > > the most prominent difference (and that part could, in theory, be 
> > > > > > improved on our side), there are more differences. For example, 
> > > > > > if application writes "\rA\rB\rC", conhost will process it, 
> > > > > > update its internal buffer which changes just one character and 
> > > > > > cursor position, and emit sequence to update it in Unix terminal, 
> > > > > > which could be just "\rC" (or even "C" if cursor was already at 
> > > > > > the beginning of the line). Another example would be long lines: 
> > > > > > conhost will emit additional EOLs instead of depending on 
> > > > > > embedder to wrap the line.
> > > > >
> > > > > So conhost is essentially a Wine-specific screen(1) in that sense, 
> > > > > except that it translates Windows screen buffer manipulations 
> > > > > instead of VT100 escape codes?  As I understand ncurses also 
> > > > > implements most of this; perhaps simply delegating output to 
> > > > > ncurses would solve the problem?  If output were simply delegated 
> > > > > to ncurses, (as I understand) setting TERM=dumb should be 
> > > > > effective to eliminate escape codes from the output, since the 
> > > > > "dumb" terminal does not support them.
> > > >
> > > > unfortunately, things are not as simple as that: on one hand we 
> > > > need to mimic Windows behavior, and on the other hand let apps 
> > > > running in wine behave like regular posix applications <g>
> > > >
> > > > (Note: conhost(.exe) is not wine specific, it's part of the way 
> > > > windows handle the console input/output)
> > >
> > > Right.  So that is the name of the program that manages consoles in 
> > > Windows.  I knew it was not cmd.exe itself.  I was testing an 
> > > understanding that conhost.exe in Wine is essentially similar to GNU 
> > > screen, in that both emulate a console/terminal using a *nix 
> > > terminal.  If so, then it should be possible to delegate the actual 
> > > output (including reductions like the example "\rA\rB\rC" to "\rC") 
> > > to the ncurses library and get proper sensitivity to TERM "for free" 
> > > as well.
> > >
> > > To do that, conhost.exe would need to translate the Windows console 
> > > buffer manipulations into curses operations, or possibly lower-level 
> > > terminfo operations, if you still want to roll your own optimization 
> > > code.  If this were done, you could check if the current terminal 
> > > has sufficient support to properly emulate a Windows console and 
> > > switch to "raw" mode if the needed terminfo capabilities are not 
> > > found.  Setting TERM=dumb in the environment would then force the 
> > > use of "raw" mode.
> >
> >
> > Yes, an analogy to screen is right in many aspects, but there are 
> > also architectural difference that require implementation to be very 
> > different. ncurses operates on tty file descriptors backed by OS 
> > kernel. conhost needs to be able to operate on Windows names pipes, 
> > which are not associated with any file descriptor in Wine.
>
> Please explain this.  Programs using curses make calls into the curses 
> library, which ultimately produces I/O on a tty file descriptor.  It 
> seems to me that conhost is in the same position as the curses 
> library:  programs use Windows API calls for the console which 
> [...across an IPC bridge of some type?...] result in dispatches in 
> conhost that ultimately produce I/O on a tty file descriptor.  Does 
> conhost not actually hold the file descriptor, instead performing 
> translations and sending the result back to the client process, which 
> writes it out?


There may be no file descriptor as far as conhost is concerned, it 
operates on generic Windows handles. For example, named pipes are purely 
a wineserver object (a shared process implementing some of kernel-alike 
tasks via IPC), there is no file descriptor involved. But when a file 
descriptor is available, it's optimized so that conhost holds fd and 
writes to it directly. It's an implementation details of a different layer.


> If full curses is not usable for architectural reasons, terminfo would 
> still be an improvement.  Its setupterm() call may send any known init 
> strings, but it could be given either the write end of a pipe or a 
> file descriptor open on /dev/null if that is a problem.


See above, there may be no fd, so curses does not solve the problem.


> The terminfo database access functions tparm(), tigetflag(), 
> tigetnum(), and tigetstr() all return values to their callers for 
> further processing and the information needed to perform curses-style 
> terminal initialization is stored as string capabilities in the 
> terminfo database.


Yes, we should consider some form of better TERM compatibility.


> > Also my point was that if you capture the output sent by the 
> > application to the terminal and match that to a pattern, then any 
> > processing made by conhost could cause problems. Please correct me if 
> > I'm wrong, but my understanding is that, in the above hypothetical 
> > example, a test case doing printf(stdout, "\rA\rB\rC") and matching 
> > output to "\rA\rB\rC" would be considered valid (and fail on Wine).
>
> This type of thing is a general problem with testing curses programs, 
> so the only difference would be effectively adding curses to programs 
> that are not expected to use it.  Yes, this could break testsuites 
> that should work, so some kind of full bypass would be very helpful; 
> you already have this if wine is run inside a pipeline.
>
> > That's why we're trying to figure out a solution that bypasses 
> > conhost and makes the application write directly to stdout, like 
> > usual native application would do. Such mode would be less compatible 
> > with Windows, but if tests only does simple I/O and no other console 
> > interactions, it should work fine. Interpreting TERM=dumb would be a 
> > possible solution to enter that mode.
>
> I see two aspects to this, and I think both of them have value as 
> improvements to Wine:
>
> 1.  Programs that only use the standard handles (a la ISO C) probably 
> do not /want/ full compatibility with Windows, so their I/O should be 
> direct to the underlying POSIX fds.  Note that line endings are still 
> an issue here, but are /not/ Wine's problem---the program's I/O 
> library module is generating Windows-style line endings because it was 
> written for Windows.


That's what my earlier patch allows. Note that there are weird 
implications like the fact that in this mode, a Windows equivalent of 
isatty(1) will return 0 and a number of Windows console functions will 
not work, so the setup would be kind of weird from Windows point of 
view. I'm afraid that it will not be satisfactory for more complex 
things (gdb?).


> (Actually, an option to explicitly select an X11 Wine console window 
> might be helpful for people that want to invoke a Windows CUI program 
> from a graphical menu; otherwise, you might end up with the CUI 
> silently appearing on the console from which the X session was 
> started... I know adding xterm to the mix solves this, but it is a use 
> case.)


Currently, you'd run it through wineconsole explicitly. And yes, it's 
not perfect and default behaviour could be improved, but it's 
challenging architecturally. There were some improvements to that code 
in current release cycle, which moved things in the right direction, but 
also proved this stuff to be hard to change without breaking anything.


> I think the best goal here is that, for the standard handles, Wine I/O 
> should be equivalent to a network connection (telnet?) to a Windows 
> box.  For CUI, Wine should actually use curses or at least terminfo, 
> to allow the escape codes produced to match the user's terminal.  The 
> user's terminal might not always be a VT100-alike and custom simulated 
> terminals could be very reasonable for testing curses TUI programs.  
> (To my knowledge, there are as yet no testsuites that actually /do/ 
> that, but the composition seems reasonable to me.)


As I said, compatibility with other terminals could be improved, but 
curses does not fit the role. Anyway, for sake of testing, the attached 
patch disables escapes hiding the cursor during screen updates which 
seem the most problematic. With this patch, some tests may work without 
disabling conhost (but other discussed problems are expected).


> Finding that NightStrike's MinGW test cases are transparently using 
> CUI would be a MinGW issue, not a Wine issue.
>
> It has been a long time since I have looked at Windows API, but I am 
> guessing that there is both some equivalent to "fwrite(stdout, ...)" 
> (ok, so likely something like 
> "WriteToHandle(GetStandardHandle(STANDARD_HANDLE_OUTPUT),...)" if I 
> remember the Windows API "style" and its affinity for RSI-supporting 
> names correctly) and a more complex API for CUI that is more closely 
> akin to curses.  Could you delay initializing conhost until the 
> program actually attempts to use CUI features? For this option, 
> CUIInit() ends up calling newterm() and using curses if that succeeds, 
> (possibly, after considerable future implementation effort) popping up 
> a Wine console window using X11 if $DISPLAY is usable, or bailing out 
> if neither way to get a full Windows console works.  Until CUIInit() 
> is called, the standard handles are just that and simply mapped 
> through to the underlying POSIX handles.
>
> Is this architecturally feasible for Wine to do? 


No, at least not without ugly hacks. Some decisions need to be made 
early, conhost can't just miss the first part of output. Going straight 
to conhost for handling POSIX tty so that console works as expected by 
Windows applications is what users want. Even if application doesn't 
need full compatibility, conhost still does the job (except for this 
very specific use case).


BTW, if Expect ever plans a Windows port that's not based on Cygwin, it 
will likely need to use conhost-based pseudo consoles. It would then 
face exactly the same problem as when using Wine. Maybe long-term 
solution fits there? Problematic cursor hide/show escapes should be 
trivial to filter. Other differences may be more challenging.


Thanks,

Jacek

conhost.diff
Description: Text Data