Re: ncurses garbled characters redux

[Rich Burridge]

On 08/29/2017 05:20 PM, Thomas Dickey wrote:

On Tue, Aug 29, 2017 at 01:36:48PM -0700, Rich Burridge wrote:

Hi all,

We have a weird one. This is on Solaris with ncurses version 6.0-20170708
(with --with-abi-version=5 ;-) ), and Python 2.7.13.

We are seeing "garbage characters" when running the Solaris text installer
on a physical console (either directly on bare metal or with something
like the VirtualBox console). See the attached image.

Sun's hardware console doesn't behave the same as xterm or vt100.
I'd expect $TERM to be either "sun" or "sun-color" for that.

(A while back, Sun got in the habit of setting $TERM for their installer,
so that it would't run properly on the console, apparently assuming that
it was run on an attached serial terminal).

For what it's worth, the terminal description for "sun" does not
mention line-drawing.  (It's been quite a while since I looked at
Sun's documentation, but I probably wouldn't have overlooked that).

Also, it didn't do UTF-8.

In Unicode, ACS_HLINE would be U+2500, and the character you're showing
is (in ISO-8859-1) 226, or octal 342.

The UTF-8 for U+2500 is

	\342\224\200

So... it seems you're displaying UTF-8 on a device that supports ISO-8859-1.

Thanks Thomas. The Solaris installer team have told me that this used
to work, but started failing about the time of the ncurses update. I
will check with them whether they've made any other changes in this area
at that time.

When we are seeing this problem, the TERM setting on the physical console
is "sun-color".

As you know, the ncurses terminfo settings for "sun-color" are:

$ ginfocmp sun-color
#       Reconstructed via infocmp from file: /usr/gnu/share/terminfo/s/sun-color
sun-color|Sun Microsystems Workstation console with color support (IA systems),
        am, km, msgr,
        colors#8, cols#80, lines#34, ncv#3, pairs#64,
        bel=^G, bold=\E[1m, clear=^L, cr=\r, cub=\E[%p1%dD, cub1=^H,
        cud=\E[%p1%dB, cud1=\n, cuf=\E[%p1%dC, cuf1=\E[C,
        cup=\E[%i%p1%d;%p2%dH, cuu=\E[%p1%dA, cuu1=\E[A,
        dch=\E[%p1%dP, dch1=\E[P, dl=\E[%p1%dM, dl1=\E[M, ed=\E[J,
        el=\E[K, home=\E[H, ht=^I, ich=\E[%p1%d@, ich1=\E[@,
        il=\E[%p1%dL, il1=\E[L, ind=\n, kb2=\E[218z, kbs=^H,
        kcub1=\E[D, kcud1=\E[B, kcuf1=\E[C, kcuu1=\E[A, kdch1=^?,
        kend=\E[220z, kf1=\E[224z, kf10=\E[233z, kf11=\E[234z,
        kf12=\E[235z, kf2=\E[225z, kf3=\E[226z, kf4=\E[227z,
        kf5=\E[228z, kf6=\E[229z, kf7=\E[230z, kf8=\E[231z,
        kf9=\E[232z, khome=\E[214z, kich1=\E[247z, knp=\E[222z,
        kopt=\E[194z, kpp=\E[216z, kres=\E[193z, kund=\E[195z,
        op=\E[0m, rev=\E[7m, rmso=\E[m, rs2=\E[s, setab=\E[4%p1%dm,
        setaf=\E[3%p1%dm,
        setb=\E[4%?%p1%{1}%=%t4%e%p1%{3}%=%t6%e%p1%{4}%=%t1%e%p1%{6}%=%t3%e%p1%d%;m,
        setf=\E[3%?%p1%{1}%=%t4%e%p1%{3}%=%t6%e%p1%{4}%=%t1%e%p1%{6}%=%t3%e%p1%d%;m,
        sgr=\E[0%?%p6%t;1%;%?%p1%p3%|%t;7%;m, sgr0=\E[m,
        smso=\E[7m, u8=\E[1t, u9=\E[11t,

When I did the first Solaris update to ncurses a little while ago
to 6.0-20170408, we were previously at version 5.7 (from November 2008).

At that time I took out the following from our build process:

# XXX: Ugly hack copied directly from the desktop SUNWncurses.spec file:
# The sun-color terminfo included in ncurses does not work;
# copying the one shipped with Sun's curses fixes it:
COMPONENT_POST_INSTALL_ACTION += \
       $(CP) /usr/share/lib/terminfo/s/sun-color \
       $(PROTO_DIR)$(GNUSHARE)/terminfo/s/;

Now I don't believe that's true any more (and hasn't been for years),
because looking at the ncurses change log, it looks like Laca (Lazslo)
Peter sent sun-color changes "upstream" back in 2009.

The Solaris curses terminfo settings for "sun-color" are:

$ infocmp sun-color
#       Reconstructed via infocmp from file: /usr/share/lib/terminfo/s/sun-color
sun-color|Sun Workstation console with color text support,
        am, km, msgr,
        colors#8, cols#80, lines#34, ncv#3, pairs#64,
        bel=^G, bold=\E[1m, clear=\f, cr=\r, cub1=\b, cud1=\n,
        cuf1=\E[C, cup=\E[%i%p1%d;%p2%dH, cuu1=\E[A,
        dch=\E[%p1%dP, dch1=\E[P, dl=\E[%p1%dM, dl1=\E[M,
        ed=\E[J, el=\E[K, home=\E[H, ht=\t, ich=\E[%p1%d@,
        ich1=\E[@, il=\E[%p1%dL, il1=\E[L, ind=\n, kcub1=\E[D,
        kcud1=\E[B, kcuf1=\E[C, kcuu1=\E[A, kf1=\E[224z,
        kf2=\E[225z, kf3=\E[226z, kf4=\E[227z, kf5=\E[228z,
        kf6=\E[229z, kf7=\E[230z, kf8=\E[231z, kf9=\E[232z,
        op=\E[0m, rev=\E[7m, rmso=\E[m, rmul=\E[m, rs2=\E[s,
        setab=\E[4%p1%dm, setaf=\E[3%p1%dm,
        setb=\E[4%?%p1%{1}%=%t4%e%p1%{3}%=%t6%e%p1%{4}%=%t1%e%p1%{6}%=%t3%e%p1%d%;m,
        setf=\E[3%?%p1%{1}%=%t4%e%p1%{3}%=%t6%e%p1%{4}%=%t1%e%p1%{6}%=%t3%e%p1%d%;m,
        sgr=\E[0%?%p6%t;1%;%?%p2%t;4%;%?%p1%p3%|%t;7%;m,
        sgr0=\E[m, smso=\E[7m, smul=\E[4m,

Now I've no idea whether any of this is relevant or not to the problem we
are seeing, but I thought I'd better mention it.

As an experiment, I asked the installer team to copy over the Solaris
curses "sun-color" terminfo file (i.e. but the Ugly Hack back), and
try the:

  $ LANG=en_US.UTF-8 python example.py

test again, but I'm told that the problem still persists.