Re: Baffling accented glyphs issue

[G. Branden Robinson]

Hi Peter,

At 2023-08-27T00:06:39-0400, Peter Schaffter wrote:
> On Sat, Aug 26, 2023, Bjarni Ingi Gislason wrote:
> > Lesson to learn: always use the information you have and give it to
> > the program, for example "groff" with option "-K<encoding>.
> > 
> > groff -Kutf8 -V file
> > 
> > preconv -eutf8 file | troff -Tps | grops
> 
> Thanks, Bjarni.  Odd to be discovering this for the first time after
> decades of preparing documents.  Live and learn.

In groff 1.23.0, the preconv(1) man pages attempts to address this
subject more directly.

   iconv support
       While preconv recognizes all of the coding tags listed above, it
       is capable on its own of interpreting only three encodings:
       Latin‐1, code page 1047, and UTF‐8.  If iconv support is
       configured at compile time and available at run time, all others
       are passed to iconv library functions, which may recognize many
       additional encoding strings.  The command “preconv -v” discloses
       whether iconv support is configured.

       The use of iconv means that characters in the input that encode
       invalid code points for that encoding may be dropped from the
       output stream or mapped to the Unicode replacement character
       (U+FFFD).  Compare the following examples using the input “café”
       (note the “e” with an acute accent), which due to its short
       length challenges inference of the encoding used.
              printf 'caf\351\n' | LC_ALL=en_US.UTF-8 preconv
              printf 'caf\351\n' | preconv -e us-ascii
              printf 'caf\351\n' | preconv -e latin-1
       The fate of the accented “e” differs in each case.  In the first,
       uchardet fails to detect an encoding (though the library on your
       system may behave differently) and preconv falls back to the
       locale settings, where octal 351 starts an incomplete UTF‐8
       sequence and results in the Unicode replacement character.  In
       the second, it is not a representable character in the declared
       input encoding of US‐ASCII and is discarded by iconv.  In the
       last, it is correctly detected and mapped.

Regards,
Branden

signature.asc
Description: PGP signature