Re: Accent mystery

[Robert Goulding]

Ahhhh, thank you so much (I needed to RTFM!) - R.

On Mon, Feb 19, 2024 at 12:44 PM G. Branden Robinson <
g.branden.robinson@gmail.com> wrote:

> Hi Robert,
>
> At 2024-02-19T12:40:16-0500, Robert Goulding via wrote:
> > To answer my own question: It seems that preconv is not guessing the
> > correct encoding from the file with a single word in it.  If I specify
> > -K utf-8 everything works OK.
> >
> > preconv -v reports: GNU preconv (groff) version 1.23.0 with iconv
> > support and with uchardet support
> >
> > Is this an expected shortcoming of preconv - that if a file contains
> > just a single accented character, it won't guess it correctly? The
> > original file it failed on was a 2-page pdf, which has the word
> > kataskeuê in the middle of it.
>
> Yes.  The man page says:
>
>    Coding tags
>      Text editors that support more than a single character encoding
>      need tags within the input files to mark the file’s encoding.
>      While it is possible to guess the right input encoding with the
>      help of heuristics that produce good results for a preponderance of
>      natural language texts, they are not absolutely reliable.
>      Heuristics can fail on inputs that are too short or don’t represent
>      a natural language.
> [...]
>      The use of iconv means that characters in the input that encode
>      invalid code points for that encoding may be dropped from the
>      output stream or mapped to the Unicode replacement character
>      (U+FFFD).  Compare the following examples using the input “café”
>      (note the “e” with an acute accent), which due to its short length
>      challenges inference of the encoding used.
>             printf 'caf\351\n' | LC_ALL=en_US.UTF-8 preconv
>             printf 'caf\351\n' | preconv -e us-ascii
>             printf 'caf\351\n' | preconv -e latin-1
>      The fate of the accented “e” differs in each case.  In the first,
>      uchardet fails to detect an encoding (though the library on your
>      system may behave differently) and preconv falls back to the locale
>      settings, where octal 351 starts an incomplete UTF‐8 sequence and
>      results in the Unicode replacement character.  In the second, it is
>      not a representable character in the declared input encoding of US‐
>      ASCII and is discarded by iconv.  In the last, it is correctly
>      detected and mapped.
>
> Regards,
> Branden
>


-- 
Robert Goulding
Director, John J. Reilly Center for Science, Technology, and Values;
Assoc. Professor, Program of Liberal Studies,
Fellow, Medieval Institute,
University of Notre Dame.