Re: UTF-8 conversion problem in Texinfo 6.6 with TEXINFO_XS_PARSER

[Patrice Dumas]

On Mon, Feb 18, 2019 at 08:09:33PM +0000, Gavin Smith wrote:
> On Mon, Feb 18, 2019 at 05:36:00PM +0200, Eli Zaretskii wrote:
> > Thanks, I think I see the problem.  It's because the code manages
> > input_encoding on the input_stack.  Which means each included file
> > starts up with input_encoding of zero (which happens to stand for
> > latin-1), and when reading of the include file is exhausted, the code
> > pops input_stack, so any @documentencoding set by an include file is
> > thrown away, and any file included after @documentencoding has its
> > encoding reset to latin-1.  But @documentencoding is a global setting,
> > and once set, it should remain in effect for any stuff read
> > thereafter, until it is changed by another @documentencoding, or until
> > EOF.  I think this means input_encoding should be part of global_info,
> > not of input_stack.
> 
> Okay, this makes sense.  I don't know if I'll have time to fix this in 
> the next few days.
> 
> I can't remember what I was thinking when I wrote this code, but I may 
> have been thinking about the case of different encodings in different
> input files.  I don't think that use case is worth supporting, though.

I disagree, I think that it is worth supporting that case, as the file
is the natural scope for encoding.  That being said, the first
documentencoding could set the default even if it is in an include file.

Looking at the perl parser, it seems that the encoding is reset for all
the input files upon reading @documentencoding, so a @documentencoding
in an include file resets the main file documentencoding.  I would say
that this is incorrect.

-- 
Pat