[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
bug#46933: Possible bugs in filepos-to-bufferpos / bufferpos-to-filepos
From: |
Eli Zaretskii |
Subject: |
bug#46933: Possible bugs in filepos-to-bufferpos / bufferpos-to-filepos |
Date: |
Sat, 27 Mar 2021 10:54:28 +0300 |
> From: handa <handa@gnu.org>
> Cc: gregory@heytings.org, 46933@debbugs.gnu.org
> Date: Sat, 27 Mar 2021 14:38:56 +0900
>
> In article <83ft0obk7i.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
>
> > Kenichi, why are these 6 bytes inserted by encode-coding-region, but
> > not when we encode the same text as part of saving the buffer to its
> > file? And why does it happen near the end of the text, between those
> > 2 particular letters?
>
> There surely exists a bug. Could you please try the attached patch?
>
> The reason why that bug did not happen on file writing is that the code
> in write_region calls encoding routine repeatedly without
> CODING_MODE_LAST_BLOCK flag, and only in the case that flushing is
> required (e.g. the case of iso-2022-jp), just for flushing, it calls
> enoding routine again with CODING_MODE_LAST_BLOCK flag. In that case,
> carryover does not happen in encode_coding ().
Thanks. The patch fixes the problem with the extra 6 bytes, so I
installed it.
The results of filepos-to-bufferpos with the file attached by Gregory
are better now, but there are still problems for some values of BYTE
argument. The problem is that ISO-2022 encoding (and others like it)
include shift-in and shift-out sequences, used to switch between
character sets. As a trivial example, each CR+LF sequence has the
"ESC ( B" sequence before it and "ESC $ B" sequence after it, to
switch to ASCII before the newline, then switch to Japanese after it.
And likewise whenever there's Latin text within Japanese (there are
quite a lot of them in this particular file). These shift-in and
shift-out sequences consume bytes, but don't produce any characters.
So if the BYTE argument of filepos-to-bufferpos specifies a byte in
the middle of one of these shift sequences, the result will be
incorrect, because decoding a partial sequence produces the bytes of
that sequence verbatim, and the logic in filepos-to-bufferpos of using
the length of the decoded text breaks. We need special handling of
this and other similar coding-systems to fix these corner use cases,
similarly to what we do in filepos-to-bufferpos--dos. Patches
welcome.
I'm leaving this bug open because not all of the problem was fixed.