bug#64329: 29.0.92; treesit/fill-paragraph syntax highlighting problem

[Yuan Fu]

> On Jun 28, 2023, at 9:46 AM, Troy Brown <brownts@troybrown.dev> wrote:
> 
> I've noticed this problem on multiple tree-sitter major modes including
> c-ts-mode, c++-ts-mode, java-ts-mode, bash-ts-mode.  I haven't tried
> others, but I suspect those might also suffer from this problem.
> 
> The issue occurs when attempting to fill the paragraph of a comment
> block.  The following comment block can be used as an example to
> reproduce the problem and happens with "emacs -Q" (assuming
> corresponding tree-sitter libraries are available).
> 
> --8<---------------cut here---------------start------------->8---
> // The quick brown fox jumps over the
> // lazy dog.
> // The quick brown fox jumps over the lazy dog.
> --8<---------------cut here---------------end--------------->8---
> 
> Switch to one of the tree-sitter modes (e.g., M-x java-ts-mode).  Move
> point to the first line of the comment block above and then execute the
> fill-paragraph command (i.e., M-q).
> 
> The text which is wrapped onto the first line of the comment block will
> be highlighted incorrectly.  The results appear as if the comment
> delimiter was removed, fontification occurred, then the text was moved
> to the first line of the comment block and never refontified with the
> comment face.

Thank you very much! It’s funny that how long this went under the radar, 
presumably because we always use block comment.

The culprit is the subst-char-in-region function used by the filling function. 
It has a branch:

if (xxx)
  {
        replace_range (pos, pos + 1, string, ...);
  }
else
  {
        for (i = 0; i < len; i++) *p++ = tostr[i];
  }

I overlooked the else branch and thought subst-char-in-region always calls 
replace_range. replace_range notifies tree-sitter of the change it makes; but 
when subst-char-in-region manually replaces the text in the else branch, those 
edits are not notified to tree-sitter.

Please see the attached patch. Eli, is it more preferable to add a subroutine 
in insdel.c that does what "for (i = 0; i < len; i++) *p++ = tostr[I];” does, 
plus calling treesit_record_change, and make subst-char-in-region call that 
subroutine? (This way editfns.c don’t need to include treesit.h and call 
treesit_record_change itself.)

Yuan

notify.patch
Description: Binary data