bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably

[Yuan Fu]

On 11/18/23 12:37 AM, Eli Zaretskii wrote:
> Ping!  Yuan, I'd appreciate if you chimed in on this issue.
>
> > Cc: 66732@debbugs.gnu.org, dominik@honnef.co
> > Date: Sun, 29 Oct 2023 14:22:30 +0200
> > From: Eli Zaretskii <eliz@gnu.org>
> >
> > > Date: Wed, 25 Oct 2023 02:15:30 +0300
> > > From: Dmitry Gutov <dmitry@gutov.dev>
> > >
> > > Hi Dominik,
> > >
> > > On 24/10/2023 17:22, Dominik Honnef wrote:
> > > > Steps to reproduce:
> > > >
> > > > 1. Start emacs -q
> > > > 2. Clear the scratch buffer
> > > > 3. Switch to c-ts-mode
> > > > 4. Type the following:
> > > >
> > > >       /* foo
> > > >       bar
> > > >       baz
> > > >       */
> > > >
> > > > Notice the following:
> > > >
> > > > 1. tree-sitter will not parse the buffer contents as a comment until the
> > > > closing comment marker is typed (not an issue per se.)
> > > >
> > > > 2. When you type the closing comment marker, only that line will be
> > > > fontified.
> > > >
> > > > 3. Moving to the previous line will refontify the whole comment.
> > > >
> > > > Similarly, removing the closing comment marker will not instantly
> > > > refontify the buffer, either.
> > > >
> > > > The expected behavior would be for the whole comment to be refontified
> > > > immediately after typing the closing comment marker.
> > > >
> > > > Other buffer contents often lead to even worse refontifying, where even
> > > > motion will not fix things.
> > > Can repro.
> > >
> > > Only with 'emacs -Q', though (note to others who will try).
> > Yuan, any ideas or comments?
Some background: When buffer content changes, it might invalidate 
already-fontified region, as shown in the example: Typing the "*/" 
completes the block comment and should cause the comment to be 
refontified in comment-face.

The way we achieves this is thought parser notifiers. When tree-sitter 
parser reparses, it notifies us of which part of the buffer was affected 
by the reparse. For font-lock, we have this font-lock notifier that 
marks the affected buffer region as "not fontified", so redisplay will 
refontify those areas.

(defun treesit--font-lock-notifier (ranges parser)
  "Ensures updated parts of the parse-tree are refontified.
RANGES is a list of (BEG . END) ranges, PARSER is the tree-sitter
parser notifying of the change."
  (with-current-buffer (treesit-parser-buffer parser)
    (dolist (range ranges)
      (when treesit--font-lock-verbose
        (message "Notifier received range: %s-%s"
                 (car range) (cdr range)))
      (with-silent-modifications
        (put-text-property (car range) (cdr range) 'fontified nil)))))

This notifier function will be called during redisplay [1]. I suspect 
that because of this timing, redisplay doesn't refontify the marked 
region immediately. So I added a timer, I think that ensures we mark the 
affected region in the next command loop?

(defun treesit--font-lock-notifier (ranges parser)
  "Ensures updated parts of the parse-tree are refontified.
RANGES is a list of (BEG . END) ranges, PARSER is the tree-sitter
parser notifying of the change."
  (with-current-buffer (treesit-parser-buffer parser)
    (dolist (range ranges)
      (when treesit--font-lock-verbose
        (message "Notifier received range: %s-%s"
                 (car range) (cdr range)))
      (run-with-timer
       0 nil
       (lambda ()
         (with-silent-modifications
           (put-text-property (car range) (cdr range)
                              'fontified nil)))))))

This seems to work. Eli, do you see any problem using run-with-timer 
this way? What's the correct way to mark some region unfontified?

[1] The chain of events if roughly: user types the last "/" -> redisplay 
-> fontify that character -> access parser -> parser reparses -> calls 
notifier.

Yuan