bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably

[Yuan Fu]

> On Dec 13, 2023, at 5:43 PM, Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
> On 13/12/2023 05:28, Yuan Fu wrote:
>>>> c-ts-mode--emacs-set-ranges is registered as a range rule, so many 
>>>> tree-sitter function calls it before doing anything to make sure range is 
>>>> up-to-date. treesit-font-lock-fontify-region calls treesit-update-ranges 
>>>> at the beginning of its body, and treesit-update-ranges calls 
>>>> c-ts-mode--emacs-set-ranges.
>>> 
>>> That seems to mean that any feature accessing the parse tree should call 
>>> treesit-update-ranges first. Including syntax-propertize-functions and 
>>> *-extend-region-functions, which we currently don't do.
>>> 
>>> But which boundaries is it supposed to use? Should 
>>> treesit--syntax-extend-region call treesit-update-ranges, when wait for the 
>>> parser updates, then possibly call treesit-update-ranges again on the 
>>> extended boundaries, and so on, until the parser stops sending 
>>> notifications? Will it stop?
>> It'll stop. If the ranges don't change, no reparse will happen. And only 
>> buffer content change can cause range change. Reparse itself can't. So at 
>> most you'll see reparse -> update ranges -> reparse.
>>> Anyway, could you try my patch? Like I said, I'm not sure if the 
>>> insufficient fontification I'm observing in c-ts-mode is due to the problem 
>>> with the solution, or due to the other redisplay-related problems on my 
>>> system.
>> Yeah, it doesn't solve the problem in c-ts-mode regarding block comments.
>> We might need to run (progn (force-parse) (update-ranges) (force-parse)) 
>> before jit-lock-fontify-now and sytax-ppss.
> 
> Well... I've replaced
> 
>  (treesit-buffer-root-node (treesit-language-at (point)))
> 
> in treesit--syntax-extend-region with
> 
>  (treesit-buffer-root-node (treesit-language-at (point)))
>  (treesit-update-ranges beg end)
>  (treesit-buffer-root-node (treesit-language-at (point)))
> 
> and even tried commenting out the call to 'treesit-update-ranges' inside 
> treesit-font-lock-fontify-region, but the result looks unchanged. And the 
> region does get extended, according to my print-debugging inside 
> font-lock-default-fontify-region.
> 
> Could you check if you're seeing the same?

It should be

(treesit-buffer-root-node 'emacs-c)
(treesit-update-ranges)
(treesit-buffer-root-node ‘c)

I tried, and it doesn’t make a difference. But I must admit I’m not very clear 
on what your patch tries to do. Does it try to extend the to-be-fontified 
region before jit-lock runs?

Yuan