bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably

[Yuan Fu]

> On Dec 14, 2023, at 11:12 PM, Yuan Fu <casouri@gmail.com> wrote:
> 
> 
> 
>> On Dec 14, 2023, at 5:01 PM, Dmitry Gutov <dmitry@gutov.dev> wrote:
>> 
>> On 14/12/2023 10:29, Yuan Fu wrote:
>>>> On Dec 13, 2023, at 5:43 PM, Dmitry Gutov <dmitry@gutov.dev> wrote:
>>>> 
>>>> On 13/12/2023 05:28, Yuan Fu wrote:
>>>>>>> c-ts-mode--emacs-set-ranges is registered as a range rule, so many 
>>>>>>> tree-sitter function calls it before doing anything to make sure range 
>>>>>>> is up-to-date. treesit-font-lock-fontify-region calls 
>>>>>>> treesit-update-ranges at the beginning of its body, and 
>>>>>>> treesit-update-ranges calls c-ts-mode--emacs-set-ranges.
>>>>>> 
>>>>>> That seems to mean that any feature accessing the parse tree should call 
>>>>>> treesit-update-ranges first. Including syntax-propertize-functions and 
>>>>>> *-extend-region-functions, which we currently don't do.
>>>>>> 
>>>>>> But which boundaries is it supposed to use? Should 
>>>>>> treesit--syntax-extend-region call treesit-update-ranges, when wait for 
>>>>>> the parser updates, then possibly call treesit-update-ranges again on 
>>>>>> the extended boundaries, and so on, until the parser stops sending 
>>>>>> notifications? Will it stop?
>>>>> It'll stop. If the ranges don't change, no reparse will happen. And only 
>>>>> buffer content change can cause range change. Reparse itself can't. So at 
>>>>> most you'll see reparse -> update ranges -> reparse.
>>>>>> Anyway, could you try my patch? Like I said, I'm not sure if the 
>>>>>> insufficient fontification I'm observing in c-ts-mode is due to the 
>>>>>> problem with the solution, or due to the other redisplay-related 
>>>>>> problems on my system.
>>>>> Yeah, it doesn't solve the problem in c-ts-mode regarding block comments.
>>>>> We might need to run (progn (force-parse) (update-ranges) (force-parse)) 
>>>>> before jit-lock-fontify-now and sytax-ppss.
>>>> 
>>>> Well... I've replaced
>>>> 
>>>> (treesit-buffer-root-node (treesit-language-at (point)))
>>>> 
>>>> in treesit--syntax-extend-region with
>>>> 
>>>> (treesit-buffer-root-node (treesit-language-at (point)))
>>>> (treesit-update-ranges beg end)
>>>> (treesit-buffer-root-node (treesit-language-at (point)))
>>>> 
>>>> and even tried commenting out the call to 'treesit-update-ranges' inside 
>>>> treesit-font-lock-fontify-region, but the result looks unchanged. And the 
>>>> region does get extended, according to my print-debugging inside 
>>>> font-lock-default-fontify-region.
>>>> 
>>>> Could you check if you're seeing the same?
>>> It should be
>>> (treesit-buffer-root-node 'emacs-c)
>>> (treesit-update-ranges)
>>> (treesit-buffer-root-node ‘c)
>> 
>> This might be difficult for treesit--syntax-extend-region to do, since it's 
>> supposed to work across modes. But I suppose it could iterate through 
>> (mapcar #'treesit-parser-language (treesit-parser-list)).
> 
> Actually, only calling (treesit-update-ranges) should be enough (and is 
> generic), the range update function surely would access the parser, which 
> would force the reparse, and the update function will access the parser in 
> the correct order: host language first, then embedded languages.
> 
>> 
>>> I tried, and it doesn’t make a difference. But I must admit I’m not very 
>>> clear on what your patch tries to do. Does it try to extend the 
>>> to-be-fontified region before jit-lock runs?
>> 
>> Yes, it extends the region to-be-fontified. If you set 
>> treesit--font-lock-verbose to t, you should see the appropriate messages 
>> "Fontifying region: x-y" in the message log where x is the position before 
>> the start of the comment (printed by treesit-font-lock-fontify-region).
> 
> Hmm, let me have a closer look tomorrow. It can’t be that difficult, we must 
> have missed something.

Please see this patch, which basically does what you did in your patch. I just 
started from scratch when experimenting. I didn’t do the context thing, that’s 
a bit confusing for me. I save updated ranges in a local hash table and access 
it in extend-region functions to extend the region. Ideally treesit should have 
a function that returns the updated ranges for a parser, we don’t have that 
yet, so I’m using the hash table to simulate it.

I found that if you don’t set text prop fontified to nil for the whole extended 
region, redisplay doesn’t seem to, well, redisplay the full region, even 
thought the new face has been correctly applied to them.

Yuan

extend-region.diff
Description: Binary data