bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably

[Dmitry Gutov]

On 16/12/2023 07:56, Yuan Fu wrote:

> > Hmm, let me have a closer look tomorrow. It can’t be that difficult, we must 
> > have missed something.
>
> Please see this patch, which basically does what you did in your patch. I just 
> started from scratch when experimenting. I didn’t do the context thing, that’s 
> a bit confusing for me.

Yeah, it's more complex than I'd have liked but basically the "context" 
thing was to dispatch to the right invalidator: for font-lock and/or for 
syntax. Whether to return the range using the corresponding 
extend-function, or to invalidate push-style (when the notifier is 
called outside of any extend-function's).

I've tried your patch, and it seemed to have similar problems to mine. 
In particular, that the comment is not unfontified right away when the 
closing slash is removed.

Also, I think we're targeting Emacs 29 for this bug (the patch needed 
adjustments).

> I save updated ranges in a local hash table and access it in extend-region 
> functions to extend the region. Ideally treesit should have a function that 
> returns the updated ranges for a parser, we don’t have that yet, so I’m using 
> the hash table to simulate it.

Okay. I wonder how other users (which care about specific parsers) would 
look.

> I found that if you don’t set text prop fontified to nil for the whole extended 
> region, redisplay doesn’t seem to, well, redisplay the full region, even 
> thought the new face has been correctly applied to them.

That sounds like a bug in font-lock? At the end of jit-lock-fontify-now, 
there is a call creating a timer with jit-lock-force-redisplay.

And that function ends with this:

    ;; Don't cause refontification (it's already been done), but just do
    ;; some random buffer change, so as to force redisplay.
    (put-text-property start end 'fontified t)))))

If I just change t to nil there (or to some other value, like 42), 
either of our patches starts behaving well. Perhaps Stefan could comment.