bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably

[Yuan Fu]

> On Dec 16, 2023, at 9:43 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>> I think most of the time the parser notifier will send ranges of limited
>> size (as one or other piece of text gets recognized as e.g. a comment, or
>> a raw string -- the bounds of the respective node that changed).
> 
> Yes, of course, in practice it will usually work OK.
> I'm talking about what should happen ideally so that we can make an
> informed decision about the code we decide to use in the end, knowing
> its downsides.
> 
>>> IOW, it'd be better to mark the changed chunks with this
>>> `put-text-property`, but to do it before jit&font-lock get triggered.
>>> Basically, we'd like to call it from `after-change-functions`, but this
>>> can be called *many* times within a single command, so we want to delay
>>> it.  So we can probably just set a flag that says "there are unprocessed
>>> changes" and then sprinkle calls to a "lazy update" function which
>>> checks this flag before calling `treesit--font-lock-notifier` (or
>>> something similar).
>> 
>> I thought doing this lazily (inside font-lock-default-fontify-region), only
>> when the region is set to be redisplayed, would be more economical than
>> doing more work inside after-change-functions.
> 
> Indeed we don't want to do it directly in `after-change-functions`, but
> doing it in `font-lock-default-fontify-region` is too late because the
> redisplay and jit-lock have already (to some extent) decided what should
> be refontified at that point.
> 
> The infrastructure does offer ways to make it work (to fix previous
> incorrect assumptions about what needed to be redisplayed/refontified),
> but it's best if we can avoid it.

I tried forcing reparse in pre-redisplay-functions, and it seems to work fine! 
See attached diff. The delist part can be improved a bit, but this is just a 
POC.

> 
>>> One of the places where we'd need the sprinkle would be
>>> `pre-redisplay-functions`.  Another one might be `syntax-ppss`?
>> syntax-ppss has syntax-propertize-extend-region-functions. I haven't found
>> any better suitable hooks than that.
> 
> Indeed, I don't think there's a good hook for that yet, tho maybe
> `syntax-propertize-extend-region-functions` is good enough (basically
> make it call `treesit--font-lock-notifier` but return nil), assuming we
> have set a `syntax-propertize-function`.

That makes sense, I’ll give that a try on bug#67262.

Yuan

pre-redisplay.diff
Description: Binary data