bug#67061: [PATCH] Improve syntax highlighting for python-ts-mode

From:

Denis Zubarev

Subject:

Date:

Sun, 17 Dec 2023 05:07:01 +0300

> Thanks. I think the only thing that’s still up to discussion is the variable-definition rules. Others can be merged to emacs-29.

I can extract part with variable-definition into the next patch.

In case it is ok, I attached patch without new variable-definition feature.

17.12.2023, 04:10, "Yuan Fu" <casouri@gmail.com>:

On Dec 16, 2023, at 4:26 PM, Denis Zubarev <dvzubarev@yandex.ru> wrote:

Sorry for the delayed response.
> For many Emacs users, they want a quieter or even minimal fontification.
I'm not against it. I just think that highlighting of an interpolation
as a string is wrong. Is it possible to set quiet fontification in
emacs-lisp mode, in such a way that `keywords' in doc-strings were
fontified as a doc-string itself? I think it is similar to
interpolation, it serves the purpose of separating different semantic
elements from each other. IMHO, users who like quiet levels will benefit
from interpolation highlighted differently.

I definitely can see your point, and it makes a lot of sense. I don’t really know people who want quieter fontification wants (probably both approach has their supporters) so can’t speak for them. But in general, I think it wouldn’t hurt to have the option.

> Instead of the third level, the check should use the value
> treesit-font-lock-level. And it should check for each level smaller than
> or equal to treesit-font-lock-level.
Done.

Thank you for your hard work! I’m just here talking and you went ahead and did all the work :-)

  > Non-essential fontification like "var" shouldn't be
> activated at that level. So I suggest to put it in the variable feature,
> along with many other non-essential fontifications. (Variable feature is
> placed at level 4.)
  I added a new feature variable-definition for variables defined for local scopes and put it on the 3rd level.
I also added rules to variable-definition feature for variables in list
comprehension ( [var+1 for var in []] ) and as_pattern (with T as var:,
except E as var:, case str() as var:).
  I've noticed that vars in `for var1, (var2, var3) in []:` are highlighted by the rule from the assignment feature (specifically `pattern_list`, `tuple_pattern`).
It seems easy to fix `pattern_list`, but not so easy for
`tuple_pattern`, since this node may occur recursively.
I didn't touch these rules for now.

Ok, makes sense.

    Summary for all changes in the patch.
  New feature variable-definition:
`for var in range(3)`
`[var+1 for var in []]`
`with T as var:`
`except E as var:`
`case str() as var:`
highlight var as font-lock-variable-name-face
  assignment feature:
var := 3 (named_expression)
var *= 3 (augmented_assignment)
Highlight var as font-lock-variable-name-face.
  Make list_splat_pattern query more precise.
list_splat_pattern may appear not only in assignments: var, *rest = call(),
but in the parameter list too: def f(*args).
Highlight args only for the first case in assignment feature.
   type feature:
Fontify built-ins (dict,list,etc.) as types when they are used in type hints.
support nested union types, for example `Lvl1 | Lvl2[Lvl3[Lvl3], Lvl2]`.
This structure is represented via nesting binary_operator and subscript nodes in the grammar.
Function python--treesit-fontify-union-types iterates over all children and highlight identifier nodes.
  Fontify base class names in the class definition: class Temp(Base1, pack0.Base2):
Fontify class patterns in case statement: case [TempC() | bytes(b)]:
Highlight the second argument as a type in isinstance/issubclass call:
isinstance(var2, (str, dict, Type1)); issubclass(var1, int|str)
  For all dotted names of a type highlight only the last part of the name,
e.g. collections.abc.Iterator.
   decorator feature:
Highlight dotted names: @pytest.mark.skip
Function python--treesit-fontify-dotted-decorator iterates over all nested attribute nodes and highlight identifier nodes.
  When font-lock-level is set 4, `skip` had function-call face in: @pytest.mark.skip(reason='t')
Add `:override t` to decorator feature to override function-call face.
   string feature:
Enable interpolation highlighting only if string-interpolation is
presented on the enabled levels of treesit-font-lock-feature-list.
Fix fontification of strings inside of f-strings interpolation,
e.g. for f"beg {'nested'}" - 'nested' was not fontified as string.
   function feature:
Do not override the face of builtin functions (all, bytes etc.) with
the function call face
   keyword feature:
Add "is not" to the `python--treesit-keywords` list.

Thanks. I think the only thing that’s still up to discussion is the variable-definition rules. Others can be merged to emacs-29.

Yuan