feature/tree-sitter 47a6c23751 1/3: Add tree-sitter font-lock settings helper function/macro

[Yuan Fu]

branch: feature/tree-sitter
commit 47a6c23751ba2eb097f0d4d61976eefa19425ba1
Author: Yuan Fu <casouri@gmail.com>
Commit: Yuan Fu <casouri@gmail.com>

    Add tree-sitter font-lock settings helper function/macro
    
    1. Add treesit-font-lock-rules that helps with settings
    treesit-font-lock-settings.
    2. Remove treesit-font-lock-defaults and with it, decoration levels.
    
    Now major modes should set treesit-font-lock-settings with the output
    of treesit-font-lock-rules rather than setting
    treesit-font-lock-defaults.
    
    * lisp/treesit.el (treesit-font-lock-settings): Update docstring.
    (treesit-font-lock-rules): New function.
    (treesit-font-lock-defaults): Remove variable.
    (treesit-font-lock-enable): Remove code that interacts
    treesit-font-lock-defaults.
    * doc/lispref/modes.texi: Update manual for
    treesit-font-lock-settings, treesit-font-lock-rules,
    treesit-font-lock-defaults.
---
 doc/lispref/modes.texi |  71 +++++++++++++++++------------------
 lisp/treesit.el        | 100 +++++++++++++++++++++++++++----------------------
 2 files changed, 90 insertions(+), 81 deletions(-)

diff --git a/doc/lispref/modes.texi b/doc/lispref/modes.texi
index ba8b548554..45a44acf54 100644
--- a/doc/lispref/modes.texi
+++ b/doc/lispref/modes.texi
@@ -3883,12 +3883,12 @@ reasonably fast.
 @c if in the future more parser are supported, feel free to reorganize
 @c and rewrite this node to describe multiple parsers in parallel.
 
-Besides simple syntactic font lock and search-based font lock, Emacs
+Besides simple syntactic font lock and regexp-based font lock, Emacs
 also provides complete syntactic font lock with the help of a parser,
 currently provided by the tree-sitter library (@pxref{Parsing Program
 Source}).  Because it is an optional feature, parser-based font lock
 is less integrated with Emacs.  Most variables introduced in previous
-sections only apply to search-based font lock, except for
+sections only apply to regexp-based font lock, except for
 @var{font-lock-maximum-decoration}.
 
 @defun treesit-font-lock-enable
@@ -3897,29 +3897,34 @@ This function enables parser-based font lock in the 
current buffer.
 
 Parser-based font lock and other font lock mechanism are not mutually
 exclusive.  By default, if enabled, parser-based font lock runs first,
-then the simple syntactic font lock (if enabled), then search-based
+then the simple syntactic font lock (if enabled), then regexp-based
 font lock.
 
 Although parser-based font lock doesn't share the same customization
-variables with search-based font lock, parser-based font lock uses
-similar customization schemes.  Just like @var{font-lock-keywords} and
-@var{font-lock-defaults}, parser-based font lock has
-@var{treesit-font-lock-settings} and
-@var{treesit-font-lock-defaults}.
+variables with regexp-based font lock, parser-based font lock uses
+similar customization schemes.  The tree-sitter counterpart of
+@var{font-lock-keywords} is @var{treesit-font-lock-settings}.
 
-@defvar treesit-font-lock-settings
-A list of @var{setting}s for tree-sitter font lock.
-
-Each @var{setting} should look like
+@defun treesit-font-lock-rules :keyword value query...
+This function is used to set @var{treesit-font-lock-settings}.  It
+takes care of compiling queries and other post-processing and outputs
+a value that @var{treesit-font-lock-settings} accepts.  An example:
 
 @example
-(@var{language} @var{query})
+@group
+(treesit-font-lock-rules
+ :language 'javascript
+ '((true) @@font-lock-constant-face
+   (false) @@font-lock-constant-face)
+ :language 'html
+ "(script_element) @@font-lock-builtin-face")
+@end group
 @end example
 
-Each @var{setting} controls one parser (often of different language).
-And @var{language} is the language symbol (@pxref{Language
-Definitions}); @var{query} is either a string query or a sexp query
-(@pxref{Pattern Matching}).
+This function takes a list of text or s-exp queries.  Before each
+query, there are @var{:keyword} and @var{value} pairs that configures
+that query.  The @var{:lang} keyword sets the query’s language, and is
+currently the only recognized keyword.
 
 Capture names in @var{query} should be face names like
 @code{font-lock-keyword-face}.  The captured node will be fontified
@@ -3927,32 +3932,24 @@ with that face.  Capture names can also be function 
names, in which
 case the function is called with (@var{start} @var{end} @var{node}),
 where @var{start} and @var{end} are the start and end position of the
 node in buffer, and @var{node} is the tree-sitter node object.  If a
-capture name is both a face and a function, face takes priority.
+capture name is both a face and a function, the face takes priority.
+@end defun
 
-Generally, major modes should set @var{treesit-font-lock-defaults},
-and let Emacs automatically populate this variable.
-@end defvar
+@defvar treesit-font-lock-settings
+A list of @var{setting}s for tree-sitter font lock.  The exact format
+of this variable is considered internal.  One should always use
+@code{treesit-font-lock-rules} to set this variable.
 
-@defvar treesit-font-lock-defaults
-This variable stores defaults for tree-sitter font Lock.  It is a list
-of
+Each @var{setting} is of form
 
 @example
-(@var{default} @var{:keyword} @var{value}...)
+(@var{language} @var{query})
 @end example
 
-A @var{default} may be a symbol or a list of symbols (for different
-levels of fontification).  The symbol(s) can be a variable or a
-function.  If a symbol is both a variable and a function, it is used
-as a function.  Different levels of fontification can be controlled by
-@var{font-lock-maximum-decoration}.
-
-The symbol(s) in @var{default} should contain or return a
-@var{setting} as described in @var{treesit-font-lock-settings}.
-
-The rest @var{keyword}s and @var{value}s are additional settings that
-could be used to alter the fontification behavior.  Currently there
-aren't any.
+Each @var{setting} controls one parser (often of different language).
+And @var{language} is the language symbol (@pxref{Language
+Definitions}); @var{query} is either a string query or a sexp query
+(@pxref{Pattern Matching}).
 @end defvar
 
 Multi-language major modes should provide range functions in
diff --git a/lisp/treesit.el b/lisp/treesit.el
index cf5001eebc..b969f18514 100644
--- a/lisp/treesit.el
+++ b/lisp/treesit.el
@@ -442,7 +442,10 @@ in-order.  START and END are passed to each range 
function."
 (defvar-local treesit-font-lock-settings nil
   "A list of SETTINGs for treesit-based fontification.
 
-Each SETTING should look like
+The exact format of this variable is considered internal.  One
+should always use `treesit-font-lock-rules' to set this variable.
+
+Each SETTING is of form
 
     (LANGUAGE QUERY)
 
@@ -455,7 +458,28 @@ query.  See Info node `(elisp)Pattern Matching' for how to 
write
 a query in either string or s-expression form.  When using
 repeatedly, a compiled query is much faster than a string or sexp
 one, so it is recommend to compile your queries if it will be
-used over and over.
+used over and over.")
+
+(defun treesit-font-lock-rules (&rest args)
+  "Return a value suitable for `treesit-font-lock-settings'.
+
+Take a series of QUERIES in either string, s-expression or
+compiled form.  Same as in `treesit-font-lock-settings', for each
+query, captured nodes are highlighted with the capture name as
+its face.
+
+Before each QUERY there could be :KEYWORD VALUE pairs that
+configure the query (and only that query).  For example,
+
+    (treesit-font-lock-rules
+     :language 'javascript
+     '((true) @font-lock-constant-face
+       (false) @font-lock-constant-face)
+     :language 'html
+     \"(script_element) @font-lock-builtin-face\")
+
+For each QUERY, a :language keyword is required.  Currently the
+only recognized keyword is :language.
 
 Capture names in QUERY should be face names like
 `font-lock-keyword-face'.  The captured node will be fontified
@@ -463,39 +487,36 @@ with that face.  Capture names can also be function 
names, in
 which case the function is called with (START END NODE), where
 START and END are the start and end position of the node in
 buffer, and NODE is the tree-sitter node object.  If a capture
-name is both a face and a function, face takes priority.
-
-Generally, major modes should set
-`treesit-font-lock-defaults', and let Emacs automatically
-populate this variable.")
-
-(defvar-local treesit-font-lock-defaults nil
-  "Defaults for tree-sitter Font Lock specified by the major mode.
-
-This variable should be a list of
-
-    (DEFAULT :KEYWORD VALUE...)
-
-A DEFAULT may be a symbol or a list of symbols (specifying
-different levels of fontification).  The symbol(s) can be of a
-variable or a function.  If a symbol is both a variable and a
-function, it is used as a function.  Different levels of
-fontification can be controlled by
-`font-lock-maximum-decoration'.
-
-The symbol(s) in DEFAULT should contain or return a SETTING as
-explained in `treesit-font-lock-settings', which looks like
-
-    (LANGUAGE QUERY)
-
-KEYWORD and VALUE are additional settings could be used to alter
-fontification behavior.  Currently there aren't any.
-
-Multi-language major-modes should provide a range function for
-eacn language it supports in `treesit-range-functions', and
-Emacs will set the ranges accordingly before fontifing a region.
-See Info node `(elisp)Multiple Languages' for what does it mean
-to set ranges for a parser.")
+name is both a face and a function, the face takes priority.
+
+\(fn :KEYWORD VALUE QUERY...)"
+  (let (;; Tracks the current language that following queries will
+        ;; apply to.
+        (current-language nil)
+        ;; The list this function returns.
+        (result nil))
+    (while args
+      (let ((token (pop args)))
+        (pcase token
+          (:language
+           (let ((lang (pop args)))
+             (when (or (not (symbolp lang)) (null lang))
+               (signal 'wrong-type-argument `(symbolp ,lang)))
+             (setq current-language lang)))
+          ((pred treesit-query-p)
+           (when (null current-language)
+             (signal 'treesit-error
+                     `("Language unspecified, use :language keyword to specify 
a language for this query" ,token)))
+           (if (treesit-compiled-query-p token)
+               (push `(,current-language token) result)
+             (push `(,current-language
+                     ,(treesit-query-compile current-language token))
+                   result))
+           ;; Clears any configurations set for this query.
+           (setq current-language nil))
+          (_ (signal 'treesit-error
+                     `("Unexpected value" token))))))
+    (nreverse result)))
 
 (defun treesit-font-lock-fontify-region (start end &optional loudly)
   "Fontify the region between START and END.
@@ -535,15 +556,6 @@ If LOUDLY is non-nil, message some debugging information."
 
 (defun treesit-font-lock-enable ()
   "Enable tree-sitter font-locking for the current buffer."
-  (let ((default (car treesit-font-lock-defaults))
-        (attributes (cdr treesit-font-lock-defaults)))
-    (ignore attributes)
-    (setq-local treesit-font-lock-settings
-                (font-lock-eval-keywords
-                 (font-lock-choose-keywords
-                  default
-                 (font-lock-value-in-major-mode
-                   font-lock-maximum-decoration)))))
   (setq-local font-lock-fontify-region-function
               #'treesit-font-lock-fontify-region)
   ;; If we don't set `font-lock-defaults' to some non-nil value,