[Help-zile] Next steps [Was: [ANN] Zile Syntax Highlighter]

[Gary V. Vaughan]

Hi Reuben,

I'm wondering what might be the best way to improve the efficiency of
the new syntax highlighter. Clearly, managing a Lua table of curses
attributes per character on every line of every colored buffer - and then
rebuilding everything from point to end-of-buffer on every keystroke (at
least the ones that change the contents of the buffer) is a serious
bottleneck. It's a testament to the speed of Lua that Zz is useable at
all right now ;-)

I'm thinking that I should try to store the attributes alongside the actual
MutableString alien arrays somehow - especially since we're going to
need to make room to store a widechar at each character cell to be
capable of displaying and editing UTF8 buffers eventually.

Any thoughts on a good plan for storing at least 16bits of curses meta-
data plus a widechar in each character cell that integrates well with
`memmove`ing the buffer gap? It sure would be nice to have all the meta-
data invisibly moved around with the character cells it applies to without
maintaining a parallel alien array of attributes! But it's not obvious to me
how to achieve that without a massive slowdown in all the "buffer-region
to Lua string" conversions we currently rely on.

Of course, a parallel alien array of cell attributes will still be a good deal
faster than the slew of nested tables I'm currently slinging... Or maybe
I'm missing something more elegant by thinking along those lines?

While getting space for widechars as a side effect would be a nice bonus
that will make future changes easier, I'm also happy to put that change
off until needed if thinking about it now muddies the optimization of the
syntax highlighter.

The syntax engine assigns a stack of scopes, e.g one cell might be an
escape character in a double quoted string in a function call in Lua source...
and the user can assign colors to any of those scopes, with the highlighter
pass finally adding the curses attribute assigned to the longest top-most
matching scope of those present at each offset. The current algorithm
prematurely optimizes away the scope names, storing only the curses
attributes it renders. And that makes debugging syntax grammars a real
chore - I ended up redoing the Specl grammar in TextMate, which displays
active scope names at point, and then converting to Zile bundle format
because of that :-( So, another consideration is that a big enough speed up
in syntax highlighting would enable  similarly keeping track of the scope name
metadata by offset like TextMate, provided there is room to efficiently store
that somewhere too... with the same trade offs inherent in figuring out how
best to save and retrieve curses attribute metadata.

In any case, advice is most welcome!

Cheers,
-- 
Gary V. Vaughan (gary AT gnu DOT org)