[AUCTeX-devel] verbatim problem and proposal

[Ralf Hemmecke]

Hello,

from what I have seen in the code, it seems that

http://www.gnu.org/software/auctex/manual/auctex/Known-problems.html#Known-problems

cannot be solved without a dramatic change in the code.

Prove me wrong, but as far as I understand the code, in order to
determine that point is inside a math or a verbatim environment, it is
searched backwards until the opening environment is found. That
seemingly works in most of the cases, but it cannot work in the most
general case.

For example,

\documentclass{article}
\begin{document}
\begin{verbatim}
  \begin{verbatim}
\end{verbatim}
\end{document}

is a proper latex document. But it's highlighting with auctex (at least
in my emacs23 is wrong. Auctex believes that everything up to
\end{document} is verbatim text.

Of course, it's certainly unwise to always scan the whole buffer from
the first line in order to figure out the correct state of point, but
one could keep a datastructure (a tree) that can be parsed and updated
in nearly no time.

Since my emacs lisp knowledge is still very limited, I can only make a
conceptual proposal of how to remove that problem. (At least for all the
cases that don't change catcodes.)

Unfortunately, I don't know whether that makes really sense in a emacs
buffer, but what I imaging is that there are (as there exist already
now, see for example, texmathp-tex-commands-default

http://github.com/emacsmirror/auctex/blob/web-import/texmathp.el#L111

) some commands and environments that change certain status information
(like being in a math environment or inside a verbatim environment) that
occur in the .tex document. There structure is stored as nodes in a
tree. This tree is stored in a buffer-local variable and updated with
every change in the buffer.

Let me demonstrate how the content of this variable corresponds to
respective latex text.

(It's a similar document to what I've filed as bug report at
http://permalink.gmane.org/gmane.emacs.auctex.bugs/1706 )

---rhxBEGIN p.tex
\documentclass[a4paper]{article}
\begin{document}
blah blah
\begin{verbatim}
  \begin{myverbatim}{chunkname}    % first verbatim line
\end{verbatim}
and                                % still verbatim
\begin{verbatim}
  \end{myverbatim}                 % last verbatim line
\end{verbatim}                     % closes verbatim
\end{document}
---rhxEND p.tex

The corresponding variable should remember all the \begin{...} and
\end{...} environments, but already forgetting all things that are
inside a verbatim environment. (Remove all the LISP comments.)
The numbers that appear right after the \begin{..} or \end{..} strings
are the differences of point to the entry immediately preceding the entry.

---rhxBEGIN pp.lsp
("\begin{document}" 34
 ("\begin{verbatim}" 27;=61-34
              ;;;\begin{myverbatim}{chunkname}    % first verbatim line
  "\end{verbatim}" 74);=135-61
              ;;;and                              % still verbatim
("\begin{verbatim}" 98;=223-135
              ;;;\end{myverbatim}                 % last verbatim line
 "\end{verbatim}" 73);=296-223                    % closes verbatim
"\end{document}" 53);=349-296
---rhxEND pp.lsp

So the 34 means that \begin{document} starts at position 34.
The \begin{verbatim} starts at position 27 + 34.
The \end{verbatim} starts at position 74 + (27+34).
The next \begin{verbatim} starts at 98 + (74+27+34).

I guess with that the structure is clear. Since all positions are
relative, inserting or deleting a character means just a local change.

However, if the change introduces a new environment, then there is a
more dramatic change, but only in the tree that corresponds to point and
after.

Suppose we add a quote environment.

---rhxBEGIN p2.tex
\documentclass[a4paper]{article}
blah blah
\begin{document}
\begin{verbatim}
  \begin{myverbatim}{chunkname}    % first verbatim line
\end{verbatim}
and                                % still verbatim
\begin{quote}
   some text
  \begin{itemize}
  \item one=$\sum_{k=1}^n f(k)$
  \end{itemize}
\end{quote}
\begin{verbatim}
  \end{myverbatim}                 % last verbatim line
\end{verbatim}                     % closes verbatim
\end{document}
---rhxEND p2.tex

Then the tree would look like.

("\begin{document}" 34
 ("\begin{verbatim}" 27;=61-34
              ;;;\begin{myverbatim}{chunkname}    % first verbatim line
  "\end{verbatim}" 74);=135-61
              ;;;and                              % still verbatim
("\begin{quote}" 98;=223-135 ((taken from the line marked with "was:"))
              ;;; some text
  ("\begin{itemize}" 19;=242-223
              ;;; \item one=
    ("$" 28;=270-242
              ;;; \sum_{k=1}^n f(k)
     "$" 18);=288-270
   "\end{itemize}" 4);=292-288

 "\end{quote}");=306
("\begin{verbatim}" 98;   ((was: =223-135))
              ;;;\end{myverbatim}                 % last verbatim line
 "\end{verbatim}" 73);=296-223                    % closes verbatim
"\end{document}" 53);=349-296

And of course no change before "\begin{quote}".

The insertion of a $ means reconstructing the tree from point on. Sure.
But as I see the current font-locking, re-parsing the rest of the
document is probably done also now.

And to save time, it is only necessary to build the tree up to the
maximum visible part of the buffer.

Furthermore, the \begin{quote} ... \end{quote} and even \begin{itemize}
.. \end{itemize} levels don't need to be stored, since the only relevant
thing for the status of point is the position of the $ .. $ environment.
That would mean, one could save two levels.

I think, however, such a tree could also serve other purposes. For
example, it would be clear if someone closes an environment that is not
yet opened. Auctex could show that in red.

Just a few ideas. Comments are welcome.

Ralf