Re: [PATCH] Highlight ANSI sequences in the whole buffer (was [PATCH] ANSI color on example blocks and fixed width elements)

[Nathaniel Nicandro]

Ihor Radchenko <yantar92@posteo.net> writes:

> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> Nathaniel Nicandro <nathanielnicandro@gmail.com> writes:
>>
>>> Feedback appreciated!
>>
>> Thanks for the update!
>> ...
>>> I've finally implemented a solution to what I've discussed previously,
>> ...
>
> It has been a while since the last update in this thread.
> Nathaniel, may I know if you are still working on this?

Hello Ihor,

Yes I'm still working on this.  Attached is an updated patch with some
tests this time.  It's still a work in progress.  Below are responses to
your previous comments about my last update and some comments about this
current patch.

> This is very fragile.
> I believe that hooking into `org-fold-check-before-invisible-edit'
> would lead to simpler implementation.

Thank you for the feedback.  I indeed was able to come up with a
more simpler solution by hooking into that function.

To integrate with `org-fold-check-before-invisible-edit' I had to
introduce two variables, `org-fold-visibility-detail' which is set to
the argument of `org-fold-show-set-visibility' when that function is
called and `org-ansi-fontify-begin' to determine the start of the
fontification region to see if it's close to the beginning of an
invisible sequence that should be turned visible.

Let me know if this is an OK approach.

I ran into an issue when trying to hook into
`org-fold-check-before-invisible-edit' in that when it revealed a
sequence at the end of a line, there would be an extra fontification
cycle that would occur after the reveal which would cause the sequence
to be re-hidden again.  To counteract this I had to use
`buffer-chars-modified-tick' in the way I do.  I couldn't figure out
why redisplay was causing that extra fontification cycle when there
were no modifications to the buffer.

> 1. Open the file and move to the end of the headline "Greater elements"
> 2. <backspace> <space>
> 3. Observe fontification extending past the title.

This is fixed.  I think it was due to specifying the contents-end
position as the end of the region to highlight instead of the
line-end-position for headlines.

> I also edited it around in various places and I managed to trigger
> parser errors when the parser lost track of the modifications. This
> was presumably because your patch edited the buffer.

I no longer make edits to the buffer.  The ANSI sequences are no
longer accompanied by the zero width spaces from the idea that I had
before.

With this patch, editing around sequences should be more stable and
non-surprising.  Basically if a sequence is invisible around point and
you edit it, the sequence remains visible.  It is only after the first
edit outside of a sequence that should make the sequence invisible.
Whenever a sequence is being edited, it should always be visible and
not turn invisible while in the middle of editing it, e.g. due to an
invalid sequence turning valid.

Some comments about the patch, as it currently stands, follow.

- I've introduced two text properties `org-ansi' and
  `org-ansi-context'.

  The first is placed on the regions that actually contain ANSI
  sequences and holds information about the sequence that is useful to
  keep around to detect when a sequence has been modified or deleted
  between fontification cycles, as well as information about whether
  or not a sequence should be revealed due to modifications or because
  of visibility changes.

  The second property holds the ANSI context, as defined by
  `ansi-color-context-region', for regions that actually have been
  highlighted or processed by `org-ansi-process-region'.  Storing the
  ANSI context is done so that on fontifying some new region, the
  context that should be used can be determined simply by examining
  the property on an appropriate region before the start of the
  fontification.  The property is also used to determine the extent of
  a context or sequence, how far forward into the buffer its effects
  last.  The extent of a context is useful for extending the region
  being fontified to include the extent of a sequence which has been
  modified or deleted between fontification cycles.

  Currently I only extend the fontification region to include the
  extent when there has been a deletion or modification of a sequence
  in the region up for fontification (`org-ansi-extend-region').  I've
  not found a way to extend the fontification to a region including
  the full extent of a newly inserted sequence, in such cases the code
  as it stands now will fontify past the limit of fontification to the
  end of the element.

- The `org-ansi-process-*' functions boil down to calls to
  `org-ansi-process-region' which does the actual highlighting and
  bookkeeping of text properties on the regions.  Each of the process
  functions are just aware of the varying types of element structure
  in an Org document.  They are supposed to process an element's
  region from point to some limit or to the end of the element,
  applying properties to the highlightable regions.  If it's to the
  end of the element than they are supposed to move point to that end,
  otherwise move point to limit.
  
- `org-ansi-visit-elements' is supposed to be a function that
  traverses the element structure up to some limit and applies the
  processing functions to the lesser elements that are highlightable.
  It is supposed to take care of moving point to the beginning of the
  actual highlightable regions (if not already contained within one of
  those regions), past any begin lines, list structure, and whatnot.
  It then calls a function that processes the element and moves point
  past the element processed to the next element or to some limit.
  
- The logic to use in `org-fontify-ansi-sequences' and how to maintain
  the highlighting across edits in the buffer are my main focus at
  this point.  I think I've basically figured out the gist of the
  logic, just need to clean it up.  What I have not really considered
  that much is how to maintain/remove the highlighting across edits,
  e.g. when there is something like
  
  <ANSI>line1
  line2
  line3
  line4
  
  all lines being highlighted by the sequence, and the paragraph is
  split at line3 so it becomes
  
  <ANSI>line1
  line2
  
  line3
  line4
  
  the highlighting is removed from line3 but not line4.  And there are
  other situations where editing the buffer does not result in the
  maintenance of the highlighting across the affected elements.  I
  think I had it working in more situations when I had also placed the
  `font-lock-multiline' property on the highlighted regions, but I tried
  to simplify things by just using the `org-ansi-context' property
  which may be able to handle these kinds of situations also somehow,
  by detecting these kinds of edits and extending the region to
  account for them.

0001-Highlight-ANSI-escape-sequences.patch
Description: patch

-- 
Nathaniel