Org Minor Mode (was Re: Low level trickery for changing character syntax?)

[Thorsten Jolitz]

Tassilo Horn <tsdh@gnu.org> writes:

Hi Tassilo, 

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>
> Hi Thorsten,

> I assume you want to use parts of org-mode in programming mode buffers.
> But which parts?  I only use its convenient cycling of outline headings,
> and that's pretty easy to setup.  I can post the code if wanted.

Right, I'm very much convinced that the power of Org-mode could be
unleashed in a true org-minor-mode that works in the comment-sections of
programming major-modes. 

With outshine.el and outorg.el this does exist already, and works quite
well (and is really useful, I use it all the time). But outshine uses
outline-minor-mode as "backend/engine", thus to bring it closer to the
tru Org-mode power, Org-mode functionality would have to be
reimplemented - a true uphill battle. 

Org-mode has gone a long way since its beginnings as an extension to
outline, and almost any functionality one could imagine is already there
and implemented. 

The major "flaw" of Org-mode that inhibits its use as minor-mode is the
wide-spread use of hard-coded regexps, i.e. regexps of this form

,--------------
| "^\\*+ ... $"
`--------------

in many many variants all over the place. Those three elements "^"
(bol), "$" (eol) and "*" (star) are not portable, since in
comment-sections of programming major-modes the above should look
like:

PicoLisp  (comment-start repeated with padding)
,--------------
| "^## *+ ... $" 
`--------------

Elisp outshine-style (comment-start repeated with padding)
,--------------
| "^;; *+ ... $" 
`--------------

old-school Elisp (comment-start repeated no padding, different star)
,--------------
| "^;;;+ ... $" 
`--------------

(?) CSS (comment-start with padding, comment-end)
,--------------
| "^/[*] [*]+ ... \/[*]$" 
`--------------

I'm almost done writing a library [[https://github.com/tj64/drx][(drx.el)]] 
that allows writing
declarative dynamic regexps for elisp like this:

# default org-mode
#+begin_src emacs-lisp
 (drx " foo" t "+" t)
#+end_src

#+results:
: ^\*+ foo$

but

# old-school elisp 
#+begin_src emacs-lisp
  (let ((drx-BOL "^;;")
        (drx-STAR ";"))
    (drx " foo" t "+" t))
#+end_src

#+results:
: ^;;;+foo$

and

# css
#+begin_src emacs-lisp
    (let ((drx-BOL "^/\\* ")
          (drx-EOL "\\*/$ ")
          (drx-STAR "\\*"))
      (drx " foo" t "+" t))
#+end_src

#+results:
: ^/\* \*+ foo\*/$ 

(unfortunately I learned to late about the existance of amazing rx.el
that can do these things too I guess)

In Org-mode there are 

 - global regexp constants :: a few dozens, easy to convert
 - buffer local regexp constants :: a few dozens, easy to convert, can
      be extracted using the "orgstruct-trick"
 - regexp snippets in functions :: hundreds of (looking-at ...),
      (re-search-forward ...) etc. with hard-coded regexp-snippets
      using one of the three non-portable chars ("^", "$", "*"). 

Converting Org-mode sources would be possible (some 600+ changes to
dozens of files in /lisp), but not easy and quite risky. And it would
require a 'paradigm change' wrt to future use of regexps - no more
hard-coded strings (or only regexps that take into accounts the needs
of an org-minor-mode) but rather dynamically calculated regexps
written with drx.el or rx.el or so. I'm not the one to decide this and
I have strong doubts if such massive changes will be supported.

I'm currently working on [[https://github.com/tj64/omm][Org Minor Mode]], which 
can be seen as

 - a port of outshine from old backend outline to new backend org-mode
 - a merge of orgstruct with outshine plus new ideas

I think orgstruct-mode and outshine a both based on one core idea:

 - outshine :: don't let users specify outline-regexp and level, let
               the minor-mode calculate them from major-modes comment-syntax.
 - outstruct :: run org commands in a temporary environment with Org's
                buffer-local variables set.

Add to this a new idea

 - org-minor-mode :: make the full power of Org-mode available in
                     comment-sections of programming-modes by
                     introducing one abstraction step into Org-mode
                     regexps (e.g. use drx-BOL instead of hardcoded
                     "^", drx-EOL instead of hardcoded "$", and
                     drx-STAR instead of hardcoded "*").

There is not much code to write for implementing org-minor-mode, its
about merging orgstruct with outshine (I'm on my way in omm.el) and to
somehow get around this 'hardcoded regexp' problem.

The keymap of omm.el will look like this:

,----------------------------------------------------------
| [...]
|  (define-key map (kbd "C-c C-x p")
|    (lambda () (interactive) (omm-cmd 'org-set-property)))
|  (define-key map (kbd "C-c C-d")
|    (lambda () (interactive) (omm-cmd 'org-deadline)))
|  (define-key map (kbd "C-c C-x t")
| [...]
`----------------------------------------------------------

with `omm-cmd' taking care of 

 - setting EOL, BOL and STAR according to major-mode

 - establishing an temporary buffer-local Org-mode environment (the
   "orgstruct-trick")

 - commenting out any new line with major-mode comment-syntax
   introduced by the Org command

I tried it in some easy cases, where all I needed to do was fixing the
global and local regexp-constants, and it worked already quite
well. But otherwise hard-coded regexp-snippets are found everywhere in
Org functions, so they just don't work out of the box in
comment-section of other major-modes. 

Therefore I'm looking for a low-level trick to introduce regexp
abstraction into Org-mode before I ask the maintainers if they want to
change the whole thing.

-- 
cheers,
Thorsten