Regarding outline headings in emacs-lisp libraries

[Jonas Bernoulli]

Hello everybody!

I find it very useful when source files are divided into sections and
occasionally subsections as well.  It helps me understand the structure
of not just the file itself but ideally also of the functionality that
it implements.

This is also useful when reading code that one is not familiar with, but
unfortunately not all files are split into sections.  When I come across
a library that lacks explicit headings and I then often add the headings
myself and open a pull-request.

Yesterday I proposed such changes to Emacs itself and there is a detail
that Eli wanted to discuss here first, thus this thread.

But first some words about `outline-minor-mode', which is the section
facility that Emacs provides for use in arbitrary major-modes, for the
people who are not familiar with it yet or haven't actively used it in
a while:

  `outline-minor-mode' lets users navigate sections and lets them change
  their visibility, in pretty much the same way that `org-mode' does;
  the crucial difference being that it works in any major mode including
  programming modes.

  If you are not familiar with that minor-mode yet, then I would
  encourage you to give it try.  You might want to look at some of
  my libraries, which are very likely to actually being divided into
  sub*sections.

  From `org-mode' you might be familiar with local as well as global
  visibility cycling.  `outline-minor-mode' does not provide that out
  of the box, but (interesting historic fact:) the author of `org-mode'
  actually implemented that in the `outline-magic' package before moving
  on to writing `org-mode'.

  Nowadays `outline-magic' is moribund, but I have implemented
  `bicycle', which does the same thing but with a twist: it can also
  changes the visibility of top-level sexps, by wrapping around not only
  `outline' but also `hideshow' -- thus the "bi" prefix.

Eli pointed out that I split emacs-lisp libraries into (sub-)sections in
a way that violates one of the relevant conventions and that we should
discuss that here first.

To begin with, here are the relevant conventions:

,---- From (info "(elisp) Comment Tips")
| ‘;;;’
|      Comments that start with three semicolons, ‘;;;’, should start at
|      the left margin.  We use them for comments which should be
|      considered a heading by Outline minor mode.  By default, comments
|      starting with at least three semicolons (followed by a single space
|      and a non-whitespace character) are considered headings, comments
|      starting with two or fewer are not.  Historically, triple-semicolon
|      comments have also been used for commenting out lines within a
|      function, but this use is discouraged.
| 
|      When commenting out entire functions, use two semicolons.
| 
| ‘;;;;’
|      Comments that start with four (or more) semicolons, ‘;;;;’, should
|      be aligned to the left margin and are used for headings of major
|      sections of a program.  For example:
| 
|           ;;;; The kill ring
| 
|      If you wish to have sub-headings under these heading, use more
|      semicolons to nest these sub-headings.
`----

----------

TL;DR I want to use just *three* semicolons for "major sections of a
program", i.e. I want to consider "Code:" to be one of them instead of
their parent.

Okay then, here's the long version:

----------

When showing nothing but the top-level headings, then a library, which
is split up the way that I do it, looks like this:

,----
| ;;; foo.el --- Fooing support
| ;;; Commentary:...
| ;;; Code:...
| ;;; Options...
| ;;; Mode...
| ;;; Commands...
| ;;; Integrations...
| ;;; foo.el ends here
`----

But according to the second part of the conventions quoted above it
should look like this:

,----
| ;;; foo.el --- Fooing support
| ;;; Commentary:...
| ;;; Code:...
| ;;; foo.el ends here
`----

And the user would have to expand "Code:" explicitly to see all the
"major sections of the program", i.e. to see this:

,----
| ;;; foo.el --- Fooing support
| ;;; Commentary:...
| ;;; Code:...
| ;;;; Options...
| ;;;; Mode...
| ;;;; Commands...
| ;;;; Integrations...
| ;;; foo.el ends here
`----

Now, I agree that it makes sense to do it that way but the problem is
that it only does so in theory, in practice this approach comes with
several annoyances.

Before we discuss those, it is worth knowing that global visibility
cycling knows four states.  I am showing them here, using "my style":

(I am also attaching screenshots of these states.  Note that the look
of the headings is due to the use of my `outline-minor-faces' and
`backline' packages.)

1-overview.png
Description: 1-overview

2-toc.png
Description: 2-toc

3-trees.png
Description: 3-trees

4-all.png
Description: 4-all

,----
| 1. OVERVIEW: Show only top-level heading.
+----
| ;;; foo.el --- Fooing support
| ;;; Commentary:...
| ;;; Code:...
| ;;; Options...
| ;;; Mode...
| ;;; Commands...
| ;;; Integrations...
| ;;; foo.el ends here
`----

,----
| 2. TOC:      Show all headings, without treating top-level
|              code blocks as sections.
+----
| ;;; foo.el --- Fooing support...
| ;;; Commentary:...
| ;;; Code:...
| ;;; Options
| ;;; Mode
| ;;; Commands
| ;;;; List Commands
| ;;;; Misc Commands
| ;;; Integrations
| ;;; foo.el ends here
`----

,----
| 3. TREES:    Show all headings, treaing top-level code blocks
|              as sections (i.e. their first line is treated as
|              a heading).
+----
| ;;; foo.el --- Fooing support...
| ;;; Commentary:...
| ;;; Code:...
| (require 'bar)
| (declare-function 'baz-ing "baz" (arg))
| ;;; Options
| (defcustom foo-option nil...
| ;;; Mode
| (define-derived-mode foo-mode special-mode...
| ;;; Commands
| ;;;; List Commands
| (defun foo-list (arg)...
| (defun foo-list-all ()...
| ;;;; Misc Commands
| (defun foo-do-stuff (arg)...
| ;;; Integrations
| (defun foo-bookmark ()...
| (defun foo-helm ()...
| (provide 'foo)...
| ;;; foo.el ends here
`----

,----
| 4. ALL:      Show everything, except code blocks that have been
|              collapsed individually (using a `hideshow' command
|              or function).
+----
| ;;; foo.el --- Fooing support
|
| ;; Copyright (C) 2020 Me
|
| ;; Author: Me
|
| ;;; Commentary:
| ;;
| ;; This package implements support for fooing.
|
| ;;; Code:
|
| (require 'bar)
|
| (declare-function helm "helm")
|
| ;;; Options
|
| (defcustom foo-option nil
|   "bla")
|
| ;;; Mode
|
| (define-derived-mode foo-mode special-mode
|   "blabla")
|
| ;;; Commands
| ;;;; List Commands
|
| (defun foo-list (arg)
|   (interactive (read))
|   (things happen here))
|
| (defun foo-list-all ()
|   (interactive)
|   (things happen here too))
|
| ;;;; Misc Commands
|
| (defun foo-do-stuff (arg)
|   (interactive (read))
|   (some code)))
|
| ;;; Integrations
|
| (defun foo-bookmark ()
|   (more code))
|
| (defun foo-helm ()
|   (require 'helm)
|   (helm stuff))
|
| (provide 'foo)
|
| ;;; foo.el ends here
`----

And here are my reasons why "Options" et al. should not be subsections
of "Code:":

A) The OVERVIEW state would become almost useless.  Personally I would
   always skip it and go straight for TOC.  When following the
   convention, then the OVERVIEW state looks like this for *every*
   library.

   ,----
   | ;;; NAME.el --- DESCRIPTION
   | ;;; Commentary:...
   | ;;; Code:...
   | ;;; NAME.el ends here
   `----

   That just isn't useful.  If one wants to see the DESCRIPTION, then
   one does not have to hide any sections; one just has to make sure
   that the first line is visible by moving to the beginning of the
   buffer.

B) The OVERVIEW shown above isn't just not useful, for more complex
   libraries (which are split into more (sub*)sections) having a useful
   OVERVIEW is quite important.

   For such libraries the TOC just isn't a suitable substitute for
   OVERVIEW.  It could be deeply nested and if one only wants a list
   of the "major sections of a program", then a deeply nested tree of
   sub*sections just isn't the same.

C) If "Options" et al. are subsections of "Code:", then there likely is
   code between the section heading "Code:" and the heading of the first
   subsection "Options".  This usually includes calls to `require' and
   `declare-function' and such.

   One could deal with that by adding an additional subsection called--
   I don't know -- "Frontmatter", but I feel that just adds unnecessary
   noise.

   It is worth noting that the TOC state does not show these pre-first-
   subsection sexps.  IMO that is a problem, as it makes is very easy
   for a user to overlook these forms.

If we put "Options" et al. at the same level as "Code:", then the latter
itself becomes the "Frontmatter" section mentioned in (C).

Sorry this has gotten so long.

     Cheers,
     Jonas

Ps: In my own libraries I put the `provide' form and the ";; Local
    Variables:\n...\n;; End:" block, if any, in their own section,
    which I name just "_" (i.e. ";;; _\n").  But I don't dare suggest
    we do the same in Emacs.

Pps: When one often collapses sections, then the ";;; NAME.el ends here"
     becomes very annoying.  I am glad to have learned recently that as
     early as Emacs 31.1 that line won't be mandatory anymore. ;P