Re: did behaviour of RET change again?

[Tom Gillespie]

> possibly i'm misunderstanding, but my sense is that the value of org
> adapt indentation doesn't change what you might actually find ("in a
> .org file in the wild").  so, whatever its value, your grammar would
> have to deal with all cases?

Yep, we can't magically change all the files out in the wild.
Until I wrote this email out I agreed about the grammar too, but
as it turns out I think there might be a compromise, which is that
a grammar could be allowed to interpret certain types of lines
with leading whitespace as if they were paragraph lines instead of
as say, the start of an org-babel block. That way an interoperable
org spec could be made simpler without preventing e.g. the elisp
implementation from going above and beyond the spect to provide
support for leading whitespace.

My interest in the org-adapt-indentation setting is to try to align what
most users are doing out in the wild with a representation for their
org files that is less ambiguous and more performant (among other
things).

If I had to guess I would say that most new org files have leading
whitespace precisely because org-adapt-indentation is set to t by
default. Getting users to transition their files requires an incentive,
and some users may find that they use org in such a way that they
don't need to.

While writing this email I thought of a reasonable incentive, which is
that only files without leading significant whitespace (i.e. that look like
those that are authored with org-adapt-indentation nil) have specified
behavior for things like org-babel blocks. This would allow best effort
by the elisp org-mode implementation to allow users who don't care
about interoperability to continue as they have been doing, while also
providing clear guidance for what users must do if they want
consistent behavior on other tools that consume org formatted files.
This is critical for keeping the org spec from becoming overly complex.

The first step would thus be to reduce the rate at which new org files
are created with leading whitespace by changing org-adapt-indentation
to be nil by default.

The second step would be to give users a clear way forward to safely
normalizing their files. Org has a habit of reindenting subsets of files
from time to time, but in general doing a complete switch to have no
significant whitespace can be risky.

The third step would be to let the incentives and needs of users
play out over time, but users would generally probably be happier
because by default they would be in the "my files are portable and
generally interpretable without me having to do any additional work"
state instead of the "why did no one tell me the defaults weren't
portable" state.

> or, and maybe this would make sense, you'd define an "interoperability"
> form of .org, that all "wild" .org files could be (programmatically)
> converted into, without losing any of their semantics?

Yep exactly. For many use cases the cost of stripping the whitespace
that corresponds to heading level is not unreasonable, e.g. since it
would only have to be done once. However, if you are writing another
org implementation, then every single time you parse a heading and
its accompanying section you have to strip the whitespace, and that
will happen every single time a user modifies the file, which starts to
get expensive. Alternately you could implement it so that everything
is stripped once and then you keep a version in memory that the user
edits which doesn't have leading whitespace, but then every time you
save you have to splice it all back in to preserve the roundtrip.

This also doesn't even begin to deal with the fact that users can create
malformed org files that can have lines that are less than the expected
significant indentation. This becomes a complete nightmare when trying
to come up with some rational way of dealing with org babel blocks for
languages like python where there is significant whitespace.

Consider the horror if trying to specify the correct behavior in a situation
like this. Better just to declare it a malformed babel block and move on.
Unfortunately you can't always detect that such things are malformed
during the first pass of parsing because you have to count the number of
spaces.

#+begin_src org
# -*- org-adapt-indentation: t -*-
,*** Oh No
    ,#+begin_src python
  class What:
      have = 'you'
      done = 'now'
,#+end_src
#+end_src

In order to ensure that org files can be reliably interpreted this either
means that the specification for handling cases like this becomes
extremely complex, or the spec can say "you can have leading
whitespace, but nasal demons may come for you, there is only
specified behavior if you do not have leading whitespace."

Having unspecified behavior in cases like that would mean that an
implementation could be fully compliant if it were to treat any
#+begin_src lang line that started with whitespace as if it were
a paragraph instead of a babel block. I suspect that this will be
the best way forward.

> (anyway, good luck with that, even with any significant subset of that!)

Thanks, and thanks for the inspiration!
Tom