Re: the compatibility of man(7) (was: man(7) .TH font change, was: groff man(7) `B` macro...)

[Ingo Schwarze]

Hi Branden,

G. Branden Robinson wrote on Sat, Jul 16, 2022 at 07:37:54AM -0500:
> At 2022-06-19T19:28:26+0200, Ingo Schwarze wrote:

>> Given that the base system corpora of FreeBSD and NetBSD are of the
>> same order of magnitude as in OpenBSD (and treating DragonFly as a
>> FreeBSD fork for now, which is not fair in general, but good enough
>> for this particular purpose because Dragonfly is likely to evetually
>> merge FreeBSD manual page improvements, even if with a long delay)
>> the estamited total number of mdoc manual pages might be about 15k,
>> with each of the major BSD systems contolling rougly 20-25% each, and
>> third-party software controlling maybe on the order of 20-40%.
>> 
>> To summarize, the *BSD base systems very likely collectively control
>> the majority of existing mdoc(7) manual pages, whereas the Linux man
>> pages project likely controls on the order of 1% of the existing
>> man(7) pages - even when we consider neither commercial UNIX systems
>> nor proprietary software.
>> 
>> That makes compatibility in man(7) significantly more of a concern
>> than in mdoc(7).  All the same, i would certainly not consider
>> adding anything as disruptive as .MR to mdoc(7).

> This argument amounts to saying that because the BSDs control both the
> majority of the extant mdoc(7) pages in the world and the mandoc(1)
> renderer that has been made the default on those systems (at the expense
> of all troffs),

Not exactly "at the expense"; the systems continue providing,
recommending, and using GNU roff for serious typesetting business,
just not for the everyday task of formatting manual pages for the
terminal, nor for the purpose of serving manual pages on the web.

> then the mdoc(7) language can be allowed to evolve.

Actually, i did not yet introduce a single new feature into the
mdoc(7) language that is seriously backward-incompatible unless
i'm forgetting about something.  Oh well, we dropped the requirement
to obey the historical eight-argument-limit about a decade ago
because it was very painful for manual page authors and seriously
anachronistic.
Admittedly, the reason for refraining from backward incompatibility is
not only that even in *BSD, a seriously backward-incompatible feature
would cause disruption for many years to come (even if not quite
as bad as in man(7)), but also that while the mdoc(7) langauge
has a few minor defects, none of them are so severe that they really
necessitate backward-incompatible changes.  There have been significant
amounts of deprecentation aiming to reduce the amount to which some
of the less-well-designed features are used.  Also, i added one new
macro to the mdoc(7) language, but in a fully backward-compatible
way.

> It furthermore implies that because the groff project does not control a
> large proportion of the world's man(7) pages (though I have striven to
> make what we do provide exemplary), the man(7) language _cannot_ be
> allowed to evolve.

My opinion would indeed be that it would be better to avoid evolving
it in a backward-incompatible way and add backward-compatible
extensions in a cautious manner.  There may be exceptions; for
example, i would likely have supported the addition of the .UR
macro had i already been around groff back then because it added
a stubstantial new feature for information that typically wasn't
included in manual pages at all before it was invented, so only
being partially backward-compatible was maybe acceptable in this
case.  Yet, an equivalent design with better compatibility would
probably have been possible then, too.

> I do not accept this proposition.  You are insisting upon stagnation.

That isn't my intention.  Since that is your interpretation all
the same, we can maybe agree to disagree in this respect.

> Yes, distributors that incorporate packages with man pages using the new
> `MR` macro, racing ahead of those same distributors' update of groff
> 1.23.0 (whenever that happens), run the risk of some unhappy man page
> readers.

As i explained earlier, people doing packaging will rarely be
aware what the technical requirements for formatting any given
manual page are, and when i watch packagers, i don't usually see
them scrutinizing manual pages for good formatting.  And again, the
choice of using .MR will not be made by *any* packager, but by the
upstream project.  So the packages would only have the choice to
delay the package update, waiting for a typically *different*
packager to update groff.  It is not clear delaying a software
update for such a reason is even a good idea.

> If a distributor takes a wait-and-see attitude toward groff 1.23.0,
> whether deliberately or because their package maintainer is comatose,
> then another contributor who doesn't have a great command of *roff can
> apply the following patch to the man.local file they ship.
> 
> .if !d MR \{\
> .  de MR
> .    IR \\$1 (\\$2)\\$3
> .  .
> .\}
> 
> (This is a simplified version of what we provide in an-ext.tmac, where
> it is used only if the formatter is not groff.[1])

That may be a possible workaround, also for other manual page formatters
that some systems might package.

> You have persuaded me that the foregoing is a point worth cautioning
> people about in the groff 1.23.0 release notes, which with any luck will
> be read by a wider audience than this mailing list or any other piece of
> the groff distribution.

Maybe.  A statement like

  Even if you do not yet update the official groff package in your
  operating system just yet, it might be necessary to add the
  following workaround to your existing groff package, at the
  latest when the first application software starts using the
  new man(7) .MR macro in their manual pages.
  The same may apply to manual page formatting software other
  than groff that you operating system may provide.

is probably somewhat unusual in release notes, but it could help
to alert at least some package maintainers in some systems to
the upcoming problem.

> The man(7) language is one of many things in the traditional Unix world
> that never bothered to declare a versioned interface mechanism.  (I've
> become convinced that doing so is essential for serious software
> engineering work--yet look how long ELF symbol versioning took.)  We
> could certainly add one to it.

I already explained why this would not help very much: manual pages
are a secondary feature of most software projects, and if an upstream
project declares a dependency on a language version one of its
downstream operating systems does not or not yet ship, there is
little that can be done, except maybe crude workarounds.

> Man pages could say
> 
> .NE 2.1 \" "needs"

Inside the manual page file is the wrong level for such a declararation.
Once the manual page source code arrives at the formatter, it is *much*
too late to get a formatter supporting the version.

Instead, the upstream projects would have to declare a dependency
in a similar way as they declare "this project requires C11".
And even that helps only in part.  It's clear you can't ship a
software until you have ported the required compiler - but would
you really want to not ship a software merely because it declares
a manual page formatting dependency not or not yet avaialble
on your platform?  So, ugly workarounds is the best you could hope
for even with a perfect versioning scheme.

> and man(7) renderers could declare that they "provide", say, "3.4".
> 
> It would then be possible for interactions between future man pages and
> their renderers to be negotiated and gracefully fail if unsatisfied.

Nothing to be negotiated there.  Either the formatter supports
the requested feature and can handle the manual page, or it
cannot and the user is out in the rain.

> It wouldn't displease me in the least to apply a semantic versioning
> practice to the man(7) language and our implementation of it.

Even if all projects providing man(7) implementations would actively
participate, i'm unconviced it would help much.  Also, don't forget
there are not only many programs generating poor man(7) code, but
lots of programs attempting to parse man(7) code in some ad-hoc
way.  Yes, that is unadvisable, but i think significant numbers
if such programs exist nontheless, often obscure and barely
maintained.


Since you seem determined to ship the new, backward-incompatible
feature, i guess we should conclude the discussion, admit that
we disagree whether it's a good idea, and just wait and watch
what happens.

Maybe we are lucky and the "all the world is Linux" effect saves
us, in the sense that people using commercial operating systems
rarely speak up and many of them are used to the experience that
when something doesn't work, they need to install the latest GNU
tools in addition to whatever ships with their system.  People
using Plan9 are likely smart enough to find some way to cope.
In mandoc(1), i'll simply implement .MR such that you are less
exposed to complaints from *BSD systems, and such that *BSD users
see less disruption.

And then wait and see and hope for the best...

Yours,
  Ingo