[Groff] The future redux

[Peter Schaffter]

I'm afraid this will be a long post.  Sorry, but I don't see any way
around it.

It's ironic and instructive that the thread, "Future direction of
groff", which became a semantic-vs-presentational debate eerily
similar to a previous discussion on the same subject, originated
in the "space-width" thread, which deals with the minutiae of
presentation.

Re-reading the posts, a number of things become clear.  First,
groffers on the list value quality typesetting, and there's
considerable discomfort with moving away from the physical page,
conceptual or otherwise, to the multi-modal paradigm.  Secondly,
there's a perceived conflict between backward compatibility and future
development.  Thirdly, more than one list member expressed fear that
improving groff will somehow break it.

Lastly, there's Eric's insistence that "if it ain't semantic it's a
dinosaur," which was greeted with as much resistance this time as
the last time the subject came up.  Allow me to say here that Eric
may well be right.  His indispensable contributions to open-source
development lend considerable weight to any argument he makes.  He's
been an inspiration to me for years.  I'm disinclined to dismiss out
of hand any opinion he offers.

If groff is to have a future, if it is to remain vital, it is clear
these issues need to be looked at.  More than groff is at stake.  Do
we live in a world where typography itself is becoming an archaic
craft, like stone masonry?  Is print really dead, as some have
asserted?  Is there a future for documents that are beautifully
typeset but only render properly as PDF or PostScript?

And, if the answers are, in order, "no," "no," and "yes," do we
really need both groff and TeX when TeX is, at present, more
typographically sophisticated?

I think, too, we need to consider the distinction between
"documentation" and "a document".  Is the primary purpose of groff
to produce documentation, which requires an ordered structuring of
the subject matter, or to produce material of a more fluid nature
where the presentation (the typography) serves expressive rather
than structural needs?

* Backward Compatibility

Mike Bianchi summed up the backward compatibility concern best:

 "The fact that I can still format documents I wrote in the 1970s
  and beyond is valuable to me, and, should any of them ever become
  classics, possibly to others in the future.  So no, do not break
  groff by 'modernizing' it."

Backward compatibility has been a mainstay of *roff throughout its
history, and any effort to improve groff should remain true to that
goal.  However, ultra-strict adherence to backward compatibility
must not stand in the way of improvements to existing functionality
and the quality of final output.

While it's a wonderful convenience to use groff in 2014 to format
documents from the 1970s, it must be remembered that a) those
documents exist as plain text, thus changes to groff would have no
impact on the readability of their content, and b) the formatting
directives in those documents, equally, is plain text, and thus
easily understood presentationally and semantically--by humans, if
not by Eric's "baroque AI baby." :)

In short, if some backward compatibility were to be lost in future
versions of groff--and frankly, I doubt it would--documents written
in the 70s would remain as useful and valuable as ever.  The only
thing that might be lost is the time it would take to write a Perl
script to parse and update said documents to reflect probably very
minor changes to groff's behaviour.

Furthermore, let's not forget that under Werner's leadership, groff
underwent a considerable amount of modernization.  Nothing got
broken, and the whole groff picture improved splendidly.  I see no
reason to fear that future development will be any different.

* Fear of modernizing groff

Mike wasn't the only one who expressed fears about the modernization
of groff.  Walter Alejandro Iglesias wrote:

 "In general terms what in the name of *modernity* developers have
  being doing with Unix, by Unix I mean the idea behind, what made
  of it a well designed OS, has being just adapting it to those used
  to the MSWindows experience.  Marketing.  First that modifications
  took place in userland, but today, base, init and even the kernel
  are suffering them."

While I share Walter's feelings, I don't think they apply to the
kinds of changes we've all been dancing around that need to be
made to groff.  For example, groff's line-at-a-time approach to
formatting, if unchanged, will remain an impediment to high quality
typesetting and ensure groff's demise for anything other than
writing manpages.  Since the point of implementing page-at-once
formatting (or, as Werner dreamed, document-at-once) would be
to improve the quality of typeset output, not to change the
fundamentals of groff usage, resisting such a change seems like
misplaced Luddism.

* Groffers love good typography

I scarcely need comment on this.  The most interesting discussions
on the list, the ones that generate the most responses, deal with
fussiest of typographic details: how much space to put between
sentences, the length of dashes, hanging punctuation and margin
correction for glyphs, letter-spacing versus font expansion, and so
on.

Why this love is important relates to the future of typography in
general.  We are concerned about these things because we do not
accept shoddiness.  Since the advent of DTP, and hastened by the
rush toward generalized input suitable for multi-modal display,
there has been, globally, a serious decline in the quality of
typesetting, by which I mean the balanced, aesthetically-pleasing
arrangement of words regardless of the display medium.

Good typography lends weight--gravitas--to content, in addition
to facilitating the act of reading.  It makes words come alive
and encourages contemplation of the ideas they represent.  Good
typography says, "These words matter and deserve respect."

It is an ongoing battle, tending the fires of quality in the face of
widespread dilettantism.  Perhaps I am guilty of adopting a doomed
Romantic stance (wouldn't be the first time), but I believe groff,
and the community of users it attracts, are essential to keeping
the craft of typography alive.  The same is true of TeX and its
community.  People respond well to quality typesetting, and there's
no reason why they shouldn't expect it, if not now, at least in the
future.  If we allow groff to remain as-is, if we cease to look for
ways to improve the quality of output and refine the tools used to
create documents, we are doing a disservice to the future since it
is in our hands to preserve what, of the past, should not be lost.

* The great presentational vs semantic markup debate

>From a typographic standpoint, markup, whether presentational or
semantic with linked stylesheets, is only as useful as the program or
device interpreting it.  There's a reason why _The Binbrook Caucus_,
a novel I put online a few years back, isn't in html: browser
rendering of type sucks.  I doubt that's going to change any time
soon.  The novel uses typography to express changes of tense and
POV, as well as to convey things like verbatim newspaper articles,
train station announcements, email correspondence, and typewritten
copy.  For readability, it requires fixed margins and a degree of
control over justification that's impossible to achieve with, eg,
xhtml, which is my preferred way of formatting web pages.

For similar reasons, the novel isn't ePubbed or mobied on Kindle.
Only PDF, page-centric and type-specific, is capable of rendering
the work according to my intentions.

Eric says: 

 "What I don't believe is that there will ever again be enough
  demand for printer-*only* output to justify markup formats
  and toolchains that don't also do web and ePub or functional
  equivalent."

In this he may well be right, but he is speaking of a world where
precise control over typography no longer plays the role it
does presently in document design.  A world where "approximately"
rules--fine for certain types of writing, notably documentation,
technical papers, blogs, and the like, where, whether on paper,
a monitor or a smartphone, it is enough for a headings to "look
like" headings, paragraphs to "look like" paragraphs, nested lists
to "look like" nested lists, and so on.  And it is certainly in the
best interests of future markup formats and toolchains to do their
best at generating output renderable on the greatest number of
devices.

Interestingly, this was what the various *roffs aimed at originally:
device-independent output that could be fed to various drivers.
But somewhere along the line, I believe, that mandate lost its
importance.  After the valiant attempt of grohtml, no new drivers
were added until Deri James contributed gropdf.  Had the mandate
retained its importance, the question of whether groff should have
an XSL-FO driver would be moot because, in all likelihood, it would
already have been written, along with several other much-needed
drivers, grortf being the most pressing for many of us (and I wish
to God we didn't all hate RTF so much so somebody would, in fact,
write the driver).

I think what happened is that, over time, near-exclusive use of the
PostScript driver caused many of us to confuse groff output with
grops output--if not intellectually, at least at the conceptual
level.  We began to think of groff as a PostScript typesetting
engine.  Few are the posts in the last ten years dealing with, say,
terminal output issues, while legion are the posts about what are
clearly PostScript presentational issues (like the "space width"
thread).

I suspect the uneasiness with what Eric has to say about groff's
future, and the whole semantic-vs-presentational debate, stems from
Eric addressing the issue in terms of groff's original mandate
(device-independent output, a concept largely replaced by the notion
of display-neutral xml), while everyone else looking at groff as a
typesetter for the paper or full-screen-viewable documents that,
Eric's doom and gloom predictions aside, continue to form the bulk
of lists members' groff usage.  Consider Tadziu Hoffman's comment,
which mirrors my observations, above:

 "Personally, I use groff exclusively for printing out stuff (or
  creating PDFs as a sort of virtual paper)--something to be read,
  as is, *by humans*--nothing else.  (For the web (if I must), I
  write HTML.)  In principle, I could also print from the Browser,
  but the results are ghastly.  I would not be willing to compromise
  typesetting quality in exchange for additional media I have no
  plans of using."

We cannot ignore the need for groff to accommodate the Web and
ePub-y type things, despite its paper-centricity.  Eric is quite
right that "printer-only" will never again be enough.  However, as
he also points out, attempting to extrapolate semantic meaning from
groff output is impossible "... because the information required to
do that is thrown aware at macro expansion time."  The conclusion
he draws from this strikes me as self-evident: "The difficult but
correct thing to do is to recover structural information by looking
for cliches in the source markup *before* it goes through troff."

But why "difficult"?  Well, mostly owing to historical groff
(mis?)use, which fostered conditions where, in Mike's words,
"...presentation and formating are horribly intermingled."  The
classical macrosets are not well set up for creating stylesheets
"that really sing" (esr).  Groff-hands got in the habit of dealing
with presentational issues for semantic elements on-the-fly,
either with groff primitives or by writing their own macros--the
cliches of which Eric speaks.  Had there been, historically, a
macroset that provided a fairly complete set of semantic tags and
an easily-parsable mechanism for applying presentational markup
to those tags, Eric might never have had to write the miracle of
cleverness that is DocLifter.

Without a single change to groff as it stands now, all that's really
required to generate xml from groff source files are a) well-formed
source files, and b) mechanisms for parsing and transforming them
into xml.  (I speak here of present and future documents, not the
vintage stuff.)  If I can convert a mom file to xhtml using sed
(gasp!), xml and xml stylesheets are perfectly feasible with more
sophisticated tools.

The trick, of course, is writing well-formed source files, with
clear a distinction between metadata, stylesheet, semantic tags,
and discardable presentational markup.

* The mom macros

Mom is my baby, but that's not why I'm mentioning it here.  I was
really surprised by Mike's comment: 

 "Done right, a really great macro package would have to clearly
  separated parts: presentation and format.  But it seems *roff has
  never really provided the architecture to support that sort of
  separation, hence macro packages that mush the concepts together."

The mom macros were conceived, from the start, to do exactly what
he describes: keep presentation and formatting separate.  Every
semantic tag has a bevy of "control macros" that permit the styling
of tags separately from the tags themselves, macros that furthermore
begin with the name of the tag, making their intent instantaneously
clear, both to human eyes and to a parser assembling a stylesheet.  A
mom document begins with metadata, is followed by a stylesheet, and
begins formatting, in the sense Mike means, only after the .START
macro is issued.  Throughout the remainder of the document, there's
virtually no need for presentational markup, except discardable
markup like tightening or loosening a line with track kerning
(which, moreover, is done in macro space with the easily identified
EW [extra white] and RW [reduce white]).  In cases where it's
desirable to make available "in-document" presentational markup of
a tag, say to adjust the position of a table, the presentational
markup is attached directly to the tag as an optional argument, which
can be flagged and ignored at parse time.

Furthermore, mom's typesetting functions can be used independently
of document formatting (again, in Mike's sense) to create documents
never likely to be rendered anywhere but on paper (a poster
advertising kittens free to a good home, for example).

In short, groff has always had the "architecture" to support source
files that are both human- and xml-friendly, it's just that it's
rarely been taken advantage of.

* Summing up

Groff can, and does, have a future, one that accommodates both
printer (or analogous) output and display-neutral xml.  The
typesetting engine/pipeline (ie ditroff=>grops/gropdf) needs to be
overhauled to remove the hurdles posed by line-at-a-time processing.
Historical baggage like not respecting order of precedence can,
I believe, be fixed without earth-shattering consequences.  Adding
arrays, as has been suggested, wouldn't break anything.  (Myself, I
wouldn't mind case statements, as I find I'm increasingly using
while loops to simulate getopts functionality.)

I think we all agree some requests could do with a dose of sanity,
which could be accomplished by creating new requests in the manner
of .de1 or .am1 whenever altering their historical behaviour would
break existing documents.

And all this is all doable, it seems to me, without affecting backward
compatibility.  I don't think we have to consider forking; in fact,
I'm against it.  What we do have to do is acknowledge that groff, as
a typesetting engine, has the potential to stand next to TeX, and,
as such, remain a viable choice for quality typesetting.  I really
don't believe that printer output, or the physical page paradigm for
screen documents, is anywhere near falling into desuetude.

As for xml output, I'm convinced that's a source file, macro level
issue.  The mom macros point the way for xml-friendly structuring of
source files; who knows what a joint-development effort in a similar
vein could accomplish?  

Again, sorry for the long post.

-- 
Peter Schaffter
http://www.schaffter.ca