lynx-dev README.TSST

[Klaus Weide]

Here's a vapourware announcement.

Well it's not complete vapour, it exists, but still needs some work.
Part of the purpose of this announcement is to give me a reason to
finish it up... (strike the "it wasn't too difficult" below, I guess,
although for really-SIMPLE tables it's true.)

If someone wants a patch (which may not apply cleanly to an existing
dev.N) right now, in unfinished but mostly working state, send mail
to me directly.

    Klaus
------------------------------------------------------------------
Tabular Support for Simple Tables
=================================
Some definitions first:

* NO table support
  What it says. :)  Table related tags are treated like other
  completely unrecognized tags.
  Only listed for completeness, this does not describe Lynx.

* MINIMAL table support
  Table related tags are recognized, and are used to separate
  the contents of different cells (by at least a space) and rows
  (by a line break) visibly from each other.

* LYNX minimal table support (LMTS)
  The minimal table support as implemented by Lynx up to this point,
  also includes the way ALIGN attributes are handled on TABLE, TR
  and other specific tweaks (e.g. handle TABLE within PRE specially).
  LMTS formatting is briefly described in the Lynx User Guide, see
  the section "Lynx and HTML Tables" there. (The Users Guide has not
  yet been updated for tabular support.)

* TABULAR support for tables
  Support for tables that really arranges table cells in tabular form.

* Tabular Support for SIMPLE Tables (TSST)
  Tabular support for some tables that are 'simple' enough; what this
  code change provides.

One basic idea behind providing TSST is that correct tabular support
for all tables is complex, doesn't fit well into the overwhelmingly
one-pass way in which Lynx does things, and may in the end not give
pleasant results anyway for pages that (ab-)use more complex table
structures for display formatting purposes (especially in view of Lynx
limitations such as fixed character cell size and lack of horizontal
scrolling; see also emacs w3 mode).  Full table support within Lynx
hasn't happend so far, and continues to seem unlikely to happen in the
near future.

The other basic idea is the observation that for simple tables, as
used mostly for data that are really tabular in nature, LMTS rendering
can be transformed into TSST rendering, after parsing the TABLE element,
by two simple transformations applied line by line:
- Insert spaces in the right places.
- Shift the line as a whole.

And that's exactly what TSST does.  Once the simple idea existed, it wasn't
too difficult to implement.  On encountering the start of a TABLE element,
Lynx generates output as ususal for LMTS.  But it also keeps track of cell
positions and lengths in parallel.  If all goes well, that additional
information is used to fix up the already formatted output lines when the
TABLE ends.  If not all goes well, the table was not 'simple' enough, the
additional processing is cancelled.  One advantage is that we always have a
'safe' fallback to well-understood traditional LTMS formatting: LSST won't
make more complex tables look worse than before.

So what are 'simple' tables?  A table is simple enough if each of its TR
rows translates into at most one display line in LTMS formatting (excluding
leading and trailing line breaks), and the width required by each row
(before as well as after fixup) does not exceed the availabe screen size.
Note that his excludes all tables where some of the cells are marked up as
block elements ('paragraphs').  Tables that include nested TABLE elements
are always specifically excluded, but the inner tables may be subject to
LSST handling.  Also excluded are some constructs that indicate that markup
was already optimized for Lynx (or other browsers with no or minimal table
support): TABLE in PRE, use of TAB.

The description so far isn't completely accurate.  In many cases, tables are
not simple enough according to the last paragraph, but parts of each TR row
can still benefit from some LSST treatment.  Some partial treatment is done
for some tables in this grey zone, which may or may not help to a better
display, depending on how the table is used.  This is an area where tweaks
in the future are most expected, based on experience with real-life pages.
Specifically and currently, tables qualify for this if they meet the
following conditions:
- The table is 'simple' according to all criteria set out in the previous
  paragraph, except that some cells at the beginning and/or end of TR rows
  may contain block elements (or other markup that results in formatting
  like separate paragraphs).
- There is at most one range of (non-empty) table cells in each row whose
  contents is not paragraph-formatted, and who are rendered on one line
  together by LMTS, separate from the paragraph-formatted cells.  Let's
  call these cells the 'core' of a row.
Fixups are then only applied to the text lines showing the 'core' cells.
The paragraph-formatted cells are effectively pulled out before/after
their row (no horizontal space is allocated to them for the purpose of
determining column widths for core line formatting).

This is expected to be most useful for tables that are mostly
simple tabular data cells, but with the occasional longer
text thrown in.  For example, a table with intended rendering:

      --------------------------------------------------------
     |  date  |   item no.   |  price  |       remarks        |
     |--------|--------------|---------|----------------------|
     | date-1 | item #1      |   $0.00 |                      |
     |--------|--------------|---------|----------------------|
     | date-2 | item #2      | $101.99 | A longer annotation  |
     |        |              |         | marked up as a block |
     |        |              |         | of text.             |
     |--------|--------------|---------|----------------------|
     | date-3 | long item #3 |  $99.00 |                      |
      -------------------------------------------------------- 

It may now be shown by Lynx as

    .................................................

      date    item no.    price  remarks
     date-1 item #1        $0.00
     date-2 item #2      $101.99

     A longer annotation marked up as a block of
     text.

     date-3 long item #3  $99.00

    .................................................

As can be seen, this is still quite far from the intended rendering,
but it is better than without any tabular support.

TSST and Partial Display
------------------------
[ Partial display mode is the feature which allows viewing and scrolling
of pages while they are loaded, without having to wait for a complete
transfer. ]  During partial display rendering, table lines can sometimes
be shown in the original formatting, i.e. with horizontal fixups not yet
applied.  This is more likely for longer tables, and depends on the state
in which partial display 'catches' the TSST code.  Sometimes the display
may flicker: first the preliminary rendering of table lines is shown, then
after loading is finished it is replaced by the fixed-up version.  I do
not consider this a serious problem: if you have partial display mode
enabled, presumably you want to be able to see as much data as possible,
and scroll up and down through it, as early as possible.  In fact, the
approach taken keeps Lynx free from a problem that may graphical browsers
have: they often cannot render a table at all until it is received in full.

------------------------------------------------------------------------

To summarize:
 - TSST is a solution that works in many cases where lack of tabular support
   was most annoying.
 - TSST doesn't implement a full table model, and it is extremely unlikely
   that it will ever be the basis for that.  So keep on exploring
   those external solutions, or perhaps waiting for (better: working on) a
   fundamental redesign of Lynx's rendering engine.