lynx-dev Lynx is dying !?

[Pascal Rigaux]

First of all, this mail isn't a flame at lynx which is my prefered browser.
All i want here is some comments, ideas, (flames)... If you think something
below is wrong/nasty/stupid/naïve/incomplete tell me. That's what i want.

I really like lynx and am using it everyday for browsing on the net. But for
browsing on the net you need incremental rendering, caching of the html source,
background downloading. As far as i understand, most of the lynx users only use
it for local files (yes, i know, lynx is really great for that), and that's a
part of the explanation of why these important net features are not in lynx.

I've looked through nearly all the lynx-dev mailing list (40Mb) searching for
words like ``render'', ``incremental''. I'm just astonished only a few complain
about the non-incremental rendering done by Lynx though it is a one-pass engine.
What i found is a lot of claimings about caching the html source and adding 
real 
table support. And although it seems feasible, nothing's included in lynx yet.
Here i'd like to quote a passage taken from the mailing list:

> Caching issue is just one of the many Lynx annoyances the users have to deal
> with, but comparing it to table formatting, style sheets, Java(script), and
> Dynamic HTML, it doesn't seem to require as many major changes to the sources
> as the latter ones, as Lynx already has cache schemes, and the only real
> problem left is how to identify both versions of an URL, and I provided a
> suggestion above, which is better than adding a '\' at the end.

> I don't know exactly how simple it is in code level either, but it seemed the
> only reason it hasn't been implemented is because certain key developers
> loathe any close resemblance to a 'broken' GUI browser or GUI environmental
> ideas, even for sensible features on every other browser in the world but
> Lynx. To my knowledge, my suggestions for changing history list behaviours and
> the need for NEXT_DOC command were 2 of the GUI ideas that have never seemed
> to be taken seriously, even though I had provided reasons without referring to
> any graphical browsers. If Netscape and IE had never had these 2 features,
> they might have been implemented into Lynx long before reaching 2.8.

> Subsequently, with the nature of lynx-dev, many Lynx users' suggestions have
> been ignored by developers, especially the unofficial patches to get around
> many Lynx's 'features', which most users don't (care), and even if they do,
> the solutions may require exhaustive searches to the lynx-dev archive, or the
> users may not have the means to do the improvements themselves. Table
> formatting falls into this category.

> (Or course, I can add the fact that lynx developers have never got paid, but I
> doubt it has any relevance to caching issues at all since Lynx developers have
> done much more complicated things than the dumb cache scheme.)

> Combined with certain developers' insentivities to users' environments (for
> example, the great assumption that every Lynx user should be a C expert who
> should use nothing less than a 64-bit machine and some Unix variant called
> 'The real OS'), caching issue is unlikely to get fixed anytime soon unless
> they finally awaken from their great assumptions and begin to drop some of
> their elitist attitudes towards Lynx users.

This mail was written by Jacob Poon. And it seems to be the truth. If something
as easy as the NEXT_DOC command he asked for in december 96 hasn't been
``implemented'' yet, it means that lynx should fork between people wanting to
keep it the smaller it can, and people wanting more.

It seems from all this that lynx is stuck. That any addition of features is
impossible or it'll break a lot of other features. This doesn't surprise me.
Lynx is able to manage a lot of different architectures. It also started long
ago, and the html specifications have evolved in the meantime. The result of
this is any major improvement will break a lot of things.

The conclusion of all this is: if you want more than lynx you must rewrite it.
I've talked about it with some friends and here is the result:


*WHY****************************************************************************
Main lynx weaknesses:
 - no incremental display (you can't see the beginning until the whole page has
been loaded)
 - no table handling
 - minimal frame handling
 - no internal saving of the html source file
 - the errors and warnings aren't nohup'ed, you have to wait for the message to
end. So either you reduce the time the message is displayed and sometimes you
can't even read it or you keep a good time and message like "You are already at
the beginning of the document" are very nasty.
 - no incremental search

*WHAT***************************************************************************
text-based:
 - enough most of the time
 - you don't get any advertising

emacs-based to begin with (other interfaces could be added):
 - ease
 - you don't have to leave emacs -> cut/paste...
 - the emacs's windows could be used to display the frames

features:
 - incremental rendering (you can see the beginning while downloading the end)
 - frame handling (most of the time a bad rendering is enough to let choose the
frame you want to see in full. With lynx you sometimes don't know what frame to
choose when the name of the frames are not well chosen)
 - send the current to netscape (be able rapidly to see it through a full
featured browser)
 - download files in background (à la netscape) -> multi-threading
 - be able to see the html source without reloading

*HOW****************************************************************************

diagram:
            ___
           /   \
          |     |
        server  |
          |   _/
          v   /|
          |  /
          | /
         GET ------------<--------.
          |                        \
          |                         \
          |                          \
      html file                       \
          |                            \
          |                             \
          |                              \
          |---->----- RENDERER ----<------.
          |              |                |
          |              |                |
          |              |                |
          |       formatted text       commands
          |              |                |
          |              v                |
          |              |                |
          |              |                |
          `---->----------------->----- EMACS


* GET: simple program able to download URIs and send POSTs.
* RENDERER: it takes the html file from GET, and it takes the width of the
display from EMACS. It outputs a formatted text.
* EMACS: it does all the interaction with the user, and it manages GET and
RENDERER.

The problem being broke into pieces, it's much easier. And instead EMACS one 
can 
use a curses-based interface (or anything someone need). Starting by using 
EMACS 
for this is nice because a lot of thing is already done.


*GET****************************************************************************
- be able to get html, ftp files

- be able to post forms ??? (TODO)


*RENDERER***********************************************************************
- is incremental

- characteristics of the standard output
  * text formatted for a given width (in characters)
  * contains formatting tags to indicate some styles (like bold, reverse, 
colors)
  * contains URI-like tags to indicate to which part of the text is associated
  which action
  * contains tags indicating radio buttons, toggles, menus, submit... that are
  part of the <FORM> tag
  * MAYBE(??) can contains a special tag meaning that all the rendering is
  restarted. That way in really extreme cases, we are able to not do things
  incrementally.        

- the stderr is used to output warnings and errors


*EMACS**************************************************************************
- handles the browsing, scrolling (pageUp...)
- handles the anchor tags (<A>)
- must handle all the <FORM> tag widgets (radio buttons, toggle buttons,
entries, combo boxes)



So that's the project i want to start. A friend of mine wants to do the emacs
part. Another one is interested by the RENDERER, so am i.



PS: I'm not good at all at writing. Moreover I'm french and my english is, well,
not great, so don't be to hard upon me for the (many) faults of english you'll
find.



Pixel.