Re: LYNX-DEV libwww5

[Klaus Weide]

A while ago, Henry asked...

> [...] WHAT the www library is?  About all I am vaguely cognizant of is
> that half or more of the time compiling lynx is really compiling
> the www library, and that Fote has overhauled the code (libwww3 ?)
> so much that he calls it `libwww-FM/2.14'.  In more simple terms,
> I mean why does lynx need this library?  For example, I know that
> to get things printed on the screen, lynx needs libcurses (or slang
> or ncurses). 

The ex CERN, now W3C, Library Of Common Code or Reference Library,
was probably the first widespread piece of Web code.  It seems that
as long as the Web has existed there was this library, and in the 
beginning, the library was the Web...  but maybe I am just building
my own mythical explanations here for things about which I have no
knowledge.

Anyway, I'll *try* to say something more real..  The Library contains,
in theory, all the code that a Web programs would need in order to,
well, access the Web.  I mean, everything *common* to such applications,
be they client or server or spider, independent of user interface or
display, or of special additions.  That's why (more than? less than?
too lazy to count kilobytes now) half of the Lynx code is Library code.
That way Lynx didn't have to reinvent all the wheels when it became 
a Web browser.  (You know, I wasn't there, I am making this stuff up 
as I go, but it makes sense doesn't it :) )

But the library is not being used in the same way as for example the
ncurses library, which has a well-defined interface in terms of functions
which are called by an application, which is otherwise totally separate
from the application and doesn't become a part of it.  I don't know
whether the Library was ever *meant* to be used that way, anyway;
it seems that it was always kind-of experimental, kind-of example code,
and any application which would use it was expected to modify part
of it according to its own needs.  And new versions of the Library
just cannot be used like a new version of e.g. ncurses or slang, 
compile-and-go (or download-binary-and-go) style, there isn't anything
like the level of stability or compatibility with older versions that
you would expect from a binary library.

There is still the notion of an "application", which is "using" functions
from the Library.  But the distinction between the "application" portion
of the Lynx application and the "Library" portion of it is not clearcut,
and it has been blurred more as time passed and more and more for-Lynx 
changes were made (*had* to be made) in the Library part of code as
well as in the, strictly, "application" part.

It also doesn't help maintain a clear distinction that now part of the
W3C Library is regarded as "core", while important and large parts are
regarded as "belonging to the application" but are clearly part of the
library distribution and a focus of its development (e.g. the HTTP 1.1
protocol). 

Anyway...

So, what does the Library do?  Well, I have tried to say that the question
should rather be phrased "What does the Library portion of the Lynx code
do?", and the rough answer is "all kinds of things" (which would be useful
to more than one Web application), a slightly better answer is "go to the W3C
Web site and browse their material" (obviously not for what, exactly,
the Library code is doing for the Lynx application, but for the general
scope of things that fall under library functions) - and my answer (which
is probably the worst of all) is, summarily:  

Everything that is close to the network belongs to the Library part:
The TCP protocol, protocols like HTTP, Gopher, FTP, etc.
Access to the local file system is, theoretically, a function of the 
library, but has been modified and enhanced so much that de facto it
isn't any more...
Parsing URLs and keeping track of them is a Library function.
(but also with lots of modifications in the Lynx code).
Basic data structures (like HTAnchor, there's one for every URL Lynx
ever encounters) and conventions are defined in the Library, as are
methods for passing data as streams between modules.
The Library provides some utility functions and data types that are
used throughout the Lynx code (StrAllocCopy & friends, HTAtom) and
of general usefulness (not just restricted to Web applications).

Managing the display, accepting keystrokes, interacting with the user
and keeping track of what he/she wants are not functions of the library.
Building the data structures for parsed text (and parsed links), and
everything from there to displaying a page are not functions of the
Library.
Checking permissions and restrictions, and filtering out nonsensical
requests from the user, are done in the non-Library portion (but handling
authorization for remote servers is the Library's domain).

While I have listed some important things that came to my mind, you will
probably have noticed that so far I haven't mentioned the part of Lynx
which is probably the most important to many people, and the one where
changes have happened all the time: the parsing of HTML.  I didn't talk
about it because I don't know where to draw the line between library
and application, and I really haven't looked much at that code.  
Very roughly, the Library provides (and determines) the basic structure,
but Lynx has to fill in all the details.  But let somebody else talk about
it who understands it better.

While Lynx was using (and adapting to its needs) an older version of the
Library code, new functions were added to the developing W3C Library.
That includes provisions for doing several things asynchronously or
simultaneously, and support for a more event-driven structure of the
application.  Using these wouldn't make eg. table support suddenly
magically appear, but seem necessary for seriously implementing some
newer HTML features.  Also the whole stucture of how requests are handled
has drastically changed etc.

But you probably should have asked somebody else...

> Then I would like to know what libwww5 will be implementing
> above what libwww-FM is presently doing.

I hope Fote gave a better answer to that one than I could.

But he's seen it all before, since libwww3 an upgrade to a new library
version was always around the corner, somebody was working on it, it
would happen real-soon-now...  (I did some archive browsing that left
me with that impression.)  So let me make clear that I have not taken
on the task of doing all that.  I would be happy to get some of the
new network modules (actually just HTTP would be fine...) to work reasonably
well with Lynx, without making all the modifications and redesigning
throughout the current code that would be necessary to truely make use
of the new features.  That is much less than previous hopes, but may have
a better chance of success.

Now back to figuring out where those SIGSEGVs in HTAnchor_delete are
coming from...

   Klaus

;
; To UNSUBSCRIBE:  Send a mail message to address@hidden
;                  with "unsubscribe lynx-dev" (without the
;                  quotation marks) on a line by itself.
;