Re: LYNX-DEV hodge-podge of topics (was Lynx updates)

[Klaus Weide]

On Mon, 11 Nov 1996, Foteos Macrides wrote:

>       I am still disappointed that the unicode handling you volunteered,
> and were working on, was left as a hodge-podge of mods to an old FM code
> set instead of a clean, up to date, distributable supplement to the v2.6
> release, when I announced, in early Sept., that I wasn't starting another
> FM code set, and that people should bring their patches and supplements
> up to date with the v2.6 release.

Believe me, I am not happy about the state of that.  But if you were 
expecting from me a "clean, up to date, distributable supplement", done
by me alone, then you were expecting to much.

What I was trying to offer - maybe I didn't make it clear enough then -
was to provide some functions (or methods in another form) to do the
translations between different character sets.  I did not aspire to
provide a full solution on my own.  There are some questions about
how such translations should be used, and how they would fit into the 
Lynx code, that I didn't want to answer alone.  I started bringing
some of them up on the list (like: is the SGML.c function the best
place to do this, should this be done in a separate stream object),
but came away with the impression that either you were not interested
in that level of discussion, or the list just plain doesn't work for
that kind of questions.  So I stopped trying to figure out these basic 
questions in a collaborative manner on the list, and instead just
continued hacking my own code version.

I would have appreciated some indications whether I was on the right 
track with my ideas, what I got were some expressions of general
interest; but nothing enough to go on, to make decisions like "where
should these functions be called" for integration into the distributable
Lynx code in a clean way.  If you hoped I would figure it all out
myself, well it didn't work out that way..  that's why my personal
hacks remained just that.  I don't want to become another contributor
of additions that seem great just now because they add some needed
function, but become a burden for further development later because
they don't really fit in with the overall architecture.

Let me restate my offer:  I provide a set of functions (or other means)
to translate between charsets (including ways to load and store those
translations).  YOU tell me what interface you want, and put calls to
those functions in all the necessary places where translations have
to be done; or YOU are willing to tell me, slowly enough, where exactly
I have to look to do this myself (which means I have to know where a
byte is supposed to express _which_ charset); or WE enter into a 
discussion to figure these things out together. 

Now, the capitalized "YOU" in the last paragraph doesn't have to be
"Foteos Macrides, as project leader".  It could be FM, as an expert
on the Lynx code in the best position to know and discuss.  It could
also be "you Lynx developers and hackers".  It's up to YOU to answer. :)

Those translation functions are basically already there - my hodge-podge 
version is proof that they can work.  The current charset handling in
Lynx is sufficiently complex, scattered over different files, full of
exceptions, and confusing to me that, sorry, I cannot do it on my own
(within a reasonable amount of effort and time).

>                                     MSIE has begun supporting unicode,
> Netscape surely will follow, and this will become a glaring inadequacy
> of Lynx in due time.

We don't have to accept that for a fact just yet.
(Except that Lynx will always be limited by what characters terminals
and emulators provide.)

In fact, if WE really want it, we could probably have a Lynx patch
version with limited but usable, and clean (as far as the major Lynx
code) Unicode support, in a week.  Really.  
We would have to define what we mean with limited support, maybe just
being able to display pages with charset=UTF-8 (there are some) by
translating to Latin-1.

In case YOU are still interested (and choose not to just put the
function calls in), here are some concrete questions 
and ideas (just as a beginning):

Is it reasonable oberhead to call a translation function for each
(maybe just non-ASCII) character?
Should there be a general-purpose charset translation HTStream object,
which could be plugged into the stream pipe before SGML.c?
(The current libwww5 doesn't deal with it as far as I have seen, but
does something similar as the StreamStack for translating between
Content-Encodings.  That could be extended to be applied to charsets.)
Is it possible, and realistic, to have *one* internal representation
(which could be UTF-8) to pass data between SGML->HTML->HText, instead
of the current model?
What _is_ the current model?  Normal text gets translated (if applicable)
in SGML_character() from Latin 1 to display charset, but what about 
ALT text, other attibute text, and where may these translations have
to be reversed?

>       I don't know what you are doing with the v5 wwwlib, but I don't
> see signs that it might amount to more than the unicode handling w.r.t.
> generation of code for the general distribution.

I don't know what kind of signs you are expecting (some droppings I left
in WFBR's server logs are probably not enough :) ).  Tell me (this is
to the capitalized YOU again) you want to look at what I have and I'll
make it available.  Tell me YOU want to look at it to make some
concrete suggestions, and go over parts of it with me, and I'll put
it on a Web page _immediately_ (well after doing whatever is necessary
to avoid disk quota problems..).  The chances that one day I'll pull
a Lynx2.7 libwww5 mega-patch out of my hat, ready for general consumption,
are ehm slim.

There's a recurring pattern here: I find some thing interesting to do,
start bouncing the idea off the list, there is a general level of
interest.  Then I start hacking some code, but there isn't anybody
to talk to about it, and there is the dilemma that patches-that-dont-
work-yet won't be looked at, while patches-ready-for-distribution-in-
a-clean-way are impossible to make in one step (for me).  So any
ideas how I can solve this (maybe just my personal) problem? 

[ SNIP ] 
>       More importantly, in all this time since Lynx was GPLed to be
> supported by the Lynx User Community, we've gotten good support for
> information, such as the Lynx Enhanced Pages, the Lynx-me services,
> etc., but we haven't developed into an effective group for actual
> code development.  That's in part a matter of more people needing to
> acquire a real understanding of how the Lynx code works, but it also
> requires a better meshing of "personalities" than we have, in some
> cases, right now.

I don't know whether this mail helps or not.  

Personally, I would probably need some more exchange on the detail level 
of function calls, rather than the prevailing "here are some patches,
they should work" approach, to contribute better.

>       There's also the problem that HTML standarization has fallen
> apart, and a client like Lynx is left in the position of now always
> "reverse engineering" for what the "major vendors" have actually done,
> rather than being guided by truly open, publicly discussed drafts and
> RFCs.  

I think we can agree to agree here.

> For someone who works on Lynx purely as a hobby, that takes a
> lot of the fun out of it.

Hope you still have some left, really.

   Klaus






;
; To UNSUBSCRIBE:  Send a mail message to address@hidden
;                  with "unsubscribe lynx-dev" (without the
;                  quotation marks) on a line by itself.
;