Re: lynx-dev on caching (long)

[Leonid Pauzner]

>
> What do you think?
>

Well, thank you Klaus for detailed (nice) responce.
I will try to reply you later, not now, should think on it.

Just a more practical thought.

How does display partial logic appears in lynx?
One boy from Check found out that repaint done by HText_pageDisplay()
and the appropriate place to call it from HTFormat.c/HTCopy() while
reading stream and LYUtils.c/HTCheckForInterrupt() to enable scrolling.
He made an alpha patch and disconnected from lynx-dev.
Of cause, we got a lot of bugs and and side effects.
Now I jump into and fix them by introducing several flags:
including two linecounters, durty workaround for "forms input" problems,
and a synchronization semaphore (!),
we got it in 5-10 sequential attempts...
What I want to say: it wasn't supposed to do so by lynx original designers
but the internal structure permits this (except few more recent bugs -
we find a workarond for them).

Now with cacheing. If we create a library from the beginning we do so and so.
It may be a good idea to discuss its future language but the implementation
is a thremendous long-time job, definitely not for me new at programming.
We are at other position: we have a "working" code stable for years
with its own "language". I think on kludging an internal cache
without changing lynx logic. So I think on a high level:

LYAddVisitedLink() called just before repainting the rendered document,
it obviously transferred from network OK. So we can add it to cached sources
list (there is a check for repeated documents in Visited Links already,
need a minore change to exclude #fragments and file://localhost).

>From the other side, HTLoad() calls get_physical() to access network layer
with proxy and origin (L3 and O on your diagram). So we can add
try_internal_cache() which will call for get_physical()
if no valid cached source found. Also a kludge inside HTLoadHTTP
for IF-Modified-Since request.
Yes, this is not a linear transparent model.
And definitely not the best for CPU/memory usage either
(we could do several manipulation just inside mainloop() instead,
but we will get many special cases in getfile(), here and there,
probably will look not so clear unless we rewrite the code significantly).

The main subject is validation of a cached source.
There are two cases:
1) HTTP request. Calculate expiration status and load from cache if valid,
otherwise (or we are too lazy) do as much as sending conditional GET request
to L3 or O. (no more).
2) history. We need a document for '*', '['... probably few other
cases like <- key or #fragment that point out to the document inside
the history stack (more cases discussible). Here the documents assumed valid.

Originally, my system running Squid and Lynx can access it nicely
but since the case 2) differ from 1) - few documents with "no-cache" pragma
appears reloading when I want history. That is a lynx fault. So
I start thinking on it and found that 1) and 2) can be handled the same time.



>
> In principle, and IMO, caching should be like this:
>
>      U             L1            L2            L3            O
>    -------       -------       -------       -------       -------
>    |     | ----> |     | ----> |     | ----> |     | ----> |     |
>    |     |       |     |       |     |       |     |       |     |
>    -------       -------       -------       -------       -------
>     user          cache         cache         cache         origin
>