Re: lynx-dev PATCH [dev21]: source caching

[Leonid Pauzner]

11-Apr-99 12:46 Scott Bigham wrote:
> On Sun, 11 Apr 1999, Leonid Pauzner wrote:

>> One (minor) problem: keeping a temporarily files for sources cache
>> we need to be sure to clean these files on exit. Currently it implemented
>> in HText_free() but atexit(free_all_texts) was recently ignored
>> without LY_FIND_LEAKS.

> And I had LY_FIND_LEAKS on for testing purposes; no wonder I didn't see
> it.  See, I knew I'd miss something.

> (examines code) Wait a minute.  The declaration and definition of
> free_all_texts() are wrapped in '#ifdef LY_FIND_LEAKS', but the
> atexit(free_all_texts) call in HText_new() isn't.  How can that even
> compile without LY_FIND_LEAKS #define'd?  Well, at any rate, the quick

That was the very recent changes that #define atexit to nothing
to decrease shutdown time (and free_all_texts() is the main contributor),
hope will be corrected for dev22 since several problems were reported.

> fix would seem to be to change those wrappers to
> '#if defined(LY_FIND_LEAKS) || defined(SOURCE_CACHE)'.

>> Probably these files will be deleted by clean_files() anyway,

> No, cleanup_files() only deletes files listed in the ly_temp list.  And
> I hesitate to use that mechanism, because then I'd have to go back and
> remove those files from ly_temp when they get uncached.
I do not understand this.

>> ha, don't forget to disable the feature in NONINTERACTIVE mode - it
>> will not clean temp files currently (if should not create temp files,
>> so a nice check that everything OK).

> Hmm, that could be done in the 'if (dump_output_immediately)' block in
> main(), yes?
sure.

>> Also we increase disk activity which may be not good for users
>> under disk quota or leaving too much junk if lynx crash unconditionally.

> Well, the low-disk-quota user could always put SOURCE_CACHE:FALSE in her
> lynx.cfg (or we could have it off by default and make the user turn it
> on in lynx.cfg; I only had it on by default for testing purposes), or
> compile it out if she's building her own.  As for the crash, that will
> be the same as with temporary files, yes?

I prefer to have a choice between caching sources in temp files or keeping
them in RAM by managing a dynamic list of Cached Stream's or so. The latter
may be the choice for those with low disk-quota but having enough memory -
a standard condition when you run Lynx at ISP machine (not your personal one).
The first choice may also slowdown the system (a wild guess).

Also, we can expand this little further and implement HTTP/1.1 cache so
HTLoad will send conditional GET requests when we already have a source
locally.  Anyway this may be a next good step.

Not sure whether the 'd'ownload command will use the cached sources currently.
Lots of minor fixes will be done until the things stabilized.


>                                               -sbigham