Re: lynx-dev local directory listing - TOO SLOW

[Bela Lubkin]

Leonid Pauzner wrote:

> >> Lynx's rendering of local directory listing apperently too slow,
> >> directory for 150 entries fetched the same time as ~200K long text
> >> (this is especially strange with my slow DOS mashine - there is
> >> no permissions to look so it is a single small file).
> 
> > It does a 'stat()' call for each file - which is a little slow (but
> > ls -l would do the same).  I don't have a slow machine to test it on
> > (even my old machine is 60Mhz - but I'll do some testing on that to
> > compare).
> 
> >> Why not to do "ls -l" or "dir" or so
> >> and than translate the output to internal HTML format?
> > I don't think it would be faster in terms of the underlying system calls.
> > But it should be possible to test/compare by using the code that interprets
> > directory listings from remote servers.
> 
> I made a test with our Linux machine at the /home directory of ~500 entries:
> lynx render this directory - 12 sec
> ls -l > tmpdir  - 3 sec
> 
> so there is a significant slow down with lynx
> (the CPU rather fast - lynx_users_guide.html of 150Kb renders << 1 sec).
> Probably "ls -l"/"ls -all" would be a good default for any autoconf'ed
> platform (I have no idea on VMS).
> 
> and here on DOS I have even a greater difference.
> 
> >> The code somethere in HTFile.c

Not sure who to attribute the other quotes to.  Anyway...

Looking at a system call log, on my OpenServer Release 5.0.4 system,
most of the system calls come as a result of getpwuid() and getgrgid()
calls in HTFile.c.  Commenting those out (so the other path gets used --
printing numeric UID and GID) greatly reduces the number of system
calls.  However, this doesn't quite double Lynx's speed, which still
leaves it lagging behind `ls`.

Testing on a few different directories, I get the following times (all
in seconds; averages of 4 runs after a priming run to get everything in
cache):

 #files   ls -a  ls -al  lynx(1)  lynx(2)
   1      0.03    0.03    0.07     0.07
  86      0.04    0.07    0.23     0.13
 449      0.05    0.24    1.27     0.69
 640      0.06    0.38    1.88     1.06

lynx(1) is 2.8.1dev.10, because that's what happens to be on my laptop
at home.  lynx(2) is the same, with the getpwuid(), getgrgid() calls
cancelled out.

`ls -a` appears to have about 0.03 seconds of overhead + .005 seconds
per 100 files.  `ls -al` has similar overhead + .05 seconds per 100
files.  Both Lynxes have 0.07 overhead; lynx(1) takes about .27 sec/100
files, and lynx(2) about .16 sec/100.  I ran Lynx as:

  HOME=/tmp lynx -dump -nolist -cfg null.cfg /name/of/dir >/dev/null

where null.cfg contains:

  PERSONAL_MAILCAP:.nosuchfile
  GLOBAL_MAILCAP:/local/lib/lynx/nosuchfile
  PERSONAL_EXTENSION_MAP:.nosuchfile
  GLOBAL_EXTENSION_MAP:/local/lib/lynx/nosuchfile

This eliminates as many distractions as possible from the system call
call traces & timings.

getpwuid() and getgrgid() are slow, on my system, because they open,
read, and close the appropriate file (/etc/passwd or /etc/group) every
time.  Maybe other libc implementations cache this information.  If not,
this might be a good candidate for a little cache inside HTLoadFile().
It doesn't need to be persistent across calls.

In fact, the following patch gives most of the benefit by caching only
the very most recent return from each function.  This is against old
source (2.8.1dev.10), probably won't patch current source due to
sprintf() and gettext() changes.

>Bela<

*** HTFile.c.orig       Sat May  2 16:39:12 1998
--- HTFile.c.new        Sat Jan  9 03:33:57 1999
***************
*** 166,167 ****
--- 166,171 ----
  #ifdef LONG_LIST
+ struct passwd op;
+ int p_lastid = -1;
+ struct group og;
+ int g_lastid = -1;
  PRIVATE void LYListFmtParse ARGS5(
***************
*** 315,318 ****
                case 'o':       /* owner */
!                       sprintf(fmt, "%%%ss", start);
!                       p = getpwuid(st.st_uid);
                        if (p) {
--- 319,328 ----
                case 'o':       /* owner */
!                       if (st.st_uid == p_lastid) p = &op;
!                       else {
!                               p = getpwuid(st.st_uid);
!                               if (p) {
!                                       op = *p;
!                                       p_lastid = st.st_uid;
!                               }
!                       }
                        if (p) {
***************
*** 328,330 ****
                case 'g':       /* group */
!                       g = getgrgid(st.st_gid);
                        if (g) {
--- 338,347 ----
                case 'g':       /* group */
!                       if (st.st_gid == g_lastid) g = &og;
!                       else {
!                               g = getgrgid(st.st_gid);
!                               if (g) {
!                                       og = *g;
!                                       g_lastid = st.st_gid;
!                               }
!                       }
                        if (g) {