Re: lynx-dev hyphenation (was tech. question: translating strings)

[Vlad Harchev]

On Wed, 8 Sep 1999, Vlad Harchev wrote:

> On Mon, 6 Sep 1999 address@hidden wrote:
> 
> > In message <address@hidden>, 
> >            Vlad Harchev writes:
> > > 
> > >  As for 35 Mb of VSS - colorstyle changes takes 1Kb for each HTLine. There
> > > were 22000 lines in that file - so that's why so big memory usage. I want 
> > > it
> > > to be fixed.
> > 
> > "You have the source."
> > 
> > I'll look into it, since it's my fault in the first place.
> 
>  Currenly, HTStyleChange is defined in following way.
> 
> typedef struct _stylechange {
>         int     horizpos;       /* horizontal position of this change */
>         int     style;          /* which style to change to */
>         int     direction;      /* on or off */
>         int     previous;       /* previous style */
> } HTStyleChange;
> 
> It takes 16 bytes on x86. IMO nothing will be hurted if it was defined as:
> 
> typedef struct _stylechange {
>       unsigned int direction:2;
>         unsigned int horizpos:(sizeof(short)*CHAR_BITS -2);    /* horizontal
> position of this change */
>       unsigned short style;
> } HTStyleChange;
> 
> It will take 4 bytes on x86. If this is bit-fields are non-portable (or the
> size of the bitfield as constant expression is non-portable), then this should
> be defined as #ifdef __GCC__ etc.
> 
> Also I'd like to use memory chunks (a big pool of memory dedicated to
> allocating HTStyleChange in it. When no space in pool is left, another pool is
> allocated with malloc). So HTLine.stylechanges will be a pointer pointing
> inside that pool (when the line was assembled in split_line), rather than
> array of MAX_LINE HTStyleChange's.
> 
>  It seems that the average line contains 6 colorstyle changes. So,
> if such approach was used, the stylechanges would take 
> (6*4*22000 = 528 000) - compare to 22 000 000.
> 
>  Not using memory pools will just reduce the 22M to 22/4 = 5 500 000 (but this
> is IMO still not very good).

 I implemented all this. Let me repeat, the original lynx's VSS is 30Mb (21900
lines in the rendered representation).
Without using pools (but with HTStyleChange reorganization), VSS is 10Mb on
the same file. With pools, VSS is 5.5 Mb (the pool holds 4096 entries). Lynx
loads this file much faster (50Kb/s vs 33Kb/s) when memory allocation is
optimized (especially when the memory pools are used). It's now possible to
raise MAX_STYLES_ON_LINE to more reasonable value like 128 (since 64 it's too
few for psrcview of some documents). I wrote generic macros for working with
pools, so it will be easy to apply that approach to other things - I'd like to 
do this for HTLines (and the text containing in it) sometime later, so lynx 
will be much clever when using memory.

 Unfortunately, a lot of hyphenation-oriented changes are done to
my GridText.c, so it will be difficult for me to derive only 
memory-optimization changes from it. 
  This is another reason to integrate hyphenation :)
 
>[...] 

 Best regards,
  -Vlad