Re: flex and more namespace clashes

[Bruce Lilly]

W. L. Estes wrote:
> On Saturday, 08 March 2003,00:03 -0500, Bruce Lilly wrote:
>
>
> > If a prefix of m, c, or re is used, flex generates namespace
> > clashes with the standard library functions malloc, calloc,
> > or realloc.  The problem is "yyalloc", which is renamed to
> > malloc, calloc, or realloc.  If it were yy_alloc, there
> > wouldn't be a clash; better still if the standard malloc,
> > realloc, and free were used, the problem wouldn't arise.

Incidentally, at least std and str also won't work as prefixes because
of clashes with sdtio (stdin, stdout) and strerror; that's probably
unavoidable because yyin, yyout, and yyerror are the required
standard names.  Maybe worth a mention in the documentation for
prefixes.

> The point of using different names is so that users of flex can choose
> their own memory allocation functions. but yy_alloc is probably a good
> idea.

Yes, I understand the impetus. But a couple of additional points:
1. Some allocation packages provide file and line-number tracing,
   which is nullified by the current 2.5.x scheme (simply using a
   compatible call (i.e. w/o the additional yyscan_t arg.) wouldn't
   cause that problem).  Most replacement allocation packages (e.g.
   vmalloc) have a mechanism to work with code that uses standard
   malloc/calloc/realloc/free calls  -- and incidentally I have
   used vmalloc to find and report an allocation problem in flex
   2.5.19 (see details including a discussion of the difficilties
   caused by yyalloc etc. in my list message dated 14 Sep 2002).
2. Whatever is done should ideally be carefully planned. Since there
   may be various versions of flex in use, implementors of packages
   have to be able to deal with all of them.  I already have a
   considerable amount of makefile cruft which is necessary to deal
   with various flex issues (including anticipated switch to yy_alloc,
   etc.) -- it would be nice if the amount of cruft didn't become
   unmanageable. Here's what I have to deal with now:

# GACK!
# flex provides no way to get user code into the .c file before boilerplate 
includes
# GACK! GACK! GACK!
# flex 2.5.10 and later makes using malloc replacements difficult
# GACK! GACK! GACK! GACK! GACK! GACK!
# flex-generated header includes a lot of unnecessary cruft -- remove some bits
#       which unnecessarily lengthen compilation and/or cause errors
lex.$(PREFIX).c $(PREFIX)flex.h :       $(PREFIX)_lex.l $(FLEX) $(MAKEFILES)
        @-rm -f lex.$(PREFIX).c z.$(PREFIX).c z.h
        $(LEX) $(LFLAGS) $(PREFIX)_lex.l
        sed -e 's/#include <stdio.h>/#include "private.h"/' \
                -e 's/^#[       ]*define[        ]*yyalloc[     ]*.*alloc[      
]*$$//' \
                -e 's/^#[       ]*define[        ]*yyrealloc[   ]*.*realloc[    
]*$$//' \
                -e 's/^#[       ]*define[        ]*yyfree[      ]*.*free[       
]*$$//' \
                -e 's/yyalloc[  ]*(\([^;]*\)[   ]*YY_CALL_LAST_ARG[     
]*/malloc(\1/' \
                -e 's/yyalloc[  ]*(\([^;]*\)[   ]*,[    ]*NULL[         
]*/malloc(\1/' \
                -e 's/yyrealloc[        ]*(\([^;]*\)[   ]*YY_CALL_LAST_ARG[     
]*/realloc(\1/' \
                -e 's/yyrealloc[        ]*(\([^;]*\)[   ]*/realloc(\1/' \
                -e 's/\([^;]*size[ +2]*\)[      ]*YY_CALL_LAST_ARG[     ]*/\1/' 
\
                -e 's/yyfree[   ]*(\([^;]*\)[   ]*YY_CALL_LAST_ARG[     
]*/free(\1/' \
                -e 's/^#[       ]*define[        ]*yy_alloc[    ]*.*alloc[      
]*$$//' \
                -e 's/^#[       ]*define[        ]*yy_realloc[  ]*.*realloc[    
]*$$//' \
                -e 's/^#[       ]*define[        ]*yy_free[     ]*.*free[       
]*$$//' \
                -e 's/yy_alloc[         ]*(\([^;]*\)[   ]*YY_CALL_LAST_ARG[     
]*/malloc(\1/' \
                -e 's/yy_alloc[         ]*(\([^;]*\)[   ]*,[    ]*NULL[         
]*/malloc(\1/' \
                -e 's/yy_realloc[       ]*(\([^;]*\)[   ]*YY_CALL_LAST_ARG[     
]*/realloc(\1/' \
                -e 's/yy_realloc[       ]*(\([^;]*\)[   ]*/realloc(\1/' \
                -e 's/yy_free[  ]*(\([^;]*\)[   ]*YY_CALL_LAST_ARG[     
]*/free(\1/' \
                -e 's,/\*[      ]*fall[         ]*through[      
]*\*/,/*FALLTHROUGH*/,' \
                -e 's/static[ \t]*\(.*yy_push_state\)/\1/' \
                -e 's/static[ \t]*\(.*yy_pop_state\)/\1/' \
                lex.$(PREFIX).c > z.$(PREFIX).c
        mv -f z.$(PREFIX).c lex.$(PREFIX).c
        egrep -v 'yylval[^p]|<(string|errno|stdlib)\.h>' $(PREFIX)flex.h > z.h
        sed \
                -e 's/^#[       ]*define[        ]*yyalloc[     ]*.*alloc[      
]*$$//' \
                -e 's/^#[       ]*define[        ]*yyrealloc[   ]*.*realloc[    
]*$$//' \
                -e 's/^#[       ]*define[        ]*yyfree[      ]*.*free[       
]*$$//' \
                -e 's/yyalloc[  ]*(\([^;]*\)[   ]*YY_CALL_LAST_ARG[     
]*/malloc(\1/' \
                -e 's/yyalloc[  ]*(\([^;]*\)[   ]*,[    ]*NULL[         
]*/malloc(\1/' \
                -e 's/yyrealloc[        ]*(\([^;]*\)[   ]*YY_CALL_LAST_ARG[     
]*/realloc(\1/' \
                -e 's/yyrealloc[        ]*(\([^;]*\)[   ]*/realloc(\1/' \
                -e 's/\([^;]*size[ +2]*\)[      ]*YY_CALL_LAST_ARG[     ]*/\1/' 
\
                -e 's/yyfree[   ]*(\([^;]*\)[   ]*YY_CALL_LAST_ARG[     
]*/free(\1/' \
                -e 's/^#[       ]*define[        ]*yy_alloc[    ]*.*alloc[      
]*$$//' \
                -e 's/^#[       ]*define[        ]*yy_realloc[  ]*.*realloc[    
]*$$//' \
                -e 's/^#[       ]*define[        ]*yy_free[     ]*.*free[       
]*$$//' \
                -e 's/yy_alloc[         ]*(\([^;]*\)[   ]*YY_CALL_LAST_ARG[     
]*/malloc(\1/' \
                -e 's/yy_alloc[         ]*(\([^;]*\)[   ]*,[    ]*NULL[         
]*/malloc(\1/' \
                -e 's/yy_realloc[       ]*(\([^;]*\)[   ]*YY_CALL_LAST_ARG[     
]*/realloc(\1/' \
                -e 's/yy_realloc[       ]*(\([^;]*\)[   ]*/realloc(\1/' \
                -e 's/yy_free[  ]*(\([^;]*\)[   ]*YY_CALL_LAST_ARG[     
]*/free(\1/' \
                -e 's/static[ \t]*\(.*yy_push_state\)/extern \1/' \
                -e 's/static[ \t]*\(.*yy_pop_state\)/extern \1/' \
                z.h > $(PREFIX)flex.h


I suspect that there are some implementations of sed that will already choke
on such a long list of edit args, and if this keeps up, there's a good
chance of hitting a command-line arg limit ceiling as well.  Now if some
approach other than changing yyalloc etc. to yy_alloc etc. is used, I could
potentially remove or change 16 of those lines; in general, though, once I
need to add a workaround for some problem, it has to stay because somebody
else may be using the version of flex that requires the workaround.  Though
I've anticipated yy_alloc, it is still possible to do something better.