Re: [Grammatica-users] benchmarking Grammatica performance

[Per Cederberg]

On Tue, 2004-01-13 at 18:10, Alex Rousskov wrote:
> Hi,
> 
>       I am working on putting together a performance comparison page
> for parser generators. An alpha version with the overall methodology
> and a results teaser (mind the warning!) is available at
> 
>       http://www.hapy.org/benchmarking/

Interesting! A brief look at the Hapy and Spirit web sites seem to
indicate that these are both tools for generating C++ parsers. As
Grammatica does not (yet) support C++, I guess one has to expect
lower performance than from the other two.

Last I made my own totally non-scientific benchmarks, the Java 
version of Grammatica was faster than the C# version (running under
Mono on Linux). Also, the Windows version of Java is supposedly
much faster than the Linux version (at least before the new 2.6.x
kernel), so that might also be valuable to be aware of. And then
there are all these subtle things to consider with interpreted
languages (like Java and C#) - when to start counting the parsing
time and whether or not the language runtime memory should be 
counted? Grammatica being a run-time parser generator (i.e. 
creating the parser when the input is to be parsed) might also 
mess a bit with the timing. Not that it would matter much for 
large inputs, but it is nonetheless valuable to know.

> There are currently two parser generators represented: Spirit[1] and
> Hapy[2]. We expect Spart[3] results soon. Spirit folks suggested to
> add Grammatica into the mix. If any expert would like to help us with
> publishing Grammatica results, please let me know!

Well, I guess I'm an "expert" on Grammatica... ;-)  Unfortunately 
I won't be able to help you much right now, as I'm moving to 
Madrid (Spain) in just a few days. Actually, if I'm not responding 
to things here on the list the upcoming few weeks, it will be due 
to this.

Anyway, I don't think it would be much trouble to create the XML
parsers with Grammatica for the benchmark. A parse tree is 
generated by default, so you shouldn't have to write much code
yourself. Or maybe there is somebody else on the list that would 
like to help?

> General feedback and improvement suggestions are also welcome, of
> course. The above URL is not linked from Hapy web site yet to give us
> a chance to review preliminary results and fix any serious errors.

It would be great to have some nice performance comparisons... 
Kind of encourages performance optimization (and I've hardly even 
started with Grammatica). Actually, while you're at it, you might 
consider benchmarking Bison/Flex (as a baseline) and ANTLR (the 
main competitor to Grammatica I think) as well.

Cheers,

/Per

-- 
Per Cederberg, Software Consultant
http://www.percederberg.net/software