Re: [Grammatica-users] benchmarking Grammatica performance

[Alex Rousskov]

On Wed, 14 Jan 2004, Per Cederberg wrote:

>> http://www.hapy.org/benchmarking/

> Interesting! A brief look at the Hapy and Spirit web sites seem to
> indicate that these are both tools for generating C++ parsers. As
> Grammatica does not (yet) support C++, I guess one has to expect
> lower performance than from the other two.

If both Hapy and Spirit performed similarly, I would agree.  However,
the significant (albeit preliminary) difference in their performance
indicates that many trade-offs can be solved differently and
programming language alone does not determine performance. I expect to
see meaningful difference in same-language parser performance
depending on use cases and environments.

Also, let's assume for a second that Grammatica is slower than the
fastest C++ parser generator. Depending on user needs, the difference
in performance can be marginal compared to other factors where Java
wins. Now, it would still be important to know what the performance
difference is, and that is what we are trying to accomplish here.

In other words, we want to give our users meaningful results. We are
not after selecting winners or losers (though individual projects are
free to od that based on our data).

BTW, we are expecting Spart results soon. Spart is a C# parser, so
there will be some language diversity. Adding Java or another C#
parser to the mix would be highly desirable though.

> Last I made my own totally non-scientific benchmarks, the Java
> version of Grammatica was faster than the C# version (running under
> Mono on Linux). Also, the Windows version of Java is supposedly much
> faster than the Linux version (at least before the new 2.6.x
> kernel), so that might also be valuable to be aware of.

Yes, Spirit folks also suggested that their parser performance will
depend a lot on the C++ compiler and other external factors. We will
publish results from different environments if/as developers submit
them. Again, the primary goal is to provide meaningful information to
our users.

> And then there are all these subtle things to consider with
> interpreted languages (like Java and C#) - when to start counting
> the parsing time and whether or not the language runtime memory
> should be counted? Grammatica being a run-time parser generator
> (i.e.  creating the parser when the input is to be parsed) might
> also mess a bit with the timing. Not that it would matter much for
> large inputs, but it is nonetheless valuable to know.

Good point. We currently start measuring time when the parse() method
or equivalent is called with pre-loaded input. We report increase in
memory usage once that method returns. I agree that it may be
impossible to find a one-size-fits-all procedure here, but we should
try to keep results comparable to the extent possible.

> Anyway, I don't think it would be much trouble to create the XML
> parsers with Grammatica for the benchmark. A parse tree is generated
> by default, so you shouldn't have to write much code yourself. Or
> maybe there is somebody else on the list that would like to help?

My environment lacks both Windows and Java at the moment (not to
mention spare cycles). I am also worried about screwing things up for
Grammatica, performance-wise. I will wait for somebody to volunteer to
prepare and submit the results. I do agree that it should not take
more than a few days (of that) for somebody who knows Grammatica well.

> It would be great to have some nice performance comparisons...
> Kind of encourages performance optimization (and I've hardly even
> started with Grammatica). Actually, while you're at it, you might
> consider benchmarking Bison/Flex (as a baseline) and ANTLR (the main
> competitor to Grammatica I think) as well.

Yes, having LALR parsers in the mix would be terrific. I am sure they
will win (at least short term), but I am curious what the difference
in performance is going to be (see my Java performance comments
above). If anybody on this list can help, please let me know!

Thank you,

Alex.

-- 
Protocol performance, functionality, and reliability testing.
Tools, services, and know-how.
http://www.measurement-factory.com/