Re: [gnugo-devel] Go as SPEC benchmark?

[Gunnar Farneback]

(This reply is only to the list, for internal discussion.)

Jim Koehler wrote:
>     There are many benefits to having a code used as a SPEC benchmark.
> Perhaps the biggest one is publicity and "mindshare".  The particular
> application chosen to represent an application area gets tremendous
> visibility.  In the course of benchmark development the code gets ported
> to many platforms.  SPEC member companies can provide valuable help in
> that area.  Another benefit is that compiler developers have great
> incentive to compile that code well, and many  of the most commercially
> significant compilers will reflect that within a few years.  There are
> also modest financial rewards.  More details can be found at
> http://www.spec.org/osg/cpu/CPU2004/search_program.html .

I'm positive. Our portability is already fairly good, I think, but
publicity and having the compiler writers optimize for our code
(instead of the other way round) sounds great. :-)

Excerpts from http://www.spec.org/osg/cpu/CPU2004/search_program.html:
> Criteria SPEC considers important for the next CPU benchmark suite:
> 
> * the program can be made compute bound

No problem.

> * the program can be made portable across different hardware
>   architectures and operating systems 

This is mostly done already. As long as they don't expect portability
to systems with int less than 32 bits there shouldn't be a problem.

> * how close the program is to the state of the art for the given field 

There are a couple of stronger programs around, but the difference
isn't very large and with respect to involved programming techniques I
would guess we are no further from state of the art than other
programs.

> * the proposed application source code meeting the following requirements:
>       + the exit code for the program must be 0 if it ran
>         successfully, and a non-zero otherwise.

Already done, I hope.

>       + Disabling all platform dependent code (e.g, gcc-specific
>         extensions and GNU-specific header files).

That should only be the trace macros and is trivial to do.

>       + main() in C/C++ programs taking one of the following forms:
>             # int main(void);
>             # int main(int argc, char * argv[]);

Already done.

> * a proposed reference input to be used as the official benchmark
>   workload with a verifiable output file(s) 

Some GTP script should do the trick. The entire test suite is probably
too large.

> * that the SPEC CPU Subcommittee will be able to observe the benchmark
>   run to completion, with basic compiler optimization enabled (e.g. -
>   "-O"), on: 
>       + A 64-bit big endian UNIX system
>       + A little endian Win2000 or Win/XP system.

I see no problems here.

> * that the SPEC CPU subcommittee, on those same systems with basic
>   compiler optimization enabled (e.g. - "-O"), will be able to observe
>   that: 
>       + Over 95% of the execution time is spent in the submitted code
>       + Over 95% of the time is compute bound
>       + The proposed workload takes no less than 600 seconds on a
>         machine with a SPEC CPU2000 baseline metric of ~700 for
>         integer codes and ~900 for floating point code 

Here neither.

> *  Demonstrate to SPEC's satisfaction that the benchmark runs on
>    additional systems, including (but not limited to) a little endian
>    UNIX system.

As commented earlier, I expect no complications as long we can only
have int 32 bits or larger.

> Submitter hereby grants Sponsor a perpetual, non-exclusive license to
> copy, modify, display, and sub-license the entry without geographical
> limitations or further compensation to Submitter of any kind, provided
> that Sponser's sole use of the entry will be for the purposes of
> inclusion of the entry in Sponser's CPU benchmark suite.

This is actually potentially problematic. The requirements don't
necessarily sound compatible with GPL, but on the other hand they may
be flexible enough to accept GPL instead.

/Gunnar