Re: [gnugo-devel] tumego benchmark

[Gunnar Farneback]

T. Yoshikawa wrote:
> Dear friends
> I have two discussions about tumego program.
> One is how to evaluate tumego programs.
> My proposal is:
> two point for correct answer(status and move),
> one point for semi-correct(correct move)
> zero point for answer unknown,
> -2 point for wrong answer.

I don't think it's possible to come up with a definite measure. The
relative value between answering wrong and not answering depends a lot
on how the information is used by other parts of move generation. In
some situations there could also be difference in the harm of
incorrectly saying that something is alive compared to incorrectly
saying that something is dead.

On the other hand I don't think it matters much in practice how these
things are measured.

> For a fixed book,

It's difficult to use books as sources unless the copyright holder gives
permission to distribute the problems electronically. Although the
copyright laws may vary between countries it's at least out of the
question to include them in GNU Go without an explicit permission.

> we assign some adequate time.
> For example, 1001 life-and-death ,600 seconds.
> We compete the total points in the given time interval. 
> Overlaid of time option for specific command line is not allowed.

I tried 1001 life-and-death on the current CVS version of GNU Go. The
results at default level 10 were 671 correct and 330 incorrect solutions
in 45 seconds. I'm happy with the speed but not with the results. :-)

> Another is how to define the battle field.
> My tumego program requires battle field in which
> attack-and-defese is limited. All stones out side
> of battle field are assumed as living.

This is a key issue when writing a life-and-death reader for general
use. Enclosed problems are much easier to handle than ones with
uncertain escape potentials. GNU Go certainly isn't great at
understanding the latter but at least makes a reasonable attempt in the
owl code.

> Owl of GUN is smart but is limited in reading.
> I want to include multiple-iterative-deepening
> in GNUGO.

Sounds interesting. If you want it to eventually be included with GNU Go
there are a few barriers of entry that you need to be aware of from the
start:

* It must be written in C.

* It must be effective, i.e. better than the owl code in at least some
  respect so that it's useful to us. (E.g. faster, more accurate, or
  simpler code.)

* We must be able to understand it so that it can be maintained in the
  future.

* You need to be willing to assign copyrights to the Free Software
  Foundation. Feel free to ask if you need more information on this.

/Gunnar