Re: [gnugo-devel] GNU Go 3.1.32 - VC inconsistencies

[Gunnar Farneback]

Trevor wrote:
> ./regress.sh . trevora.tst
> 140 unexpected FAIL: Correct 'G4', got 'D4'
> ./regress.sh . lazarus.tst
> 6 unexpected FAIL: Correct 'H3', got 'H2'
> ./regress.sh . nngs.tst
> 770 unexpected FAIL: Correct '!K2', got 'K2'
> ./regress.sh . trevorc.tst
> 350 unexpected FAIL: Correct 'E3', got 'J4'
> ./regress.sh . global.tst
> 14 unexpected FAIL: Correct 'S9', got 'E1'

Dan wrote:
> Interesting that they're all failures.

On Cray I have so far an unexpected pass for nngs1:45. I'll try to
track that one down. It would be good if someone could take a shot on
VC. The requirements are to have access to both VC and some other
platform without the unexpected failures, plus some time for detective
work. It helps to have some knowledge of the GNU Go internals but with
a little luck and consultation from the list it may be possible to
reach results anyway. The methodology is as follows:

First run the test case standalone with a certain amount of debug and
trace output. E.g. for trevora:140 this would mean

gnugo --quiet -a -w -t -d0x101800 -l games/trevor/auto/a005.sgf -L 12

Do this for both platforms and save the debug output to files. Compare
them to see where they differ. The most likely result (my guess) is
that some owl_attack or owl_defend trace differs. In that case, run

gnugo --quiet -a -w -t -d0xa -l games/trevor/auto/a005.sgf -L 12 
--decide-dragon E5 -o vars.sgf

where E5 should be replaced by the dragon for which there was a
deviation. Again do this for both platforms and compare the outputs,
searching for the first deviation (after that they'll probably diverge
wildly). Also compare the generated variation trees in vars.sgf. With
this information it may be possible to identify some pattern which
matches only on one platform, probably because there is a constraint
with differing results. This would then need closer examination, but
it's meaningless to discuss how to proceed until we reach there.

The second most likely result of the first level comparison is that
there is no difference. This means that the failure (or possibly the
non-failure on the other platform) is dependent of the preceding test
cases. The next thing to do in this case is to try to trim the test
file as much as possible until a minimum of test cases remain which
give a different result from only the test case in isolation. Then one
would run with trace and debug output as above for the minimum test
set and the test case alone, and compare the outputs as before. For
the minimum test set one of course only uses the output coming from
the diverging test case.

Generally speaking it's useful to try with various debug flags (listed
in gnugo.h as DEBUG_* constants) and with the various --decide-*
options if you have some specific suspicion. If you get very close to
the deviation it's probably a good idea to use a debugger.

When you don't get any further, report your results here and we may be
able to provide further guidance or possibly even see the problem.

/Gunnar