Re: A very old bug reported on RGB

[playbg-rgb]

Very strange. I just tested again with gnubg-1.06.002-dev-20220908
Same thing, running money session of 10 games, first with GUI then
with CLI (after verifying player settings are the same as GUI).

============================================
gnubg random
Chequer Play Statistics:
Total moves 115 119
Unforced moves 95 95
Unmarked moves 115 119
Moves marked doubtful 0 0
Moves marked bad 0 0
Moves marked very bad 0 0
Error total EMG (Points) -0.000 ( -0.000) -0.000 ( -0.000)
Error rate mEMG (Points) -0.0 ( -0.000) -0.0 ( -0.000)
Chequerplay rating Supernatural Supernatural

Luck Statistics:
Rolls marked very lucky 0 0
Rolls marked lucky 0 0
Rolls unmarked 0 0
Rolls marked unlucky 0 0
Rolls marked very unlucky 0 0
Luck total EMG (Points) +0.000 ( +0.000) +0.000 ( +0.000)
Luck rate mEMG (Points) +0.0 ( +0.000) +0.0 ( +0.000)
Luck rating None None

Cube Statistics:
Total cube decisions 89 37
Close or actual cube decisions 23 15
Doubles 9 4
Takes 4 1
Passes 0 8
Missed doubles below CP (EMG (Points)) 0 0
Missed doubles above CP (EMG (Points)) 0 0
Wrong doubles below DP (EMG (Points)) 2 (-1.905 ( -8.978)) 3 (-2.293 ( -6.419))
Wrong doubles above TG (EMG (Points)) 5 (-1.901 ( -1.901)) 0
Wrong takes (EMG (Points)) 0 0
Wrong passes (EMG (Points)) 0 1 (-2.341 (-74.906))
Error total EMG (Points) -3.807 (-10.879) -4.634 (-81.325)
Error rate mEMG (Points) -165.5 ( -0.473) -309.0 ( -5.422)
Cube decision rating Awful! Awful!

Overall Statistics:
Error total EMG (Points) -3.807 (-10.879) -4.634 (-81.325)
Error rate mEMG (Points) -32.3 ( -0.092) -42.1 ( -0.739)
Snowie error rate -16.3 ( +0.000) -19.8 ( +0.000)
Overall rating Beginner Awful!
Actual result +68.000 -68.000
Luck adjusted result +68.000 -68.000
Advantage (actual) in ppg +6.182 -6.182
95% confidence interval (ppg) [62.050,73.950] [-73.950,-62.050]
Advantage (luck adjusted) in ppg +6.182 -6.182
95% confidence interval (ppg) [62.050,73.950] [-73.950,-62.050]
============================================
(No game) show player
Player 0:
Name: gnubg
Type: gnubg:
Checker play:
Neural net evaluation:
3-ply evaluation.
Using pruning neural nets. Cubeful evaluations.
Noiseless evaluations.
Move filters:
Move filter for 1 ply:
keep the first 0 0-ply moves and up to 16 more moves within equity 0.32
Move filter for 2 ply:
keep the first 0 0-ply moves and up to 16 more moves within equity 0.32
Skip pruning for 1-ply moves.
Move filter for 3 ply:
keep the first 0 0-ply moves and up to 16 more moves within equity 0.32
Skip pruning for 1-ply moves.
keep the first 0 2-ply moves and up to 4 more moves within equity 0.08
Move filter for 4 ply:
keep the first 0 0-ply moves and up to 16 more moves within equity 0.32
Skip pruning for 1-ply moves.
keep the first 0 2-ply moves and up to 4 more moves within equity 0.08
Skip pruning for 3-ply moves.

Cube decisions:
Neural net evaluation:
3-ply evaluation.
Using pruning neural nets. Cubeful evaluations.
Noiseless evaluations.
Player 1:
Name: random
Type: gnubg:
Checker play:
Neural net evaluation:
0-ply evaluation.
Not using pruning neural nets. Cubeful evaluations.
Noise standard deviation 0.060 (deterministic noise).

Move filters:
Move filter for 1 ply:
keep the first 0 0-ply moves and up to 8 more moves within equity 0.16
Move filter for 2 ply:
keep the first 0 0-ply moves and up to 8 more moves within equity 0.16
Skip pruning for 1-ply moves.
Move filter for 3 ply:
keep the first 0 0-ply moves and up to 8 more moves within equity 0.16
Skip pruning for 1-ply moves.
keep the first 0 2-ply moves and up to 2 more moves within equity 0.04
Move filter for 4 ply:
keep the first 0 0-ply moves and up to 8 more moves within equity 0.16
Skip pruning for 1-ply moves.
keep the first 0 2-ply moves and up to 2 more moves within equity 0.04
Skip pruning for 3-ply moves.

Cube decisions:
Neural net evaluation:
0-ply evaluation.
Not using pruning neural nets. Cubeful evaluations.
Noise standard deviation 1.000 (pseudo-random noise).
============================================
(gnubg) show statistics session
Player gnubg random

Checkerplay statistics

Total moves 150 152
Unforced moves 124 123
Unmarked moves 150 152
Moves marked doubtful 0 0
Moves marked bad 0 0
Moves marked very bad 0 0
Error total EMG (Points) -0.000 ( -0.000) -0.000 ( -0.000)
Error rate mEMG (Points) -0.0 ( -0.000) -0.0 ( -0.000)
Chequerplay rating Supernatural Supernatural


Luck statistics

Rolls marked very lucky 0 0
Rolls marked lucky 0 0
Rolls unmarked 0 0
Rolls marked unlucky 0 0
Rolls marked very unlucky 0 0
Luck total EMG (Points) +0.000 ( +0.000) +0.000 ( +0.000)
Luck rate mEMG (Points) +0.0 ( +0.000) +0.0 ( +0.000)
Luck rating None None


Cube statistics

Total cube decisions 104 58
Close or actual cube decisions 33 22
Doubles 9 8
Takes 6 3
Passes 0 8
Missed doubles below CP (EMG (Points)) 0 0
Missed doubles above CP (EMG (Points)) 0 0
Wrong doubles below DP (EMG (Points)) 3 (-3.121 (-15.281)) 7 (-7.038 (-43.629))
Wrong doubles above TG (EMG (Points)) 4 (-0.859 ( -4.244)) 0
Wrong takes (EMG (Points)) 0 0
Wrong passes (EMG (Points)) 0 2 (-4.409 (-13.313))
Error total EMG (Points) -3.980 (-19.525) -11.446 (-56.942)
Error rate mEMG (Points) -120.6 ( -0.592) -520.3 ( -2.588)
Cube decision rating Awful! Awful!


Overall statistics

Error total EMG (Points) -3.980 (-19.525) -11.446 (-56.942)
Error rate mEMG (Points) -25.4 ( -0.124) -78.9 ( -0.393)
Snowie error rate -13.2 ( +0.000) -37.9 ( +0.000)
Overall rating Casual player Awful!
Actual result +131.000 -131.000
Luck adjusted result +131.000 -131.000
Advantage (actual) in ppg +11.909 -11.909
95% confidence interval (ppg) [114.307,147.693] [-147.693,-114.307]
Advantage (luck adjusted) in ppg +11.909 -11.909
95% confidence interval (ppg) [114.307,147.693] [-147.693,-114.307]
============================================

MK

On Sunday, September 18, 2022, 01:45:42 PM MDT, Philippe Michel <philippe.michel7@free.fr> wrote:

On Wed, Sep 14, 2022 at 08:22:00PM +0000, playbg-rgb@yahoo.com wrote:
> https://groups.google.com/g/rec.games.backgammon/c/kJM9O57Lu8Y/m/ZF7bkiQgBQAJ
>
> On September 12, 2022 at 9:58:31 AM UTC+2, MK wrote:
>
> > I set one player to Grandmaster checker and cube, one player
> > to Grandmaster checker and cube level to below beginner with
> > maximum noise. After letting it run for a dozen games, I checked
> > the stats to see how things were going. What did I see..? This..!
> > ===================================================
> > ....
> > Error rate mEMG (Points) -132.0 ( -0.318) -141.3 ( -0.186)
> > Cube decision rating Awful! Awful!
> > ===================================================
>
> Thinking that this bug may not be in earlier versions, I went
> right to the oldest version I have: 0.15 from (15 years ago).
> And the bug has been there since then. What a shame... :(
> Not for the bug having never been fixed for that long but for
> nobody having discovered it since that time! Apparently I'm
> the only one to question the things that I question, look for
> answers and find bugs instead.


I don't know how exactly MK set up the play levels, but I cannot
reproduce this kind of result. I get the expected result, with the
player using 3 ply for cube decisions playing "perfectly":

Cube Statistics:
...
Wrong takes (EMG (Points))              1 (-0.014 ( -0.112))  0               
Wrong passes (EMG (Points))            0                    0
Error total EMG (Points)                  -0.014 ( -0.112)      -0.618 ( -0.618)
Error rate mEMG (Points)                  -2.0  ( -0.016)    -15.8  ( -0.016)
Cube decision rating                  Supernatural        Intermediate       

The -0.014 is apparently due to the "error" being a take in a double /
beaver / raccoon / beaver / take sequence instead of beavering
forever. If I play a match instead of a money session the cube error
is exactly 0.

I started gnubg on a linux machine from a terminal, so I get the
sequence of commands corresponding to the GUI choices:

(No game) set player 0 gnubg
(No game) set player 0 chequer evaluation plies 3
(No game) set player 0 movefilter 1 0 0 16 0.320
(No game) set player 0 movefilter 2 0 0 16 0.320
(No game) set player 0 movefilter 3 0 0 16 0.320
(No game) set player 0 movefilter 3 2 0 4 0.080
(No game) set player 0 movefilter 4 0 0 16 0.320
(No game) set player 0 movefilter 4 2 0 4 0.080
(No game) set player 0 cube evaluation plies 3
(No game) set player 1 gnubg
(No game) set player 1 chequer evaluation plies 3
(No game) set player 1 movefilter 1 0 0 16 0.320
(No game) set player 1 movefilter 2 0 0 16 0.320
(No game) set player 1 movefilter 3 0 0 16 0.320
(No game) set player 1 movefilter 3 2 0 4 0.080
(No game) set player 1 movefilter 4 0 0 16 0.320
(No game) set player 1 movefilter 4 2 0 4 0.080
(No game) set player 1 cube evaluation plies 0
(No game) set player 1 cube evaluation prune off
(No game) set player 1 cube evaluation noise 1.000

As far as I can tell the GUI sets the parameters correctly and the
evaluation function uses them as expected.