RE: [Bug-gnubg] Snowie 4 vs. GNU 0.13

[Albert Silver]

First of all, many thanks for the clear explanation, and thanks for
doing the analysis in the first place.
> I've attached a file which has the following entries:

I saw no file BTW.

> 
> game number, actual result, gnubg luck, snowie luck, luck adjusted
result
> 
> For example, for game 100 (my example above):
> 
> 100 1 -.01415 -.49717 .51698
> 
> Some of the games could be very interesting to inspect carefully. For
> bots of similar strength we expect luck adjusted results around 50%.
> However, this is not always true in the 100 match sample you've sent
me:
> 
> Examples:
> 
> 8 0 .19603 .19853 .00250
> 39 1 .07856 .00087 .92231
> 
> Either gnubg's luck analysis is totally wrong or snowie (gnubg)
> played very bad in game 39 (game 8).

Yes, there are problems with the analysis of some extreme games, and I
presume this also makes the luck analysis just as dubious. First, I'd
like to point out that I've noticed that GNU's evaluation (not
necessarily the play) of backgames at 2-ply can be (*can be*, not *is*)
extremely dubious at times and the odd-ply such as 3-ply will be very
close to reality. I have no idea why this is so. There was that position
from Dupreli's rollout comparison table, and below is another. I'm
sharing this example from match #79 game 6 as it shows a really wild
game in which GNU was extremely critical of a lot of Snowie's moves.
Considering the evaluation, I think it's probably mistaken. 

Example (I laughed at this very extreme position BTW):

GNU Backgammon  Position ID: CogAtnvYdhtQAA
                 Match ID   : MIHxAGAAMAAA
 +12-11-10--9--8--7-------6--5--4--3--2--1-+     O: Snowie4
 | X        O  O  O |   | O  O  O  X  X  X |     6 points
 |          O  O  O |   | O  O  O  X  X  X |     Rolled 34
 |                O |   |                X |     
 |                  |   |                  |     
 |                  |   |                  |    
^|                  |BAR|                  |     7 point match (Cube: 1)
 |                  |   |                  |    
 |                  | X |                  |     
 |                  | X |                  |     
 |                  | X |                  |     
 |       X  O  O    | X |          X  X    |     6 points
 +13-14-15-16-17-18------19-20-21-22-23-24-+     X: gnubg

    1. Cubeful 2-ply    16/12* 7/4                   Eq.:  +0.657
        82.8%  54.9%  35.6% -  17.2%   0.0%   0.0%
        2-ply cubeful 100% speed [world class]
    2. Cubeful 2-ply    9/5 7/4                      Eq.:  +0.607 (
-0.049)
        80.4%  53.2%  34.3% -  19.6%   0.0%   0.0%
        2-ply cubeful 100% speed [world class]
    3. Cubeful 2-ply    8/4 7/4                      Eq.:  +0.607 (
-0.050)
        80.4%  54.3%  35.7% -  19.6%   0.0%   0.0%
        2-ply cubeful 100% speed [world class]
    4. Cubeful 2-ply    16/12*/9                     Eq.:  +0.585 (
-0.071)
        79.3%  50.7%  33.0% -  20.7%   0.0%   0.0%
        2-ply cubeful 100% speed [world class]
 * 13. Cubeful 2-ply    17/14 9/5                    Eq.:  +0.503 (
-0.154)
        75.2%  48.4%  31.0% -  24.8%   0.0%   0.0%
        2-ply cubeful 100% speed [world class]

This 2-ply evaluation is absurd needless to say. The 3-ply below is much
better (no idea about the move choices) though:

    1. Cubeful 3-ply    16/12*/9                     Eq.:  +0.104
        55.2%  35.7%  24.1% -  44.8%   0.0%   0.0%
        3-ply cubeful [grandmaster]
    2. Cubeful 3-ply    17/14 16/12*                 Eq.:  +0.096 (
-0.007)
        54.8%  36.4%  23.4% -  45.2%   0.1%   0.0%
        3-ply cubeful [grandmaster]
    3. Cubeful 3-ply    16/12* 7/4                   Eq.:  +0.075 (
-0.029)
        53.8%  34.0%  23.4% -  46.2%   0.0%   0.0%
        3-ply cubeful [grandmaster]
    4. Cubeful 3-ply    17/13 8/5                    Eq.:  +0.072 (
-0.032)
        53.6%  34.8%  23.4% -  46.4%   0.1%   0.0%
        3-ply cubeful [grandmaster]
    9. Cubeful 3-ply    17/14 9/5                    Eq.:  +0.048 (
-0.056)
        52.4%  33.5%  23.2% -  47.6%   0.0%   0.0%
        3-ply cubeful [grandmaster]

                                                Albert



> 
> Match 39 re-analysed:
> 
> 0-ply:   1 .07856 .00087 .92231
> 1-ply:   1 .1150 -.0151  .87449
> 2-ply:   painfully slow; I gave up
> 
> The result is changed by 5%, but we're still far from a luck adjusted
> result of 50%. I can't explain this...
> 
> Jørn