Re: [Bug-gnubg] Confused

[Ian Shaw]

Thanks Michael. That's very useful information. 

Comparing gnubg to XG we have:

Ply:    Gnubg   XGv2
2-ply:  20.95   16.32
3-ply:  16.78   12.03
4-ply:  12.54   12.03

Plies are as measured by gnubg. XG play = gnubg ply + 1.

It looks like the new nets have reduced the errors by 40% - 50%. Excellent.

However, I'm not sure this is comparing like with like. Your results report 
5420 decisions, whereas bkgm.com reports  4903 decisions for the 2010 study. 
Perhaps the 2012 study had more decisions, but the quantity is not reported. 

Whatever, this is a great improvement by the gnubg team. Was it Philippe who 
created the latest weights? 

I searched through the archives for a report of how this was achieved, but I 
can't find it right now. If Phillippe is reading this, I'd love to know what 
was done. Longer training? Different initial weights? The inputs and number of  
hidden nodes are unchanged, I believe.

"Whether the Depreli study translates to general play and positions is not 
known." I think it's reasonable to conclude that the applies at least for bot 
play, since that's how the position set was created. Have you any particular 
concerns, or just being scientifically cautious?


-- Ian


-----Original Message-----
From: address@hidden [mailto:address@hidden On Behalf Of Michael Petch
Sent: 12 June 2015 16:07
To: address@hidden
Subject: Re: [Bug-gnubg] Confused

On 2015-06-12 8:24 AM, Ian Shaw wrote:
> 
> You make a good point about Michael's benchmark. Are you repeating these 
> tests? If not, if somebody has the positions in a file, I might be able to 
> devote some CPU time to it. 
> 
> The tests I ran were several years ago, so they weren't with the latest 
> weights. I think they'd have been the 0.14 weights.
> 
> Did Michael give you evidence that 3-ply was worse than 2-ply? I don’t 
> remember any tests being reported on the mailing list, and I'd be very 
> interested.
> 

I don't think Michael Depreli drew any conclusions but I believe others (as 
quoted at the bottom of the study reinforced the idea that 3ply for cube 
decisions was a poor choice). The study is here:
http://www.bkgm.com/articles/Keith/DepreliBotComparison/

I told Lucas about the odd/even ply effect and directed him to the Depreli 
study. That study was done with the Pre-1.00 neural nets that (The Rudman DLL 
is well over a decade old and definitely uses the older net). I don't recall 
saying "a little". There is no conclusive evidence since I don't recall a full 
study being done, but Philippe Michel did ask me (around the time of the 1.00 
release) to run the new neural net through the Depreli positions to see how the 
new net faired. The results of that can be found here:

http://www.capp-sysware.com/downloads/gnubg/gnubg_prelim_results-1_00.txt

If you review Grandmaster and the experimental 2ply chequer/3ply cube with the 
old and new net relative to 2ply and 4ply you will see that it did does quite a 
bit better. Whether the Depreli study translates to general play and positions 
is not known. If someone wishes to do such a study then all the power to them 
(I haven't discouraged anyone from doing such an experiment)

The numbers in my chart can be used as a reasonable comparison to the other 
data in Depreli's "Bot Comparison — 2012 Update" which can be found in the 
first link I gave.

--
Michael Petch
GNU Backgammon Maintainer / Developer
OpenPGP FingerPrint=D81C 6A0D 987E 7DA5 3219 6715 466A 2ACE 5CAE 3304

_______________________________________________
Bug-gnubg mailing list
address@hidden
https://lists.gnu.org/mailman/listinfo/bug-gnubg