Re: [Bug-gnubg] Confused

[tchow]

On 2015-06-15 08:24, Ian Shaw wrote:
> I was under the impression that many more games were required to
> separate backgammon bots.
>
> I suppose that a match can only have a 0 or 1 outcome, unlike $ games.
> This must reduce the number of required trials somewhat.

Well, it certainly simplifies the calculation; whether it reduces the 
number of trials is not immediately clear to me because I'm not sure 
exactly how I would do the money-game calculation.  There are couple of 
different plausible possibilities.

But more to the point, I actually have the same impression that you do, 
that more trials are needed.  Lucas's results surprise me because on the 
surface, they suggest that 2-ply is about 75 Elo stronger than 3-ply.  
That is a much larger Elo gap than I would have expected.  I suspect 
that 2-ply isn't actually as much as 75 Elo stronger than 3-ply.  As I 
said before, while Lucas's results do strongly suggest that 2-ply is 
stronger than 3-ply at 5-point matches, they don't give us a very sharp 
estimate of *how much* stronger 2-ply is.  If we wanted that then we 
should start a fresh series of matches, stating ahead of time what 
hypothesis we're testing (e.g., that 2-ply has at least a 55-45 edge 
over 3-ply in 5-point matches).

Tim