I can comment on that: my experience from 20 years ago was that at some stage adding positions started to hurt the net performance. It is always a balancing act between getting the common/regular positions right and getting the edge cases right. I think that whatever you do you might want to start fresh and see how my "method" (as you outlined above) can be improved.
Yes, I think I remember that you have mentioned that before. The reasoning behind it might be due to the size (hence capacity) of the neural network. Maybe, with a bigger and deeper neural network, and modern training algorithms, a bigger training set can be used and still get better performance. As you say, there is a sweet spot between getting the common positions right, and then getting the edge cases right.
Yes, the outlined method is (of course) Joseph's idea. In my view, he is the best backgammon neural network trainer. Maybe I should start this process on my own, and gain some experience before involving a community with effort. It will be really unfortunate if we waste resources on a braindead idea.
How can the outlined idea be improved? Before I get into that, I think I need some experience, but maybe see if there's some special kind of positions that's over or under represented in the set, and then automagically (in some way) detect these?
-Øystein