I don't really understand the criticism. The authors aren't claiming to have the strongest chess engine without search. They are just showing that they got a chess engine to a respectable level with their process, which is somewhat different from LC0. They do in fact explain that explicitly:
> Leela Chess Zero’s networks, which are trained with self-play and RL, achieve higher Elo ratings without using explicit search at test time than our transformers, which we trained via supervised learning. However, in contrast to our work, very strong chess performance (at low computational cost)
is the explicit goal of this open source project (which they have clearly achieved via domain-specific adaptations). We refer interested readers to [https://arxiv.org/abs/2409.12272] (which was published concurrently to our work) for details on the current state-of-the-art and a comparison against our network.
And I don't think the criticism of their writing is on point either. I don't think they are secretly implying that their engine is better than Stockfish. And it's 100% plausible for human masters to rigorously analyze many positions with engine assistance and correctly establish whether Stockfish's evaluation is right or not.
Can the experiment be summerised by saying that training the model is a kind of probabilistic pre-calculation that converts the Stockfish expert system into a different, rather distinct representation that is worse than Stockfish, but still quite good?
> Particularly egregious is that they then elect to resolve this difference in opinion by appeal to human masters, who are hundreds of elo weaker than Stockfish!
What a petty complaint. A human expert analyzing a move, with access to stockfish and every other chess program they want, can be a very good analyst.
Well, if you have a perfect evaluation function, you don't need to search. And if you can do a perfect search to the end, you don't an evaluation function. Un(?)fortunately none of these extremes seems reasonable for a game like chess (and even less for go). So most software use both search and evaluation. And a whole lot of optimizing and other tricks. With impressive results.
5 comments
[ 3.0 ms ] story [ 24.2 ms ] thread> Leela Chess Zero’s networks, which are trained with self-play and RL, achieve higher Elo ratings without using explicit search at test time than our transformers, which we trained via supervised learning. However, in contrast to our work, very strong chess performance (at low computational cost) is the explicit goal of this open source project (which they have clearly achieved via domain-specific adaptations). We refer interested readers to [https://arxiv.org/abs/2409.12272] (which was published concurrently to our work) for details on the current state-of-the-art and a comparison against our network.
And I don't think the criticism of their writing is on point either. I don't think they are secretly implying that their engine is better than Stockfish. And it's 100% plausible for human masters to rigorously analyze many positions with engine assistance and correctly establish whether Stockfish's evaluation is right or not.
What a petty complaint. A human expert analyzing a move, with access to stockfish and every other chess program they want, can be a very good analyst.