It is well known that Kramnik baselessly accuses everyone. The article seems to be more about statistics than chess, and doesn’t make any accusations. Kind of a click bait title IMO.
The underlying flaw in this analysis is it assumes ratings reflect actual performance in a given game. A long winning streak becomes far more likely if one of the players is part of several matches while tired, drunk, etc. Similarly a players peak performance is going to be higher than their ELO because that ELO includes games played under less ideal conditions.
ELO is presumably more accurate for over the board games at tournaments where players bring their A game than low stakes online games where someone may be less engaged. That’s IMO more worth testing.
This is basically an article describing why you can’t just look at an event after it occurs, see that it has some extremely rare characteristics, and then determine it was unlikely to happen by chance.
It is like asking someone to pick a random number between 1 and 1 million and then saying, “oh my god, it must not actually be random… the chances of choosing the exact number 729,619 is 1 in a million! That is too rare to be random!”
>This is basically an article describing why you can’t just look at an event after it occurs, see that it has some extremely rare characteristics, and then determine it was unlikely to happen by chance.
No. That's not it. In this case, if you properly control for all the factors, it turns out that the odds of Nakamura having that kind of a win-streak (against low-rated opponents) was in fact high.
My favorite way to describe this is in the context of predictions. It's the difference between throwing a dart to hit a target and throwing a dart to paint a target around where it lands.
> This is basically an article describing <snip hot take>
This is entirely wrong and missing basic high school mathematics for non-theater kids.
The original claim is not archived, if you can be bothered you can track it down and do the correct 'hot take'. You can't just grab the first statistical principal you think of even if everyone else on Hacker News does.
Article - "it violates the likelihood principle", this seems wrong and Nakamura seems right, but you'd have to look at the original claim.
They were finding patterns in a long biased list of numbers, probably.
I'd be much more suspicious if his online performance didn't track with his professional over-the-board performance, where cheating would be much more difficult.
If you want a deep dive into chess cheating, including a lot of wild stories, Sarah Z put out an entertaining Youtube video [1] a couple of months ago that explores the concept. It's a long video, but well worth the watch.
Kramnik has proven himself a troll and did a lot of damage to the reputation of honest and otherwise wholesome people and this without credible proof at all. That those allegations persist under the form of news article is very unfortunate to those victims of his smearing.
While it is good to see some Bayesian statistics in use, I wouldn't in this case put so much emphasis on an exact calculated probability that he did or din not cheat, the prior in this case is simply too wishy-washy for that.
The sound conclusion is that this is not evidence of cheating, but it is not evidence of the contrary either.
There is probably 1000 hours of videos online of Hikaru talking through games, literally exhibiting his skill in full transparency. Hard for me to even understand what it would mean for him to cheat, his brain at times feels like the cheat.
>>The researchers note that there’s a problem with this argument, too, as it violates the likelihood principle. This principle tells us the interpretation should only rely on the actual data observed, not the context in which it was collected.
and then in the publication itself:
>>The likelihood principle [Edwards et al., 1963]
is a fundamental concept in Bayesian statistics that states that the evidence from an experiment is
contained in the likelihood function. It implies that the rules governing when data collection stops
are irrelevant to data interpretation. It is entirely appropriate to collect data until a point has been
proven or disproven, or until the data collector runs out of time, money, or patience
Surely there is a difference when you look at someone who played 46 games online in his life and scored 45.5 and when you look at someone who played 46000 games and scored 45.5/46 once.
The difference is that Kramnik wasn't "collecting the data" but looked at the whole Nakamura's playing history and found a streak.
Another example would be looking at coinflips and discarding everything before and after you encounter 10 heads in a row to claim you have solid evidence that the coin is biased.
They are misapplying the principle here. If what they wrote was correct then someone claiming: "Look, Nakamure won 100 out of 100 if you just look at games 3, 17, 21, 117...." would be proving Nakamura cheated if they applied methodology from the paper even assuming one in 10000 guilty players.
Just because you can choose sampling strategy and stopping rules (what the likelyhood principle states) doesn't mean you can discard data you collected or cherry pick parts that support your hypothesis.
How the data is collected is absolutely relevant and Nakamura is right to point it out.
> Nakamura responded to Kramnik’s allegations by arguing that focusing on a particular streak while ignoring other games was cherry-picking. The researchers note that there’s a problem with this argument, too, as it violates the likelihood principle. This principle tells us the interpretation should only rely on the actual data observed, not the context in which it was collected.
I don't quite understand this objection? If I won the lottery at odds of 10 million to 1, you'd say that was a very lucky purchase. But if it turned out I bought 10 million tickets, then that context would surely be important for interpreting what happened, even if the odds of that specific ticket winning would be unchanged?
Tangently related, reading this I couldn't help but think about the biological passports for professional cyclist. It tracks blood and other values overtime, so that anomalies will jump out.
What are the odds that a cheating accusation accurately identifies an instance of cheating?
I don't say this lightly: Kramnik very likely has some sort of untreated psychiatric disorder. He is effectively a lolcow in the chess community because he regularly (as in, almost daily) accuses much better chess players of cheating.
It's honestly a bit undignified to treat his accusation against Nakamura as anything other than a man yelling at the sky.
I find it counter intuitive that the frequency of cheating matters. It's not something that happens randomly, people choose to. And if the #2 in the world decides to cheat it may be for different reasons than other players.
But, of course he doesn't. He streams all his games and gives constant stream of consciousness commentary. If you can explain your top level moves live with seconds per move, you aren't cheating.
> It's not something that happens randomly, people choose to.
In Bayesian analysis, probability does not refer to the long-term frequency but instead to the subjective credence given to the event. Otherwise the probability of any one-off event would be undefinable. Therefore it follows that you need to have a prior over possible hypothesis in order to update your beliefs systematically according to the laws of probability theory. If it were known that Hikaru had cheated in the past, but typically does not, we might use a different prior (e.g. a Laplacian prior in this case); if we knew cheating to be dependent on some other measurable variable (e.g. the emotional state of the player), we would incorporate this into our evidence.
It's frustrating that their entire analysis is based on the claim that cheating occurs in maybe 1 out of 10,000 games; they got this from a quote in an interview with the deputy president of the World Chess Federation after he had been beaten in a charity match by someone who admitted cheating. To their credit they also ran the analysis assuming cheating is 1/500 and the odds rose to 7%. I suppose it makes sense that they are merely rebutting the accusations based on the same methodology but it's still frustrating.
> Kramnik pointed out the statistical improbability of Nakamura’s streak and stated that such a winning run would require the chess prodigy to play at a level higher than his current Elo rating (an estimate of a player’s skill level based on their historical play).
While ELO ratings are a probabilistic model, who said wins and losses have to be randomly distributed, there can be bad days and good days, for example if you haven't slept or if you are at the peak combination of study and cognitive, say because you are well rested on a monday and have been studying on the weekend.
Online chess is nonsense. One can beat all detection of using engines. Use 10 different engines and take their moves in random order. To avoid getting same move from all engines, make them to have varying strength.
31 comments
[ 2.8 ms ] story [ 58.4 ms ] threadELO is presumably more accurate for over the board games at tournaments where players bring their A game than low stakes online games where someone may be less engaged. That’s IMO more worth testing.
It is like asking someone to pick a random number between 1 and 1 million and then saying, “oh my god, it must not actually be random… the chances of choosing the exact number 729,619 is 1 in a million! That is too rare to be random!”
No. That's not it. In this case, if you properly control for all the factors, it turns out that the odds of Nakamura having that kind of a win-streak (against low-rated opponents) was in fact high.
This is entirely wrong and missing basic high school mathematics for non-theater kids.
The original claim is not archived, if you can be bothered you can track it down and do the correct 'hot take'. You can't just grab the first statistical principal you think of even if everyone else on Hacker News does.
Article - "it violates the likelihood principle", this seems wrong and Nakamura seems right, but you'd have to look at the original claim.
They were finding patterns in a long biased list of numbers, probably.
"For safety, I always bring a bomb with me when I fly"
"Why?"
"Because the odds of two people bringing a bomb on the same plane are so low"
[1]: https://www.youtube.com/watch?v=ZtN-i-IkRWI
The sound conclusion is that this is not evidence of cheating, but it is not evidence of the contrary either.
and then in the publication itself:
>>The likelihood principle [Edwards et al., 1963] is a fundamental concept in Bayesian statistics that states that the evidence from an experiment is contained in the likelihood function. It implies that the rules governing when data collection stops are irrelevant to data interpretation. It is entirely appropriate to collect data until a point has been proven or disproven, or until the data collector runs out of time, money, or patience
Surely there is a difference when you look at someone who played 46 games online in his life and scored 45.5 and when you look at someone who played 46000 games and scored 45.5/46 once.
The difference is that Kramnik wasn't "collecting the data" but looked at the whole Nakamura's playing history and found a streak.
Another example would be looking at coinflips and discarding everything before and after you encounter 10 heads in a row to claim you have solid evidence that the coin is biased.
They are misapplying the principle here. If what they wrote was correct then someone claiming: "Look, Nakamure won 100 out of 100 if you just look at games 3, 17, 21, 117...." would be proving Nakamura cheated if they applied methodology from the paper even assuming one in 10000 guilty players. Just because you can choose sampling strategy and stopping rules (what the likelyhood principle states) doesn't mean you can discard data you collected or cherry pick parts that support your hypothesis.
How the data is collected is absolutely relevant and Nakamura is right to point it out.
I don't quite understand this objection? If I won the lottery at odds of 10 million to 1, you'd say that was a very lucky purchase. But if it turned out I bought 10 million tickets, then that context would surely be important for interpreting what happened, even if the odds of that specific ticket winning would be unchanged?
What are the odds that a cheating accusation accurately identifies an instance of cheating?
I don't say this lightly: Kramnik very likely has some sort of untreated psychiatric disorder. He is effectively a lolcow in the chess community because he regularly (as in, almost daily) accuses much better chess players of cheating.
It's honestly a bit undignified to treat his accusation against Nakamura as anything other than a man yelling at the sky.
But, of course he doesn't. He streams all his games and gives constant stream of consciousness commentary. If you can explain your top level moves live with seconds per move, you aren't cheating.
In Bayesian analysis, probability does not refer to the long-term frequency but instead to the subjective credence given to the event. Otherwise the probability of any one-off event would be undefinable. Therefore it follows that you need to have a prior over possible hypothesis in order to update your beliefs systematically according to the laws of probability theory. If it were known that Hikaru had cheated in the past, but typically does not, we might use a different prior (e.g. a Laplacian prior in this case); if we knew cheating to be dependent on some other measurable variable (e.g. the emotional state of the player), we would incorporate this into our evidence.
why was it changed? This isn't a 'recent' story, it's from January.
The actual chess community's takeaway from this (if consensus is important to you) is that Kramnik (the accuser) has lost it a bit.
While ELO ratings are a probabilistic model, who said wins and losses have to be randomly distributed, there can be bad days and good days, for example if you haven't slept or if you are at the peak combination of study and cognitive, say because you are well rested on a monday and have been studying on the weekend.