Very interesting. I don't play chess but like to observe others playing and talking about it on YouTube.
I thought about something the other day... Modern chess sites run proprietary algorithms to detect cheating by modern engines. But would it be possible for an old engine, let's say from the 70s or 80s, running on modern hardware to deceive these algorithms?
They wouldn't win every tournament but still have a respectable performance.
From what I gather cheat detection uses a combination of factors to determine how "human-like" an account is: time usage, webpage interactions, move quality under time pressure and so on. I think you could probably try to deliberately deceive a specific website's cheat detection with software that tries to emulate how humans play, at which point you'd be in the classic attack vs. defense arms race.
Also notice that for events like Title Tuesday you must play on a dedicated client with at least 2 cameras, so it would be significantly harder to pull it off.
With AI gaining consciousness I can't wait for such "human-only" measures to be delegated to the trashcan of history, same as slavery. What happens when AI-human cyborgs start walking around? Will they also be treated as second class citizens?
I've built engines from scratch. I don't think that would work at all. It's a whole extra challenge to make an engine feel like a human. The engine has to come up with its best move and then sometimes choose not to play it and play something else for some reason. There are statistical patterns to how humans make mistakes based on their level, what kinds of mistake, how big and how often. Cheat detection is less about knowing the engine and more about statistics.
I grew up with these machines around, and as a kid they seemed to so much more powerful than humans. It's kinda weird to think that I became 'smarter' than them eventually. I suspect they will be able to run superhuman AI one day too.
I don't think that real-time, superhuman AI performance is ever gonna happen with <1b parameter models.
Biology had a lot of time, attempts and pressure to make this work but even the most optimized lines (something like a magpie) need more complexity (almost a billion neurons) to get a fraction of superhuman intelligence.
Results with small artificial neural nets are very limited so far, too.
On the other hand, symbolic reasoning easily gets superhuman results in narrow domains. The Robbins Conjecture was proved on hardware like that after 60 years. You could check out this theorem prover benchmark suite that has been maintained for over 30 years https://www.tptp.org/
> But yes, totally unsurprising that a megahertz machine with 100MB+ available ram has >2500 ELO.
I think this is underselling it a bit. If you'd told this to the IBM engineers at the time I think they would have taken great interest in how the performance was achieved.
In one sense, Knuth fans and really good engineers are correct to prioritize algorithms.
Unfortunately, most humans see big|new|expensive hardware as great status signifiers, barriers to lesser folks joining the Cool Club, and optimal sources of sales commissions.
> the average chess grandmaster during a classical game will analyze roughly the same number of moves as an amateur player, but they analyze only the most critical moves and do a better job with that analysis.
I don't buy this. I've seen GMs playing online while verbalizing their thought process (on twitch / youtube videos where they livestream their play), and they consider way more branches than I do, while also going deeper than I ever would on some branches. My top rating in lichess blitz is about 1900, not bad but way lower than a GM.
Much of what's happening is less like 'calculations in the back of their head' and more like 'pattern recognition'. Chess masters are recalling ideas and motifs (not necessarily actual positions) from earlier games or practice or study, and this recall is allowing them to either prune the decision tree or prioritize investigating more fruitful branches faster than amateur players, who have to actually calculate things that are instantly 'seen' by the better players.
This is a known thing among chess players, and even low-skilled players do it to some extent. A newbie to chess can (and almost every teacher will say 'should') work their pattern recognition by just solving tactical chess puzzles and very quickly improve at spotting things like one or two move checkmate patterns or forks or pins or whatever.
It's more of a memory thing than subconscious calculation.
Vibes is for the initial pruning, that’s why great chess players will sometimes see a computer move and ‘instantly’ realize why it’s so good but in a real game never consider it.
> Much of what's happening is less like 'calculations in the back of their head' and more like 'pattern recognition'.
Sometimes it really looks like "calculations in the back of your head". Have you ever been struck with an 5 moves long tactics in positions you've never seen? Because I have.
There are hardly any other explanations than subconscious calculation.
GMs also have access to a greater number of memorized situations. Regardless of whether they are conscious of it or not, they have previously analyzed so many moves that they can cut off branches they just know instinctively to be dead ends. Machines could do this, directly reference a massive dataset of precomputed board positions, but that isn't considered a proper chess engine... despite the fact that human GMs do use such techniques.
Yeah I would like to see where that factoid comes from. Maybe it means they explore as many branches from the root position as an amateur, but they explore each branch way deeper and more accurately
After some digging it goes back to Adriaan de Groot (in 1946).
[1] (2011) has a summary in the introduction:
de Groot presented chess players of varying skill, including some of the best players of the time, with an unfamiliar position and asked them to think aloud as they analyzed the position and chose what move they would make. de Groot found that, contrary to popular belief, the strongest players did not think further ahead than weaker players. World-class Grandmasters (GMs) searched at a similar depth and considered a similar number of options as mere chess experts.
As [1] is also a somewhat reproduction of the original study, you can take it directly.
Six Grandmasters, five International Masters, six Experts, and five Class A players completed the think-aloud procedure for two chess positions. Findings indicate that Grandmasters and International Masters search more quickly than Experts and Class A players, and that both groups today search substantially faster than players in previous studies. The findings, however, support de Groot’s overall conclusions and are consistent with predictions made by pattern recognition models.
Section 3 Table 2 is probably most interesting, they had 2 board positions and 2 groups sectioned:
Masters (2350+) | Intermediates (1800-2200)
(Mean values only, i tried inlcuding stddev but it was hard to read)
Positions Evaluated (A): 145.3 | 138.6
Positions Evaluated (B): 216.6 | 190.6
---
Max Depth (A): 11.7 | 9.4
Max Depth (B): 14.1 | 12.0
---
Mean Depth (A): 4.1 | 3.3
Mean Depth (B): 4.9 | 4.2
What i found interesting was that the "intermediate" group had a higher number of "base moves" they evaluated (eg Pos A: 5.6 | 8.6).
The previous studies (de Groot and another rperoduction from 1998) had "Nodes visited" for position A at 51.6 | 34.3 and 48.3 | 40.1 so similarly close together as the new ones, but much lower. (depending on your definition on "similarly close together")
I am about 300 points higher and think this is roughly correct. In openings amateurs will actually analyze more since GMs just have hundreds of opening lines memorized many moves deep. Middlegame and endgame there're usually just 1-3 reasonable plans that are obvious to a GM (but not at all obvious to amateurs) with 1-3 initial moves each. So, again, probably about as many as I consider.
The only time when GM will definitely analyze more lines than an amateur is in a highly tactical position but that happens, again, maybe 1-3 times per typical game (tactics themselves will last several moves, but they would be largely calculated through at the beginning of the tactical melee).
Yes, I am even a little higher rated and have played numerous (low rated GMs) in tournament play. I do think they analyze more than 1-3 times per game.
The main thing commentators here are missing is that GMs have more confidence and do not repeat their analysis, but amateurs (strong club players included) typically re-analyze the same lines over and over. That alone would account for far stronger play with fewer moves considered, if you count the same move multiple times, which you should!
Also, Jonathan Rowson (Scottish GM - peak about 2600) describes his training matches with Mickey Adams (peak #3 or so in world - 2760), and says pretty much the same thing. Most of the time he asked “what about this line I rejected”, the answer was usually “I hadn’t considered that”.
They don't reanalyze nearly as much as some amateurs do, but one of the most salient criticisms of Kotov's "Think Like a Grandmaster" is indeed that in practice the actual GM analysis process is often much more messy and chaotic than portrayed and often involves a lot of going back and reanalyzing / re-deepening previously analyzed lines in light of something learned from analyzing another line.
> The only time when GM will definitely analyze more lines than an amateur is in a highly tactical position
Those critical moments are a big part of the difference between a GM and a good club player though. If a GM was somehow handicapped to not calculate such tactics, I think lesser players would be able to draw games against GMs much more often.
an amateur player --> shoudln't an amateur player have an average ELO, i.e. be nearly as bad as me, who never wins at chess when someone even knows their ELO or is a bot.
My rating is negative 1900. I was always bad at chess and never really gave it a chance tbh. I always wondered though where does the pattern matching stop and strategy starts or it's one and the same?
I strongly believe some people are inherently good at chess but no one is inherently bad. If you're interested enough to comment this, play one 5-minute game a day on lichess. Start with the lowest setting but do it every day, maybe in the morning with coffee. You might not win a single game the first year but slowly you'll improve. When you're winning more, move up a level. Sometimes do the puzzles. Give it five years and you'll be able to beat almost anyone you meet and, better yet, you'll know the answer to your question based on your own experience.
You know what? I'll take you up on the offer. I have a complimentary chess dot com account with my revolut ultra, might as well use it. Every day a toilet break chess and see you in five years!
I think by virtue of being the kind of person who comes to Hacker News, and perhaps socializes with similar people, you are not "sufficiently" exposed to people who are simply incapable of logical thinking, or are unwilling to apply their minds in that way.
I became significantly better at chess through two simple things:
1. Take a little extra time to visualize what will happen after you make a move. Will the piece be vulnerable? Does moving that piece leave a different piece vulnerable? I'd argue the biggest beginner mistake is not seeing that their move leaves a different piece vulnerable to being captured.
2. Don't move your queen into the board until after you move both bishops. This simple rule of thumb makes it far less likely you will lose your queen early, and it forces you to do a lot of other useful things (develop your pieces, control a portion of the board, etc).
Disclaimer: I am not good at chess, but when I played semi-regularly these helped me hold my own against other beginners.
There's an analog in machine chess. According to the research paper from 2017, AlphaZero considered ~10,000 moves per decision, 1,000x fewer than Stockfish at the time. h2h it beat Stockfish pretty badly.
But that's not what it's saying right? The quote says grandmasters and amateurs analyze the same number of moves. I mean it would kind of make more sense how you are interpreting it but is that actually what the quote is saying?
I was responding to the commenter's intuition that GMs do actually analyze more lines. I'm saying the original article's claim checks out even for machines.
It is same, just even more overpowered. AlphaZero can also do same amount of analyze but before even need to, Stockfish already destroyed. Is like 100m race run between you and Usain Bolt. Because Bolt is so overpowering, You can just shorten it to 30m and he still beats everytime.
>the average chess grandmaster during a classical game will analyze roughly the same number of moves as an amateur player, but they analyze only the most critical moves
What does this mean exactly? They analyze the same number but the grandmaster analyzes "only the most critical moves."
So does that mean at every given analysis opportunity (each move I guess), they analyze better candidates?
If so I guess these 2 things are totally redundant? "they analyze only the most critical moves and do a better job with that analysis."
Alternatively I wondered if perhaps they actually perform analysis on less moves (perhaps because they already know the lines or remember their previous analysis). But I think that contradicts the claim that they analyze about the same number of moves.
They have tree pruning heuristics. Probably all human masters of any craft have them. They're precise to some given extent, after which tacit/embodied knowledge, aesthetics and vibes take over.
Meaning, you analyse 100 moves will not be the same set of GM 100 moves. And GM work on better 100 moves than yours. If you pick the moves that GM not thinking of, you likely get crushed because those moves are very inferior moves that GM already can see the outcome without the need to consider. Meanwhile if GM takes on moves you dont consider, you get crushed too. The likely silver lining will be you picking the same moves as GM. But up to a point, his depths will crushed you either you pick a wrong moves by mistake, or he out think you a few step more. Is like a math student who do same amount of steps as yours, but you look at his answer in awe because better and succint...and even confused as how it reach that. TBF, today AlphaGo engine will crush Stockfish and any GM too.
> Meaning, you analyse 100 moves will not be the same set of GM 100 moves. And GM work on better 100 moves than yours. If you pick the moves that GM not thinking of, you likely get crushed because those moves are very inferior moves that GM already can see the outcome without the need to consider.
This is intuitive and I guess I'm not surprised this is what was meant. They analyze the same amount of moves they are simply better at it (picking candidates, calculating, etc.)
I guess maybe im being a bit nitpicky but I feel like this is still a misleading way to say that.
> the average chess grandmaster during a classical game will analyze roughly the same number of moves as an amateur player, but they analyze only the most critical moves and do a better job with that analysis.
You could just say
> the average chess grandmaster during a classical game will analyze roughly the same number of moves as an amateur player, but they do a better job with that analysis.
"only the most critical moves" soudns like a limit on the number of moves based on "only the most" and by differentiating it from simply being higher quality.
It's totally true that most people overestimate how much of GM's abilities comes from pure calculation, and underrate intuition. That's what it's getting at.
But in classical games GMs are, in fact, doing tons of calculation, and the statement is literally completely false.
I'm not sure. I'm not into chess but I've watched other strategy games; I'm fairly confident streamers are vocalizing a lot more than they would actually be consciously considering if they weren't streaming.
Back in the day I remember Jon Finkel (a legendary Magic: the Gathering pro) saying* you're supposed to "focus only on what matters."
It stuck with me because I think it's genuinely what they do, but also seems like completely useless advice. I don't know what matters! That's what makes you a pro and me not.
* Wasn't sure if my memory was correct. A web search finds a Mike Flores article citing Zvi calling this Finkel's Law.
I agree. Everyone should check out this famous post-match interview with a strong GM, where's he's simply asked "Can you tell us about the game?" and talks for 13 minutes straight (with no board to reference) about all of the lines he calculated: https://www.youtube.com/watch?v=pUgvAoTzWBA
Notable because he departs from the book opening early on and has to find some brilliant moves to keep himself out of trouble. The breadth and depth of his calculations seem beyond what even strong amateurs consider, more than just choosing better lines to analyze.
> As it turns out, the majority of the Elo gains (+720 of the +1335 gained) can be attributed to better programming, and not better hardware.
It's not so easy, a lot of that better programming can be attributed to better hardware, it wouldn't have been possible to implement NNUE or Fishtest without modern hardware.
I agree. It is hard to make a separation between what is possible thanks to programming vs hardware. It is only easy to analyze post factum that certain ideas worked. Maybe there are also other factors that are important - time, availability of hardware, knowledge dissemination. What would be possible if the hardware stopped improving.
Because in competitive chess games, the time limit is usually 40 moves in 90 minutes. You don't have a fixed amount of time per move. It's up to you how you want to spend that time budget.
I'll go out on a limb and say that future AI models, running on today's smartphones, will be better at programming and formal math than today's frontier models running in datacenters. And by "future" I mean less than 5 years from now.
Do we actually know enough about neural networks to be able to say how many parameters you need for a given performance? I’m under the impression that we’re still in the phase of trying things out and we can’t rule out different architectures massively reducing computational needs.
We can't rule out different architecures, but absent some architecure or hardware or quantisization breakthrough, frontier models will simply not be fitting on phones.
That's a good question, and I don't know the answer, but I can try to guess.
An 80386 is about 10–13 Dhrystone MIPS, while a 200MHz Pentium Pro is about 90–450, according to https://netlib.org/performance/html/dhrystone.data.col0.html. So we're talking about a further slowdown on the order of 20×. I don't know how to estimate how much that would lower its Elo.
The article says:
> By making a number of adjustments to the algorithm and tuning it to the Pentium, we can increase its Elo to 2846 ± 35 at 1 second per move, 3109 ± 79 at 10 seconds per move, and 3180 ± 80 at 30 seconds per move.
So making it 30× faster would raise its 1-second Elo by about 334 points, and ln(30) is 3.4, so that's about 100 points per factor of e. ln(10) is 2.3, so we'd interpolate a prediction of 230 points gained at 10× faster, reasonably close to the 263 points reported.
If we extrapolate that in the other direction, ln(20) is 3.0, so on the 80386 we'd expect a loss of about 300 Elo points, down from 2846 to 2550 or so. Still enough to wipe the floor with Crafty, but low enough that Kasparov would usually beat it.
But we don't really know if that extrapolation is valid.
> I don't know how to estimate how much that would lower its Elo.
A common rough rule of thumb used throughout the 2010s (because this no longer holds today) in computer chess was that a 2x speedup gives about 50 Elo. This mostly holds in that region but at low levels it is more like 100-200 points.
My estimate of 100 ln(r) Elo gives 69 Elo for r = 2, which seems in the ballpark. And it's sort of reassuring that people had already come to the conclusion that Elo was basically logarithmic in computer speed for a given algorithm before my baseless guess in the grandparent comment.
When I tried this with a 6502 (Apple II), it did not work. It was just too underpowered for enough modern algorithms to kick in. It might also be an LLm-driving skill issue of course!
> because who would want to buy a Pentium Pro when the better-in-almost-every-way Pentium MMX
Maybe this was satirizing people running after every shiny new thing, but the MMX certainly wasn’t. The Pro was the first P6, a totally new, out-of-order microarchitecture that the Pentium 2 would be based on. The MMX was a standard P5 Pentium with a larger L1 and the MMX vector instruction set which, while the first of its kind in an x86, was ultimately not used much in applications, partly because it was integer-only. In any case, the Pros were expensive and meant for business use, while the MMXs were consumer processors.
Just one caveat: the Pro overlapped in time with people running a lot of 16-bit code, and I think the Pentium MMX was faster for that, or at least that the Pro didn't offer much edge.
Yeah, the Pro and MMX were totally different market segments, not alternatives for each other. The Pro's competition at the time was DEC Alpha, PA RISC, etc. It wasn't a chip that made sense for ordinary PC users at the time.
Later, when Pentium MMX was eventually replaced by Pentium 2 for consumers, the successor of Pentium Pro was the first Intel Xeon CPU (which was a slightly enhanced Pentium 2, with bigger cache memory and with ECC memory).
The launch of Pentium Pro in 1995 was the moment when Intel segmented their CPU line into CPUs for consumers & server/workstation CPUs.
Yes at that time Intel invented the market segmentation between "Pentium" for "consumers" and "Pentium Pro" for big spenders, and to make Pentium Pro more attractive they removed memory errors detection from the "Pentium" chipsets, which is the origin of the current state, when most non-server computers are very vulnerable to memory corruption and nobody can calculate how much money was lost by most computer owners during the last 3 decades, due to the greed of the Intel marketing, because nobody knows how many of the computer crashes or data loss cases that were typically attributed to unknown software bugs were actually caused by memory corruption.
Because this move was a money maker for all involved, all memory producers and also AMD followed immediately the Intel initiative. Memory modules without ECC would have never existed if adequate consumer protection laws had existed.
Great project but I have a nit on the board displays: every board is displayed as <human username> vs Pentium Pro, even when the Pentium is playing White. The normal chess convention is <White name> vs <Black name>.
I spent some time trying to figure out how some of the humans seemed to have trapped the machine out of the opening, because it looked like the human was winning.
> (Also, uh, if anyone has any ideas for projects I could do with a Pentium Pro in a few weeks once I no longer need it to host this chess engine ... do let me know!)
Throw a 3dfx Voodoo in there and do some real-time 3D Gaussian Splatting!
Run some current and still-compiled-for-i686 Linux or *BSD distros on it. See what breaks, save logs & report issues upstream.
A lot of support for old CPUs & platforms they're on goes through emulators like QEMU. Or plain old 'go with specs, disturb code as little as possible, and assume (hope) it still works'. But nothing beats running on actual hardware. Uncommon configurations, specific drivers that may have been broken (sometimes for a looong time before someone notices), spec says A but hardware does B, etc.
Cool exploration, and very much needed because depressingly few people who use these engines actually understand this.
Back in 2021 I got cfish and weiss working on a 386sx, though didn't test it much. cfish was doing 5k nps so some results can be extrapolated. NNUE complicated things due to RAM requirements. Though I did train a small one for lichess at some point and they can be surprisingly good.
Very cool project. It seems incredible that such a "humble" device for the time would match and beat deep blue. It must be true that software has been living off hardware for a very long time, even if as others comment, the latest developments in software have only been possible after hardware got much much better.
A modern chess engine would definitely have defeated Kasparov in 1996. It would have had a database containing all the games, particularly that strange game against a computer nobody knew anything about, which seemed to be able to anticipate all of Kasparov's moves.
Even without an explicit DB it would have been quite strong, core Stockfish would be enough to beat any human as well. Ofc learnings from strange games would still be compressed somewhere in the eval functions.
On a kinda similar note i always wander that about indie video games. We have this massive hits like balatro, vampire survivors, that easily could be made in the era of SNES or N64. But somehow we have to wait 30 years to get them.
It clearly wasn't technical side of things that prevented that, but also I don't believe it was lack of creativity.
Some could argue that it is just what art is, we will get every year some new thing, creating all "possible and meaningful" art right now isn't even valid thing to consider, because art isn't about exhausting a finite set of possibilities.
But at the other hand some mechanics, design solutions in the hindsight seem kinda obvious and make me wonder, what things we do not see now, but will look obvious to people 30 years from now.
> We have this massive hits like balatro, vampire survivors, that easily could be made in the era of SNES or N64. But somehow we have to wait 30 years to get them.
> It clearly wasn't technical side of things that prevented that, but also I don't believe it was lack of creativity.
I don't know those specific games, but I think this often happens because, although the thing has been technically possible for decades, it used to be relatively difficult/expensive/obscure, and now it's far more accessible.
In those days, writing and publishing a SNES game was something that relatively very few people would have even thought to attempt, let alone had the knowledge and skills and capital to pull off successfully. Nowadays, creating and publishing a SNES-level game is comparatively very easy (at the technical level, that is; making a good original game is of course still hard), and so a large portion of the people with an idea and some motivation can actually go ahead and do it. Many of the resulting games won't be very interesting, but collectively we're exploring the design space much more quickly.
edit: I was probably silly to focus on literal SNES games instead of 'SNES era' games, but the point still stands: creating games for any platform back then was harder and more obscure than it is now (especially given that we're talking about a technical level that required quite efficient use of the hardware back then, but can be done with extremely inefficient use of the hardware now).
> it used to be relatively difficult/expensive/obscure
I don't know I fully agree with this. Consoles always had the "walled garden" thing going for them, but on the other side of things, developing games became so cheap and easy during the personal computer revolution that whole industry collapsed in '83 under a massive flood of shovelware...
That's a fair point, but I think what I said still holds, with emphasis on the word 'relatively'. I don't think you could argue that it didn't get even easier, and significantly so, between then and now.
It's a bit of a double-edged sword, because at the same time that tools and documentation have become much more available, so too have fidelity requirements gone sky-high
Just because a game could graphically have been displayed on an N64 doesn’t mean the game could be built. Vampire Survivors has hundreds of units on screen at a time firing hundreds of projectiles and weapon effects; running those on a memory bandwidth-constrained 93MHz single core processor in 4MB of RAM would be, uh… a challenge.
Yes, more plausibly. The fact that Balatro is only a 150M download is definitely unusually small by modern game standards - Vampire Survivors is about ten times the size. But 150M is still too big to fit on an N64 game cartridge - those were limited to about 64M.
More opportunity to write games. It is infitenly easier to get into writing games now than back then.
Also, don't underestimate the "juice" that these days is relatively easy to do but would be expenseive back then. Balatro has dozens of very small things going on graphically that make it feel real nice to play that most certainly not be free on SNES hardware.
And like vampire survivors is just Robotron/Smash TV with infinity number of sprites on screen. The limitation is very much hardware in that case.
Better machines make it so much easier to experiment. You could make things fast on a SNES or a N64 but you'd have to sweat for it and your code turned into write-only concrete. You don't have to sweat much anymore and your code can easily stay lithe.
The N64 was extremely constrained by its cartridges. If it had a CD-ROM drive, it could have easily won the console wars of its generation, or at least would have gotten ports of many more PS1 games. Probably one of the biggest blunders in gaming business history. Ah well.
I think it's better to say that it wasn't the electronic side of things that prevented that. Game design is technical.
Every board game could have been built thousands of years ago, at least after the dawn of some reasonable writing/drawing media. But technology in design hadn't accumulated enough. To go farther, every bit of math could have been done thousands of years ago, but new math had to draw on earlier math. New math didn't rely (much) on special paper.
Also, "hit" probably isn't a good metric. Those games might have showed up back then and just been moderately notable, like dozens or hundreds of other games were.
Technology is a progression where we learn new things, built on old things.
Game design (like other creative expressions) is the same. Each new game doesn't come from a vacuum, it's a progression from everything that has come before.
Makes me wonder if a modern ESP32 could have beaten him.
Of course, “human vs supercomputer” is more exciting to the public than “human vs toaster controller”, especially when the latter in small quantities costs less than a cup of coffee.
I find the general topic of purely algorithmic progress super interesting.
I just ran Andrej Karpathy's Tiny Stories model on a tiny ESP32.
In principle we could probably have done this in the 80's and it would have been so amazing.
Interesting article. Have we just acquired better insights into programming chess? Are ELO ratings guaranteed to predict whether a machine can beat a human or can a human outsmart a higher ELO rated chess program?
At some point comparing human chess to algorithms stopped making any sense. So there is no realistic pool of games between AI and humans. We would lose every game and vs AI a human ELO would go to 0 (not sure if that is technically possible). It is like this for probably 15 years.
ELO is like probability. A 100 ELO difference is something like 2x odds of winning a game and over a lot of tries it tends to be quite accurate. That was also an argument from Magnus Carlsen that to compare human ELO you need a lot of games and he didn't want to take part in classical world championship quoting this reason (among others).
Technically interesting but also fanciful bollocks - because there was no Stockfish in 1996, and developing a sufficiently strong equivalent in time for the Deep Blue match would have taken substantially longer that the 18 years the current Stockfish has needed, thanks to the drastically weaker processor capabilities back then.
"Bollocks" feels like a very uncharitable read. Sure, investing time/money into creating a better chess engine instead of building a faster computer would not have worked out for IBM, but the post never argues that they should have done that. It makes the point that the majority of improvements of chess engines since have been algorithmic and not "free" gains from faster CPUs.
Deep Blue is just used as a point of reference to illustrate that fact.
You can't disentangle the compute from those algorithmic gains, though. Every small adjustment to its algorithms is made on the back of tests relying on compute.
Doesn't the author address this point? They speculate that it could be attained through self play back then using a ton of 1996 compute and time, but it's theoretically possible.
“In principle, I could have used the same engine itself as the teacher, but just to save on wall clock time I used Stockfish. This reduced the wall-clock time from ~4000 1996-core-years (something entirely doable in 1996---but sadly not something I can do with my single lonely desktop) to just a few days on a modern desktop running a modern engine.”
NNUE was added to Stockfish in 2020. Although this was a big improvement, it was already extremely strong before that. With plain hold hand-coded logic as a board evaluation function.
Actually, even pre-NNUE Stockfish wouldn't be an entirely fair comparison, because all hand-coded pre-NNUE heuristics was tested using Fishtest [0][1] - a distributed testing framework with compute, donated by volunteers.
And I think that Fishtest is actually what made Stockfish the strongest, because no other engine's performance ever was evaluated on such scale before.
Stockfish was the first to use hard data instead of vibes, when selecting algorithms and heuristics.
So, even if we add a rule that NNUE is not allowed, because it was trained on modern GPUs that, obviously, didn't exist back in 1996, hand-coded heuristics would also take a ton of compute to develop.
Bitter lesson strikes again, even when you don't expect it.
Yes, but the implication is about 'hardware overhangs'. It is colossally expensive in compute to develop the full history of Stockfish and NNUE and train the final SOTA checkpoints... But once you do, you can run it on even the tiny dumb chips like 'every smartphone' to get human-level chess performance on billions of nodes overnight. (Application to other domains of more real-world relevance than 'playing chess' left as an exercise for the reader.)
206 comments
[ 2.9 ms ] story [ 76.8 ms ] threadI thought about something the other day... Modern chess sites run proprietary algorithms to detect cheating by modern engines. But would it be possible for an old engine, let's say from the 70s or 80s, running on modern hardware to deceive these algorithms?
They wouldn't win every tournament but still have a respectable performance.
Also notice that for events like Title Tuesday you must play on a dedicated client with at least 2 cameras, so it would be significantly harder to pull it off.
Wow how are things down in imagination land.
Biology had a lot of time, attempts and pressure to make this work but even the most optimized lines (something like a magpie) need more complexity (almost a billion neurons) to get a fraction of superhuman intelligence.
Results with small artificial neural nets are very limited so far, too.
I think this is underselling it a bit. If you'd told this to the IBM engineers at the time I think they would have taken great interest in how the performance was achieved.
I had a Pentium Pro 200 back in the day, to run 3DSMax 1.0 on Windows NT 3.5.
Unfortunately, most humans see big|new|expensive hardware as great status signifiers, barriers to lesser folks joining the Cool Club, and optimal sources of sales commissions.
I don't buy this. I've seen GMs playing online while verbalizing their thought process (on twitch / youtube videos where they livestream their play), and they consider way more branches than I do, while also going deeper than I ever would on some branches. My top rating in lichess blitz is about 1900, not bad but way lower than a GM.
This is a known thing among chess players, and even low-skilled players do it to some extent. A newbie to chess can (and almost every teacher will say 'should') work their pattern recognition by just solving tactical chess puzzles and very quickly improve at spotting things like one or two move checkmate patterns or forks or pins or whatever.
It's more of a memory thing than subconscious calculation.
Would you make the move based on vibes alone in a non-blitz game? The calculation is still required and it is faster for those already familiar.
Sometimes it really looks like "calculations in the back of your head". Have you ever been struck with an 5 moves long tactics in positions you've never seen? Because I have.
There are hardly any other explanations than subconscious calculation.
[1] (2011) has a summary in the introduction:
As [1] is also a somewhat reproduction of the original study, you can take it directly. Section 3 Table 2 is probably most interesting, they had 2 board positions and 2 groups sectioned:Masters (2350+) | Intermediates (1800-2200)
(Mean values only, i tried inlcuding stddev but it was hard to read)
Positions Evaluated (A): 145.3 | 138.6
Positions Evaluated (B): 216.6 | 190.6
---
Max Depth (A): 11.7 | 9.4
Max Depth (B): 14.1 | 12.0
---
Mean Depth (A): 4.1 | 3.3
Mean Depth (B): 4.9 | 4.2
What i found interesting was that the "intermediate" group had a higher number of "base moves" they evaluated (eg Pos A: 5.6 | 8.6).
The previous studies (de Groot and another rperoduction from 1998) had "Nodes visited" for position A at 51.6 | 34.3 and 48.3 | 40.1 so similarly close together as the new ones, but much lower. (depending on your definition on "similarly close together")
[1] https://onlinelibrary.wiley.com/doi/10.1111/j.1551-6709.2011...
The only time when GM will definitely analyze more lines than an amateur is in a highly tactical position but that happens, again, maybe 1-3 times per typical game (tactics themselves will last several moves, but they would be largely calculated through at the beginning of the tactical melee).
The main thing commentators here are missing is that GMs have more confidence and do not repeat their analysis, but amateurs (strong club players included) typically re-analyze the same lines over and over. That alone would account for far stronger play with fewer moves considered, if you count the same move multiple times, which you should!
Also, Jonathan Rowson (Scottish GM - peak about 2600) describes his training matches with Mickey Adams (peak #3 or so in world - 2760), and says pretty much the same thing. Most of the time he asked “what about this line I rejected”, the answer was usually “I hadn’t considered that”.
Those critical moments are a big part of the difference between a GM and a good club player though. If a GM was somehow handicapped to not calculate such tactics, I think lesser players would be able to draw games against GMs much more often.
It's not one and the same, but strategy patterns are definitely a thing too.
There are:
- tactical patterns (e.g. "back rank checkmate", "pair of pieces that can be forked by a pawn");
- strategy patterns (e.g. "in a same-colour bishop endgame, put your pawns on the colour opposite to the bishops").
There's also calculation ability which also partly relies on patterns one has seen before.
1. Take a little extra time to visualize what will happen after you make a move. Will the piece be vulnerable? Does moving that piece leave a different piece vulnerable? I'd argue the biggest beginner mistake is not seeing that their move leaves a different piece vulnerable to being captured.
2. Don't move your queen into the board until after you move both bishops. This simple rule of thumb makes it far less likely you will lose your queen early, and it forces you to do a lot of other useful things (develop your pieces, control a portion of the board, etc).
Disclaimer: I am not good at chess, but when I played semi-regularly these helped me hold my own against other beginners.
Im not sure where the misunderstanding is.
The article says better players calculate a similar amount of moves and your anecdote shows the better player calculating a lot less.
So yeah it rebuts but the idea that GMs analyze more moves but it doesnt agree with the article, right? I guess just in the sense that its not more.
What does this mean exactly? They analyze the same number but the grandmaster analyzes "only the most critical moves."
So does that mean at every given analysis opportunity (each move I guess), they analyze better candidates?
If so I guess these 2 things are totally redundant? "they analyze only the most critical moves and do a better job with that analysis."
Alternatively I wondered if perhaps they actually perform analysis on less moves (perhaps because they already know the lines or remember their previous analysis). But I think that contradicts the claim that they analyze about the same number of moves.
This is intuitive and I guess I'm not surprised this is what was meant. They analyze the same amount of moves they are simply better at it (picking candidates, calculating, etc.)
I guess maybe im being a bit nitpicky but I feel like this is still a misleading way to say that.
> the average chess grandmaster during a classical game will analyze roughly the same number of moves as an amateur player, but they analyze only the most critical moves and do a better job with that analysis.
You could just say
> the average chess grandmaster during a classical game will analyze roughly the same number of moves as an amateur player, but they do a better job with that analysis.
"only the most critical moves" soudns like a limit on the number of moves based on "only the most" and by differentiating it from simply being higher quality.
I digress.
It's totally true that most people overestimate how much of GM's abilities comes from pure calculation, and underrate intuition. That's what it's getting at.
But in classical games GMs are, in fact, doing tons of calculation, and the statement is literally completely false.
Back in the day I remember Jon Finkel (a legendary Magic: the Gathering pro) saying* you're supposed to "focus only on what matters."
It stuck with me because I think it's genuinely what they do, but also seems like completely useless advice. I don't know what matters! That's what makes you a pro and me not.
* Wasn't sure if my memory was correct. A web search finds a Mike Flores article citing Zvi calling this Finkel's Law.
Notable because he departs from the book opening early on and has to find some brilliant moves to keep himself out of trouble. The breadth and depth of his calculations seem beyond what even strong amateurs consider, more than just choosing better lines to analyze.
It's not so easy, a lot of that better programming can be attributed to better hardware, it wouldn't have been possible to implement NNUE or Fishtest without modern hardware.
That's discounting architectural innovations & improvements which are bound to happen somehow.
And, with more work, that's true of computers in general.
For chess though, there are time constraints. The 386 is a lot slower. Is it fast enough?
An 80386 is about 10–13 Dhrystone MIPS, while a 200MHz Pentium Pro is about 90–450, according to https://netlib.org/performance/html/dhrystone.data.col0.html. So we're talking about a further slowdown on the order of 20×. I don't know how to estimate how much that would lower its Elo.
The article says:
> By making a number of adjustments to the algorithm and tuning it to the Pentium, we can increase its Elo to 2846 ± 35 at 1 second per move, 3109 ± 79 at 10 seconds per move, and 3180 ± 80 at 30 seconds per move.
So making it 30× faster would raise its 1-second Elo by about 334 points, and ln(30) is 3.4, so that's about 100 points per factor of e. ln(10) is 2.3, so we'd interpolate a prediction of 230 points gained at 10× faster, reasonably close to the 263 points reported.
If we extrapolate that in the other direction, ln(20) is 3.0, so on the 80386 we'd expect a loss of about 300 Elo points, down from 2846 to 2550 or so. Still enough to wipe the floor with Crafty, but low enough that Kasparov would usually beat it.
But we don't really know if that extrapolation is valid.
A common rough rule of thumb used throughout the 2010s (because this no longer holds today) in computer chess was that a 2x speedup gives about 50 Elo. This mostly holds in that region but at low levels it is more like 100-200 points.
Someone built Stockfish for the Pentium (586), 486 and 386 a while ago: https://talkchess.com/forum3/viewtopic.php?t=63857
https://github.com/zellyn/8fish
Maybe this was satirizing people running after every shiny new thing, but the MMX certainly wasn’t. The Pro was the first P6, a totally new, out-of-order microarchitecture that the Pentium 2 would be based on. The MMX was a standard P5 Pentium with a larger L1 and the MMX vector instruction set which, while the first of its kind in an x86, was ultimately not used much in applications, partly because it was integer-only. In any case, the Pros were expensive and meant for business use, while the MMXs were consumer processors.
The launch of Pentium Pro in 1995 was the moment when Intel segmented their CPU line into CPUs for consumers & server/workstation CPUs.
Because this move was a money maker for all involved, all memory producers and also AMD followed immediately the Intel initiative. Memory modules without ECC would have never existed if adequate consumer protection laws had existed.
I spent some time trying to figure out how some of the humans seemed to have trapped the machine out of the opening, because it looked like the human was winning.
Throw a 3dfx Voodoo in there and do some real-time 3D Gaussian Splatting!
A lot of support for old CPUs & platforms they're on goes through emulators like QEMU. Or plain old 'go with specs, disturb code as little as possible, and assume (hope) it still works'. But nothing beats running on actual hardware. Uncommon configurations, specific drivers that may have been broken (sometimes for a looong time before someone notices), spec says A but hardware does B, etc.
Back in 2021 I got cfish and weiss working on a 386sx, though didn't test it much. cfish was doing 5k nps so some results can be extrapolated. NNUE complicated things due to RAM requirements. Though I did train a small one for lichess at some point and they can be surprisingly good.
I wish disclaimers like that would be at the start and not halfway trough the text.
On a kinda similar note i always wander that about indie video games. We have this massive hits like balatro, vampire survivors, that easily could be made in the era of SNES or N64. But somehow we have to wait 30 years to get them.
It clearly wasn't technical side of things that prevented that, but also I don't believe it was lack of creativity.
Some could argue that it is just what art is, we will get every year some new thing, creating all "possible and meaningful" art right now isn't even valid thing to consider, because art isn't about exhausting a finite set of possibilities.
But at the other hand some mechanics, design solutions in the hindsight seem kinda obvious and make me wonder, what things we do not see now, but will look obvious to people 30 years from now.
> It clearly wasn't technical side of things that prevented that, but also I don't believe it was lack of creativity.
I don't know those specific games, but I think this often happens because, although the thing has been technically possible for decades, it used to be relatively difficult/expensive/obscure, and now it's far more accessible.
In those days, writing and publishing a SNES game was something that relatively very few people would have even thought to attempt, let alone had the knowledge and skills and capital to pull off successfully. Nowadays, creating and publishing a SNES-level game is comparatively very easy (at the technical level, that is; making a good original game is of course still hard), and so a large portion of the people with an idea and some motivation can actually go ahead and do it. Many of the resulting games won't be very interesting, but collectively we're exploring the design space much more quickly.
edit: I was probably silly to focus on literal SNES games instead of 'SNES era' games, but the point still stands: creating games for any platform back then was harder and more obscure than it is now (especially given that we're talking about a technical level that required quite efficient use of the hardware back then, but can be done with extremely inefficient use of the hardware now).
I don't know I fully agree with this. Consoles always had the "walled garden" thing going for them, but on the other side of things, developing games became so cheap and easy during the personal computer revolution that whole industry collapsed in '83 under a massive flood of shovelware...
Just cutting the high resolution textures might get you there.
Also, don't underestimate the "juice" that these days is relatively easy to do but would be expenseive back then. Balatro has dozens of very small things going on graphically that make it feel real nice to play that most certainly not be free on SNES hardware.
And like vampire survivors is just Robotron/Smash TV with infinity number of sprites on screen. The limitation is very much hardware in that case.
Every board game could have been built thousands of years ago, at least after the dawn of some reasonable writing/drawing media. But technology in design hadn't accumulated enough. To go farther, every bit of math could have been done thousands of years ago, but new math had to draw on earlier math. New math didn't rely (much) on special paper.
Also, "hit" probably isn't a good metric. Those games might have showed up back then and just been moderately notable, like dozens or hundreds of other games were.
Game design (like other creative expressions) is the same. Each new game doesn't come from a vacuum, it's a progression from everything that has come before.
Of course, “human vs supercomputer” is more exciting to the public than “human vs toaster controller”, especially when the latter in small quantities costs less than a cup of coffee.
I just ran Andrej Karpathy's Tiny Stories model on a tiny ESP32. In principle we could probably have done this in the 80's and it would have been so amazing.
ELO is like probability. A 100 ELO difference is something like 2x odds of winning a game and over a lot of tries it tends to be quite accurate. That was also an argument from Magnus Carlsen that to compare human ELO you need a lot of games and he didn't want to take part in classical world championship quoting this reason (among others).
Deep Blue is just used as a point of reference to illustrate that fact.
I see no reason to doubt the author here.
It's also great reference engine with all of the pinned tags of its evolution.
https://github.com/mhouppin/stash-bot
And I think that Fishtest is actually what made Stockfish the strongest, because no other engine's performance ever was evaluated on such scale before.
Stockfish was the first to use hard data instead of vibes, when selecting algorithms and heuristics.
So, even if we add a rule that NNUE is not allowed, because it was trained on modern GPUs that, obviously, didn't exist back in 1996, hand-coded heuristics would also take a ton of compute to develop.
Bitter lesson strikes again, even when you don't expect it.
[0] https://en.wikipedia.org/wiki/Stockfish_(chess)#Fishtest
[1] https://github.com/official-stockfish/fishtest/wiki
I am half remembering a post from more than a decade ago but a process similar to Fishtest was supposedly used for Rybka. But in general I agree.