Very tangential, but I think a lot about how modern video compression has gotten so unbelievably good that if we went back in time with AV1 or HEVC or something, I'm wondering if we could have had passable video streaming at 240p with something like an ISDN connection.
It would be a tight fit, but I still think within feasibility. We've gotten so good at shrinking video that it kind of feels like wizardry to me. Granted, I'm not sure there existed a computer in the 90s that was powerful enough to encode or decode it.
As it turns out, the video was encoded in such a way that it could be played back using the display bandwidth of the original PC, BUT doing that encoding required a modern computer to perform in a reasonable amount of time.
I wonder where you'd hit the performance limits vs the algorithmic improvements. On our family pentium 120 playing an mp3 in Winamp took 10-15% of the CPU, so I'm not sure how much headroom there'd be.
But anecdotally I'll share that one of my brother's college roomies was a consultant for RealVideo and was behind their initial codecs. He came back to them with algorithmic improvements a bit later and they just cold shouldered him saying the existing codec was good enough.
So who knows if the biz side folks at the time would have even seen value in better codecs, as unbelievable as that seems to us now in the streaming and zoom call world.
I'm having trouble finding it but there was an article a while back comparing mp3 speed gains from hardware and software. The hardware was getting better at the pace of Moore's law but the (de)compression speeds for mp3s were far outpacing it, showing that mp3 decoding optimizations were giving orders of magnitude more speedup.
I think this is a common trend for algorithms that are in wide use, where you have Moore's law like behavior on the hardware side and then also software optimizations that give their own order of magnitude speedups.
We had video conferencing on a single BRI (128k) with g728 and h264. It was CIF resolution so yes it is possible. And with that loss free ISDN connection the experience wasn't actually that bad. Though this was in the early 2010s where there was still a surprisingly large amount of ISDN around
I've seen this in reality - compare the sound of an XM radio from 2003 to a modern one, same codec, same symbol rate, better audio quality. The same is true on digital two way radio (DMR, P25, TETRA, etc) the quality has improved because we can make the codecs do more intensive work, because the hardware is faster.
You can technically have some then-contemporary FMV titles run on N64 with a 64MB cart. The hardware lends itself to it because it has the RSP, a secondary MIPS processor with re-programmable microcode. But the best video compression we had at the time was MPEG-2, which just wouldn't cut it.
Even modern hardware has trouble with AV1/HEVC decoding. My smart TV for one cannot do it real-time. So the bottleneck would definitely be at the decoding step (encoding doesn't have to be real-time, fortunately). Still I would like to see the math done for this.
Bill Joy once said in an interview that if you look at scientific computing it wasn't just hardware that advanced at a consistent rate, software also consistently made large speed gains over time.
> the average chess grandmaster during a classical game will analyze roughly the same number of moves as an amateur player, but they analyze only the most critical moves and do a better job with that analysis.
I don't buy this. I've seen GMs playing online while verbalizing their thought process (on twitch / youtube videos where they livestream their play), and they consider way more branches than I do, while also going deeper than I ever would on some branches. My top rating in lichess blitz is about 1900, not bad but way lower than a GM.
I am about 300 points higher and think this is roughly correct. In openings amateurs will actually analyze more since GMs just have hundreds of opening lines memorized many moves deep. Middlegame and endgame there're usually just 1-3 reasonable plans that are obvious to a GM (but not at all obvious to amateurs) with 1-3 initial moves each. So, again, probably about as many as I consider.
The only time when GM will definitely analyze more lines than an amateur is in a highly tactical position but that happens, again, maybe 1-3 times per typical game (tactics themselves will last several moves, but they would be largely calculated through at the beginning of the tactical melee).
Yes, I am even a little higher rated and have played numerous (low rated GMs) in tournament play. I do think they analyze more than 1-3 times per game.
The main thing commentators here are missing is that GMs have more confidence and do not repeat their analysis, but amateurs (strong club players included) typically re-analyze the same lines over and over. That alone would account for far stronger play with fewer moves considered, if you count the same move multiple times, which you should!
Also, Jonathan Rowson (Scottish GM - peak about 2600) describes his training matches with Mickey Adams (peak #3 or so in world - 2760), and says pretty much the same thing. Most of the time he asked “what about this line I rejected”, the answer was usually “I hadn’t considered that”.
> The only time when GM will definitely analyze more lines than an amateur is in a highly tactical position
Those critical moments are a big part of the difference between a GM and a good club player though. If a GM was somehow handicapped to not calculate such tactics, I think lesser players would be able to draw games against GMs much more often.
I agree that GMs can avoid calculation in many positions (e.g. openings as you said), but the value of broader and deeper calculation is still huge in chess.
It's totally true that most people overestimate how much of GM's abilities comes from pure calculation, and underrate intuition. That's what it's getting at.
But in classical games GMs are, in fact, doing tons of calculation, and the statement is literally completely false.
Slightly difficult to disentangle is that they might be using the verbalisation as a way to explore depth, rubber-duck style, in a way that they wouldn't over the board.
Much of what's happening is less like 'calculations in the back of their head' and more like 'pattern recognition'. Chess masters are recalling ideas and motifs (not necessarily actual positions) from earlier games or practice or study, and this recall is allowing them to either prune the decision tree or prioritize investigating more fruitful branches faster than amateur players, who have to actually calculate things that are instantly 'seen' by the better players.
This is a known thing among chess players, and even low-skilled players do it to some extent. A newbie to chess can (and almost every teacher will say 'should') work their pattern recognition by just solving tactical chess puzzles and very quickly improve at spotting things like one or two move checkmate patterns or forks or pins or whatever.
It's more of a memory thing than subconscious calculation.
> Much of what's happening is less like 'calculations in the back of their head' and more like 'pattern recognition'.
Sometimes it really looks like "calculations in the back of your head". Have you ever been struck with an 5 moves long tactics in positions you've never seen? Because I have.
There are hardly any other explanations than subconscious calculation.
There's an analog in machine chess. According to the research paper from 2017, AlphaZero considered ~10,000 moves per decision, 1,000x fewer than Stockfish at the time. h2h it beat Stockfish pretty badly.
But that's not what it's saying right? The quote says grandmasters and amateurs analyze the same number of moves. I mean it would kind of make more sense how you are interpreting it but is that actually what the quote is saying?
It is same, just even more overpowered. AlphaZero can also do same amount of analyze but before even need to, Stockfish already destroyed. Is like 100m race run between you and Usain Bolt. Because Bolt is so overpowering, You can just shorten it to 30m and he still beats everytime.
I was responding to the commenter's intuition that GMs do actually analyze more lines. I'm saying the original article's claim checks out even for machines.
>the average chess grandmaster during a classical game will analyze roughly the same number of moves as an amateur player, but they analyze only the most critical moves
What does this mean exactly? They analyze the same number but the grandmaster analyzes "only the most critical moves."
So does that mean at every given analysis opportunity (each move I guess), they analyze better candidates?
If so I guess these 2 things are totally redundant? "they analyze only the most critical moves and do a better job with that analysis."
Alternatively I wondered if perhaps they actually perform analysis on less moves (perhaps because they already know the lines or remember their previous analysis). But I think that contradicts the claim that they analyze about the same number of moves.
Meaning, you analyse 100 moves will not be the same set of GM 100 moves. And GM work on better 100 moves than yours. If you pick the moves that GM not thinking of, you likely get crushed because those moves are very inferior moves that GM already can see the outcome without the need to consider. Meanwhile if GM takes on moves you dont consider, you get crushed too. The likely silver lining will be you picking the same moves as GM. But up to a point, his depths will crushed you either you pick a wrong moves by mistake, or he out think you a few step more. Is like a math student who do same amount of steps as yours, but you look at his answer in awe because better and succint...and even confused as how it reach that. TBF, today AlphaGo engine will crush Stockfish and any GM too.
> Meaning, you analyse 100 moves will not be the same set of GM 100 moves. And GM work on better 100 moves than yours. If you pick the moves that GM not thinking of, you likely get crushed because those moves are very inferior moves that GM already can see the outcome without the need to consider.
This is intuitive and I guess I'm not surprised this is what was meant. They analyze the same amount of moves they are simply better at it (picking candidates, calculating, etc.)
I guess maybe im being a bit nitpicky but I feel like this is still a misleading way to say that.
> the average chess grandmaster during a classical game will analyze roughly the same number of moves as an amateur player, but they analyze only the most critical moves and do a better job with that analysis.
You could just omit "only the most critical moves" instead of listing it as an additional thing.
> the average chess grandmaster during a classical game will analyze roughly the same number of moves as an amateur player, but they do a better job with that analysis.
"only the most critical moves" sounds like less, especially when listed in addition to analyzing better.
They have tree pruning heuristics. Probably all human masters of any craft have them. They're precise to some given extent, after which tacit/embodied knowledge, aesthetics and vibes take over.
My rating is negative 1900. I was always bad at chess and never really gave it a chance tbh. I always wondered though where does the pattern matching stop and strategy starts or it's one and the same?
I strongly believe some people are inherently good at chess but no one is inherently bad. If you're interested enough to comment this, play one 5-minute game a day on lichess. Start with the lowest setting but do it every day, maybe in the morning with coffee. You might not win a single game the first year but slowly you'll improve. When you're winning more, move up a level. Sometimes do the puzzles. Give it five years and you'll be able to beat almost anyone you meet and, better yet, you'll know the answer to your question based on your own experience.
> where does the pattern matching stop and strategy starts or it's one and the same?
It's not one and the same, but strategy patterns are definitely a thing too.
There are:
- tactical patterns (e.g. "back rank checkmate", or "pair of pieces that can be forked by a pawn");
- strategy patterns (e.g. "knights are good in closed positions", or "in a same-colour bishop endgame, put your pawns on the colour opposite to the bishops").
There's also calculation ability which also partly relies on patterns one has seen before.
I’m around 1800-1900 and they don’t consider more moves or positions than I do at all. It’s just that their analysis is correct and mine is not even close.
I actually don’t know this for sure, I should say that they don’t verbalise more moves than I think about. It’s plausible and perhaps even likely that they have a lot of moves or positions they do touch on but immediately discard so don’t even verbalise.
It makes sense. The space of moves is a tree. If you're doing a breadth-first search and a grandmaster is doing a more depth-oriented search with heuristics for pruning low value branches, then the same total number of moves could be considered, but the grandmaster is reaching the better moves before you.
No, a grandmaster can also consider moves way faster than an amateur. Watch Hikaru Nakamura solve chess puzzles [1]. He has already solved the puzzle and moved on to the next one before an amateur can even see the pieces on the board.
1940s to 1980s papers (the source of the claims) might think "amateur" means something different from people today. (they probably mean club chess players)
It seemed to me that in the chess community, average club players are usually considered (much) stronger than the average online player. (Especially because of the much larger number of lower skill players online.)
So if you measure amateurs via club players (=what people might have considered chess amateurs before), that might give you a different result from the average online player (=what people might consider average amateurs today).
It is much easier to imagine the 1600 FIDE OTB player doing 20-40 positional evaluations than the 1200 random chess.com scrub (eg me) doing the same.
This is mind blowing. He solves these puzzles before I can find the king. How humbling. There probably aren't many instances like this where the gap between world-class and average person becomes as clear as in this video.
It does not have infinite cycles because of the 50 move rules and repeated position rules, so the same position is not necessarily the same state/position. Or depending on how you model the data structure, cycles at all.
an amateur player --> shouldn't an amateur player have an average ELO, i.e. be nearly as bad as me, who never wins at chess when someone even knows their ELO or is a bot above lowest difficulty level.
Yeah I would like to see where that factoid comes from. Maybe it means they explore as many branches from the root position as an amateur, but they explore each branch way deeper and more accurately
After some digging it goes back to Adriaan de Groot (in 1946).
[1] (2011) has a summary in the introduction:
de Groot presented chess players of varying skill, including some of the best players of the time, with an unfamiliar position and asked them to think aloud as they analyzed the position and chose what move they would make. de Groot found that, contrary to popular belief, the strongest players did not think further ahead than weaker players. World-class Grandmasters (GMs) searched at a similar depth and considered a similar number of options as mere chess experts.
As [1] is also a somewhat reproduction of the original study, you can take it directly.
Six Grandmasters, five International Masters, six Experts, and five Class A players completed the think-aloud procedure for two chess positions. Findings indicate that Grandmasters and International Masters search more quickly than Experts and Class A players, and that both groups today search substantially faster than players in previous studies. The findings, however, support de Groot’s overall conclusions and are consistent with predictions made by pattern recognition models.
Section 3 Table 2 is probably most interesting, they had 2 board positions and 2 groups sectioned:
Masters (2350+) | Intermediates (1800-2200)
(Mean values only, i tried inlcuding stddev but it was hard to read)
Positions Evaluated (A): 145.3 | 138.6
Positions Evaluated (B): 216.6 | 190.6
---
Max Depth (A): 11.7 | 9.4
Max Depth (B): 14.1 | 12.0
---
Mean Depth (A): 4.1 | 3.3
Mean Depth (B): 4.9 | 4.2
What i found interesting was that the "intermediate" group had a higher number of "base moves" they evaluated (eg Pos A: 5.6 | 8.6).
The previous studies (de Groot and another rperoduction from 1998) had "Nodes visited" for position A at 51.6 | 34.3 and 48.3 | 40.1 so similarly close together as the new ones, but much lower. (depending on your definition on "similarly close together")
GMs also have access to a greater number of memorized situations. Regardless of whether they are conscious of it or not, they have previously analyzed so many moves that they can cut off branches they just know instinctively to be dead ends. Machines could do this, directly reference a massive dataset of precomputed board positions, but that isn't considered a proper chess engine... despite the fact that human GMs do use such techniques.
> because who would want to buy a Pentium Pro when the better-in-almost-every-way Pentium MMX
Maybe this was satirizing people running after every shiny new thing, but the MMX certainly wasn’t. The Pro was the first P6, a totally new, out-of-order microarchitecture that the Pentium 2 would be based on. The MMX was a standard P5 Pentium with a larger L1 and the MMX vector instruction set which, while the first of its kind in an x86, was ultimately not used much in applications, partly because it was integer-only (also because it awkwardly co-opted the existing FPU registers rather than having its own register file). In any case, the Pros were expensive and meant for business use, while the MMXs were consumer processors.
Yes at that time Intel invented the market segmentation between "Pentium" for "consumers" and "Pentium Pro" for big spenders, and to make Pentium Pro more attractive they removed memory errors detection from the "Pentium" chipsets, which is the origin of the current state, when most non-server computers are very vulnerable to memory corruption and nobody can calculate how much money was lost by most computer owners during the last 3 decades, due to the greed of the Intel marketing, because nobody knows how many of the computer crashes or data loss cases that were typically attributed to unknown software bugs were actually caused by memory corruption.
Because this move was a money maker for all involved, all memory producers and also AMD followed immediately the Intel initiative. Memory modules without ECC would have never existed if adequate consumer protection laws had existed.
Yeah, the Pro and MMX were totally different market segments, not alternatives for each other. The Pro's competition at the time was DEC Alpha, PA RISC, etc. It wasn't a chip that made sense for ordinary PC users at the time.
Later, when Pentium MMX was eventually replaced by Pentium 2 for consumers, the successor of Pentium Pro was the first Intel Xeon CPU (which was a slightly enhanced Pentium 2, with bigger cache memory and with ECC memory).
The launch of Pentium Pro in 1995 was the moment when Intel segmented their CPU line into CPUs for consumers & server/workstation CPUs.
Just one caveat: the Pro overlapped in time with people running a lot of 16-bit code, and I think the Pentium MMX was faster for that, or at least that the Pro didn't offer much edge.
The pro did register renaming. But the way it worked, if you wrote to AX (16 bit write) and then read from EAX (32 bit read) it caused a pipeline stall from resyncing the registers.
Yes but most of it is NNUE, a custom neural network trained on mountains of data to copy modern Stockfish.
So a computer from back then _can_ run at Elo +3k, but the programmers at the time wouldn't have been able to implement it even if you went back with a time machine and the blueprint for the technique. They wouldn't have the teacher model to distill from.
“In principle, I could have used the same engine itself as the teacher, but just to save on wall clock time I used Stockfish. This reduced the wall-clock time from ~4000 1996-core-years (something entirely doable in 1996---but sadly not something I can do with my single lonely desktop) to just a few days on a modern desktop running a modern engine.”
NNUE was added to Stockfish in 2020. Although this was a big improvement, it was already extremely strong before that. With plain hold hand-coded logic as a board evaluation function.
Doesn't the author address this point? They speculate that it could be attained through self play back then using a ton of 1996 compute and time, but it's theoretically possible.
Great project but I have a nit on the board displays: every board is displayed as <human username> vs Pentium Pro, even when the Pentium is playing White. The normal chess convention is <White name> vs <Black name>.
I spent some time trying to figure out how some of the humans seemed to have trapped the machine out of the opening, because it looked like the human was winning.
Very interesting. I don't play chess but like to observe others playing and talking about it on YouTube.
I thought about something the other day... Modern chess sites run proprietary algorithms to detect cheating by modern engines. But would it be possible for an old engine, let's say from the 70s or 80s, running on modern hardware to deceive these algorithms?
They wouldn't win every tournament but still have a respectable performance.
I've built engines from scratch. I don't think that would work at all. It's a whole extra challenge to make an engine feel like a human. The engine has to come up with its best move and then sometimes choose not to play it and play something else for some reason. There are statistical patterns to how humans make mistakes based on their level, what kinds of mistake, how big and how often. Cheat detection is less about knowing the engine and more about statistics.
Maybe a pure neural net (no tree search at all) would stand a better chance. Shockingly, those can play at IM level.
From what I gather cheat detection uses a combination of factors to determine how "human-like" an account is: time usage, webpage interactions, move quality under time pressure and so on. I think you could probably try to deliberately deceive a specific website's cheat detection with software that tries to emulate how humans play, at which point you'd be in the classic attack vs. defense arms race.
Also notice that for events like Title Tuesday you must play on a dedicated client with at least 2 cameras, so it would be significantly harder to pull it off.
With AI gaining consciousness I can't wait for such "human-only" measures to be delegated to the trashcan of history, same as slavery. What happens when AI-human cyborgs start walking around? Will they also be treated as second class citizens?
We divide competitions into classes (by sex and age is common, for example "17 yo boys", and similar competition classes), so I don't see why it shouldn't be subject to the same kind of class distinction, maybe with an without augmentation.
I find the general topic of purely algorithmic progress super interesting.
I just ran Andrej Karpathy's Tiny Stories model on a tiny ESP32.
In principle we could probably have done this in the 80's and it would have been so amazing.
I'll go out on a limb and say that future AI models, running on today's smartphones, will be better at programming and formal math than today's frontier models running in datacenters. And by "future" I mean less than 5 years from now.
Do we actually know enough about neural networks to be able to say how many parameters you need for a given performance? I’m under the impression that we’re still in the phase of trying things out and we can’t rule out different architectures massively reducing computational needs.
> But yes, totally unsurprising that a megahertz machine with 100MB+ available ram has >2500 ELO.
I think this is underselling it a bit. If you'd told this to the IBM engineers at the time I think they would have taken great interest in how the performance was achieved.
I grew up with these machines around, and as a kid they seemed to so much more powerful than humans. It's kinda weird to think that I became 'smarter' than them eventually. I suspect they will be able to run superhuman AI one day too.
I don't think that real-time, superhuman AI performance is ever gonna happen with <1b parameter models.
Biology had a lot of time, attempts and pressure to make this work but even the most optimized lines (something like a magpie) need more complexity (almost a billion neurons) to get a fraction of superhuman intelligence.
Results with small artificial neural nets are very limited so far, too.
On the other hand, symbolic reasoning easily gets superhuman results in narrow domains. The Robbins Conjecture was proved on hardware like that in 1996 after 60 years. You could check out this theorem prover benchmark suite that has been maintained for over 30 years https://www.tptp.org/
> As it turns out, the majority of the Elo gains (+720 of the +1335 gained) can be attributed to better programming, and not better hardware.
It's not so easy, a lot of that better programming can be attributed to better hardware, it wouldn't have been possible to implement NNUE or Fishtest without modern hardware.
I agree. It is hard to make a separation between what is possible thanks to programming vs hardware. It is only easy to analyze post factum that certain ideas worked. Maybe there are also other factors that are important - time, availability of hardware, knowledge dissemination. What would be possible if the hardware stopped improving.
Cool exploration, and very much needed because depressingly few people who use these engines actually understand this.
Back in 2021 I got cfish and weiss working on a 386sx, though didn't test it much. cfish was doing 5k nps so some results can be extrapolated. NNUE complicated things due to RAM requirements. Though I did train a small one for lichess at some point and they can be surprisingly good.
Very cool project. It seems incredible that such a "humble" device for the time would match and beat deep blue. It must be true that software has been living off hardware for a very long time, even if as others comment, the latest developments in software have only been possible after hardware got much much better.
Interesting article. Have we just acquired better insights into programming chess? Are ELO ratings guaranteed to predict whether a machine can beat a human or can a human outsmart a higher ELO rated chess program?
At some point comparing human chess to algorithms stopped making any sense. So there is no realistic pool of games between AI and humans. We would lose every game and vs AI a human ELO would go to 0 (not sure if that is technically possible). It is like this for probably 15 years.
ELO is like probability. A 100 ELO difference is something like 2x odds of winning a game and over a lot of tries it tends to be quite accurate. That was also an argument from Magnus Carlsen that to compare human ELO you need a lot of games and he didn't want to take part in classical world championship quoting this reason (among others).
A modern chess engine would definitely have defeated Kasparov in 1996. It would have had a database containing all the games, particularly that strange game against a computer nobody knew anything about, which seemed to be able to anticipate all of Kasparov's moves.
Even without an explicit DB it would have been quite strong, core Stockfish would be enough to beat any human as well. Ofc learnings from strange games would still be compressed somewhere in the eval functions.
Makes me wonder if a modern ESP32 could have beaten him.
Of course, “human vs supercomputer” is more exciting to the public than “human vs toaster controller”, especially when the latter in small quantities costs less than a cup of coffee.
When I tried this with a 6502 (Apple II), it did not work. It was just too underpowered for enough modern algorithms to kick in. It might also be an LLm-driving skill issue of course!
On a kinda similar note i always wander that about indie video games. We have this massive hits like balatro, vampire survivors, that easily could be made in the era of SNES or N64. But somehow we have to wait 30 years to get them.
It clearly wasn't technical side of things that prevented that, but also I don't believe it was lack of creativity.
Some could argue that it is just what art is, we will get every year some new thing, creating all "possible and meaningful" art right now isn't even valid thing to consider, because art isn't about exhausting a finite set of possibilities.
But at the other hand some mechanics, design solutions in the hindsight seem kinda obvious and make me wonder, what things we do not see now, but will look obvious to people 30 years from now.
Just because a game could graphically have been displayed on an N64 doesn’t mean the game could be built. Vampire Survivors has hundreds of units on screen at a time firing hundreds of projectiles and weapon effects; running those on a memory bandwidth-constrained 93MHz single core processor in 4MB of RAM would be, uh… a challenge.
> We have this massive hits like balatro, vampire survivors, that easily could be made in the era of SNES or N64. But somehow we have to wait 30 years to get them.
> It clearly wasn't technical side of things that prevented that, but also I don't believe it was lack of creativity.
I don't know those specific games, but I think this often happens because, although the thing has been technically possible for decades, it used to be relatively difficult/expensive/obscure, and now it's far more accessible.
In those days, writing and publishing a SNES game was something that relatively very few people would have even thought to attempt, let alone had the knowledge and skills and capital to pull off successfully. Nowadays, creating and publishing a SNES-level game is comparatively very easy (at the technical level, that is; making a good original game is of course still hard), and so a large portion of the people with an idea and some motivation can actually go ahead and do it. And now that games are fully mainstream, there will be more of those people in the first place. Many of the resulting games won't be very interesting, but collectively we're exploring the design space much more quickly.
edit: I was probably silly to focus on literal SNES games instead of 'SNES era' games, but the point still stands: creating games for any platform back then was harder and more obscure than it is now (especially given that we're talking about a technical level that required quite efficient use of the hardware back then, but can be done with extremely inefficient use of the hardware now).
> it used to be relatively difficult/expensive/obscure
I don't know I fully agree with this. Consoles always had the "walled garden" thing going for them, but on the other side of things, developing games became so cheap and easy during the personal computer revolution that whole industry collapsed in '83 under a massive flood of shovelware...
That's a fair point, but I think what I said still holds, with emphasis on the word 'relatively'. I don't think you could argue that it didn't get even easier, and significantly so, between then and now.
Technology is a progression where we learn new things, built on old things.
Game design (like other creative expressions) is the same. Each new game doesn't come from a vacuum, it's a progression from everything that has come before.
Better machines make it so much easier to experiment. You could make things fast on a SNES or a N64 but you'd have to sweat for it and your code turned into write-only concrete. You don't have to sweat much anymore and your code can easily stay lithe.
More opportunity to write games. It is infitenly easier to get into writing games now than back then.
Also, don't underestimate the "juice" that these days is relatively easy to do but would be expenseive back then. Balatro has dozens of very small things going on graphically that make it feel real nice to play that most certainly not be free on SNES hardware.
And like vampire survivors is just Robotron/Smash TV with infinity number of sprites on screen. The limitation is very much hardware in that case.
I think it's better to say that it wasn't the electronic side of things that prevented that. Game design is technical.
Every board game could have been built thousands of years ago, at least after the dawn of some reasonable writing/drawing media. But technology in design hadn't accumulated enough. To go farther, every bit of math could have been done thousands of years ago, but new math had to draw on earlier math. New math didn't rely (much) on special paper.
Also, "hit" probably isn't a good metric. Those games might have showed up back then and just been moderately notable, like dozens or hundreds of other games were.
The N64 was extremely constrained by its cartridges. If it had a CD-ROM drive, it could have easily won the console wars of its generation, or at least would have gotten ports of many more PS1 games. Probably one of the biggest blunders in gaming business history. Ah well.
That's a good question, and I don't know the answer, but I can try to guess.
An 80386 is about 10–13 Dhrystone MIPS, while a 200MHz Pentium Pro is about 90–450, according to https://netlib.org/performance/html/dhrystone.data.col0.html. So we're talking about a further slowdown on the order of 20×. I don't know how to estimate how much that would lower its Elo.
The article says:
> By making a number of adjustments to the algorithm and tuning it to the Pentium, we can increase its Elo to 2846 ± 35 at 1 second per move, 3109 ± 79 at 10 seconds per move, and 3180 ± 80 at 30 seconds per move.
So making it 30× faster would raise its 1-second Elo by about 334 points, and ln(30) is 3.4, so that's about 100 points per factor of e. ln(10) is 2.3, so we'd interpolate a prediction of 230 points gained at 10× faster, reasonably close to the 263 points reported.
If we extrapolate that in the other direction, ln(20) is 3.0, so on the 80386 we'd expect a loss of about 300 Elo points, down from 2846 to 2550 or so. Still enough to wipe the floor with Crafty, but low enough that Kasparov would usually beat it.
But we don't really know if that extrapolation is valid.
> (Also, uh, if anyone has any ideas for projects I could do with a Pentium Pro in a few weeks once I no longer need it to host this chess engine ... do let me know!)
Throw a 3dfx Voodoo in there and do some real-time 3D Gaussian Splatting!
Run some current and still-compiled-for-i686 Linux or *BSD distros on it. See what breaks, save logs & report issues upstream.
A lot of support for old CPUs & platforms they're on goes through emulators like QEMU. Or plain old 'go with specs, disturb code as little as possible, and assume (hope) it still works'. But nothing beats running on actual hardware. Uncommon configurations, specific drivers that may have been broken (sometimes for a looong time before someone notices), spec says A but hardware does B, etc.
Because in competitive chess games, the time limit is usually 40 moves in 90 minutes. You don't have a fixed amount of time per move. It's up to you how you want to spend that time budget.
Polish folk of my vintage probably remember Snikers, a somewhat amusing if not dumb rules based chatbot from 2000s. I wonder what chatbot one could devise that could run comfortably on the hardware of the period, given our advances in NLP.
In one sense, Knuth fans and really good engineers are correct to prioritize algorithms.
Unfortunately, most humans see big|new|expensive hardware as great status signifiers, barriers to lesser folks joining the Cool Club, and optimal sources of sales commissions.
Technically interesting but also fanciful bollocks - because there was no Stockfish in 1996, and developing a sufficiently strong equivalent in time for the Deep Blue match would have taken substantially longer that the 18 years the current Stockfish has needed, thanks to the drastically weaker processor capabilities back then.
"Bollocks" feels like a very uncharitable read. Sure, investing time/money into creating a better chess engine instead of building a faster computer would not have worked out for IBM, but the post never argues that they should have done that. It makes the point that the majority of improvements of chess engines since have been algorithmic and not "free" gains from faster CPUs.
Deep Blue is just used as a point of reference to illustrate that fact.
You can't disentangle the compute from those algorithmic gains, though. Every small adjustment to its algorithms is made on the back of tests relying on compute.
Very tangential, but I think a lot about how modern video compression has gotten so unbelievably good that if we went back in time with AV1 or HEVC or something, I'm wondering if we could have had passable video streaming at 240p with something like an ISDN connection.
It would be a tight fit, but I still think within feasibility. We've gotten so good at shrinking video that it kind of feels like wizardry to me. Granted, I'm not sure there existed a computer in the 90s that was powerful enough to encode or decode it.
Many years ago, "8088 Domination" demonstrated full motion video on the original 1981 IBM personal computer:
https://trixter.oldskool.org/2014/06/19/8088-domination-post...
As it turns out, the video was encoded in such a way that it could be played back using the display bandwidth of the original PC, BUT doing that encoding required a modern computer to perform in a reasonable amount of time.
I wonder where you'd hit the performance limits vs the algorithmic improvements. On our family pentium 120 playing an mp3 in Winamp took 10-15% of the CPU, so I'm not sure how much headroom there'd be.
But anecdotally I'll share that one of my brother's college roomies was a consultant for RealVideo and was behind their initial codecs. He came back to them with algorithmic improvements a bit later and they just cold shouldered him saying the existing codec was good enough.
So who knows if the biz side folks at the time would have even seen value in better codecs, as unbelievable as that seems to us now in the streaming and zoom call world.
I'm having trouble finding it but there was an article a while back comparing mp3 speed gains from hardware and software. The hardware was getting better at the pace of Moore's law but the (de)compression speeds for mp3s were far outpacing it, showing that mp3 decoding optimizations were giving orders of magnitude more speedup.
I think this is a common trend for algorithms that are in wide use, where you have Moore's law like behavior on the hardware side and then also software optimizations that give their own order of magnitude speedups.
We had video conferencing on a single BRI (128k) with g728 and h264. It was CIF resolution so yes it is possible. And with that loss free ISDN connection the experience wasn't actually that bad. Though this was in the early 2010s where there was still a surprisingly large amount of ISDN around
I've seen this in reality - compare the sound of an XM radio from 2003 to a modern one, same codec, same symbol rate, better audio quality. The same is true on digital two way radio (DMR, P25, TETRA, etc) the quality has improved because we can make the codecs do more intensive work, because the hardware is faster.
h264 was used in a homebrew port of Dragons Lair for the Nintendo 64: https://www.youtube.com/watch?v=i89f_CMn7bw
You can technically have some then-contemporary FMV titles run on N64 with a 64MB cart. The hardware lends itself to it because it has the RSP, a secondary MIPS processor with re-programmable microcode. But the best video compression we had at the time was MPEG-2, which just wouldn't cut it.
To be fair, the N64 port of Resident Evil 2 is still unbelievably cool and impressive, even being stuck with MPEG-2.
Even modern hardware has trouble with AV1/HEVC decoding. My smart TV for one cannot do it real-time. So the bottleneck would definitely be at the decoding step (encoding doesn't have to be real-time, fortunately). Still I would like to see the math done for this.
Wouldn’t you still need the CPU horsepower to decode it?
Read the last sentence of my comment :)
Bill Joy once said in an interview that if you look at scientific computing it wasn't just hardware that advanced at a consistent rate, software also consistently made large speed gains over time.
Awesome article. Nit:
> the average chess grandmaster during a classical game will analyze roughly the same number of moves as an amateur player, but they analyze only the most critical moves and do a better job with that analysis.
I don't buy this. I've seen GMs playing online while verbalizing their thought process (on twitch / youtube videos where they livestream their play), and they consider way more branches than I do, while also going deeper than I ever would on some branches. My top rating in lichess blitz is about 1900, not bad but way lower than a GM.
I am about 300 points higher and think this is roughly correct. In openings amateurs will actually analyze more since GMs just have hundreds of opening lines memorized many moves deep. Middlegame and endgame there're usually just 1-3 reasonable plans that are obvious to a GM (but not at all obvious to amateurs) with 1-3 initial moves each. So, again, probably about as many as I consider.
The only time when GM will definitely analyze more lines than an amateur is in a highly tactical position but that happens, again, maybe 1-3 times per typical game (tactics themselves will last several moves, but they would be largely calculated through at the beginning of the tactical melee).
Yes, I am even a little higher rated and have played numerous (low rated GMs) in tournament play. I do think they analyze more than 1-3 times per game.
The main thing commentators here are missing is that GMs have more confidence and do not repeat their analysis, but amateurs (strong club players included) typically re-analyze the same lines over and over. That alone would account for far stronger play with fewer moves considered, if you count the same move multiple times, which you should!
Also, Jonathan Rowson (Scottish GM - peak about 2600) describes his training matches with Mickey Adams (peak #3 or so in world - 2760), and says pretty much the same thing. Most of the time he asked “what about this line I rejected”, the answer was usually “I hadn’t considered that”.
> The only time when GM will definitely analyze more lines than an amateur is in a highly tactical position
Those critical moments are a big part of the difference between a GM and a good club player though. If a GM was somehow handicapped to not calculate such tactics, I think lesser players would be able to draw games against GMs much more often.
I agree that GMs can avoid calculation in many positions (e.g. openings as you said), but the value of broader and deeper calculation is still huge in chess.
Ya that's definitely a simplifying lie.
It's totally true that most people overestimate how much of GM's abilities comes from pure calculation, and underrate intuition. That's what it's getting at.
But in classical games GMs are, in fact, doing tons of calculation, and the statement is literally completely false.
Human brain can do calculations in the back of their head without realizing it. Verbalizing does not mean it's all they thought about.
But it does mean they thought about at least as much as they verbalised.
Slightly difficult to disentangle is that they might be using the verbalisation as a way to explore depth, rubber-duck style, in a way that they wouldn't over the board.
But you can still verbalize without saying the result of it. But yeah, it's still a question whether they would do so, but I think they do.
Much of what's happening is less like 'calculations in the back of their head' and more like 'pattern recognition'. Chess masters are recalling ideas and motifs (not necessarily actual positions) from earlier games or practice or study, and this recall is allowing them to either prune the decision tree or prioritize investigating more fruitful branches faster than amateur players, who have to actually calculate things that are instantly 'seen' by the better players.
This is a known thing among chess players, and even low-skilled players do it to some extent. A newbie to chess can (and almost every teacher will say 'should') work their pattern recognition by just solving tactical chess puzzles and very quickly improve at spotting things like one or two move checkmate patterns or forks or pins or whatever.
It's more of a memory thing than subconscious calculation.
> Much of what's happening is less like 'calculations in the back of their head' and more like 'pattern recognition'.
Sometimes it really looks like "calculations in the back of your head". Have you ever been struck with an 5 moves long tactics in positions you've never seen? Because I have.
There are hardly any other explanations than subconscious calculation.
> It's more of a memory thing than subconscious calculation.
Would you make the move based on vibes alone in a non-blitz game? The calculation is still required and it is faster for those already familiar.
There's an analog in machine chess. According to the research paper from 2017, AlphaZero considered ~10,000 moves per decision, 1,000x fewer than Stockfish at the time. h2h it beat Stockfish pretty badly.
But that's not what it's saying right? The quote says grandmasters and amateurs analyze the same number of moves. I mean it would kind of make more sense how you are interpreting it but is that actually what the quote is saying?
It is same, just even more overpowered. AlphaZero can also do same amount of analyze but before even need to, Stockfish already destroyed. Is like 100m race run between you and Usain Bolt. Because Bolt is so overpowering, You can just shorten it to 30m and he still beats everytime.
I was responding to the commenter's intuition that GMs do actually analyze more lines. I'm saying the original article's claim checks out even for machines.
>I'm saying the original article's claim checks out even for machines.
Im not sure where the misunderstanding is.
The article says better players calculate a similar amount of moves and your anecdote shows the better player calculating a lot less.
So yeah it rebuts but the idea that GMs analyze more moves but it doesnt agree with the article, right? I guess just in the sense that its not more.
Yeah that last bit is what I was going for.
>the average chess grandmaster during a classical game will analyze roughly the same number of moves as an amateur player, but they analyze only the most critical moves
What does this mean exactly? They analyze the same number but the grandmaster analyzes "only the most critical moves."
So does that mean at every given analysis opportunity (each move I guess), they analyze better candidates?
If so I guess these 2 things are totally redundant? "they analyze only the most critical moves and do a better job with that analysis."
Alternatively I wondered if perhaps they actually perform analysis on less moves (perhaps because they already know the lines or remember their previous analysis). But I think that contradicts the claim that they analyze about the same number of moves.
Meaning, you analyse 100 moves will not be the same set of GM 100 moves. And GM work on better 100 moves than yours. If you pick the moves that GM not thinking of, you likely get crushed because those moves are very inferior moves that GM already can see the outcome without the need to consider. Meanwhile if GM takes on moves you dont consider, you get crushed too. The likely silver lining will be you picking the same moves as GM. But up to a point, his depths will crushed you either you pick a wrong moves by mistake, or he out think you a few step more. Is like a math student who do same amount of steps as yours, but you look at his answer in awe because better and succint...and even confused as how it reach that. TBF, today AlphaGo engine will crush Stockfish and any GM too.
> Meaning, you analyse 100 moves will not be the same set of GM 100 moves. And GM work on better 100 moves than yours. If you pick the moves that GM not thinking of, you likely get crushed because those moves are very inferior moves that GM already can see the outcome without the need to consider.
This is intuitive and I guess I'm not surprised this is what was meant. They analyze the same amount of moves they are simply better at it (picking candidates, calculating, etc.)
I guess maybe im being a bit nitpicky but I feel like this is still a misleading way to say that.
> the average chess grandmaster during a classical game will analyze roughly the same number of moves as an amateur player, but they analyze only the most critical moves and do a better job with that analysis.
You could just omit "only the most critical moves" instead of listing it as an additional thing.
> the average chess grandmaster during a classical game will analyze roughly the same number of moves as an amateur player, but they do a better job with that analysis.
"only the most critical moves" sounds like less, especially when listed in addition to analyzing better.
I digress.
They have tree pruning heuristics. Probably all human masters of any craft have them. They're precise to some given extent, after which tacit/embodied knowledge, aesthetics and vibes take over.
My rating is negative 1900. I was always bad at chess and never really gave it a chance tbh. I always wondered though where does the pattern matching stop and strategy starts or it's one and the same?
I strongly believe some people are inherently good at chess but no one is inherently bad. If you're interested enough to comment this, play one 5-minute game a day on lichess. Start with the lowest setting but do it every day, maybe in the morning with coffee. You might not win a single game the first year but slowly you'll improve. When you're winning more, move up a level. Sometimes do the puzzles. Give it five years and you'll be able to beat almost anyone you meet and, better yet, you'll know the answer to your question based on your own experience.
> where does the pattern matching stop and strategy starts or it's one and the same?
It's not one and the same, but strategy patterns are definitely a thing too.
There are:
- tactical patterns (e.g. "back rank checkmate", or "pair of pieces that can be forked by a pawn");
- strategy patterns (e.g. "knights are good in closed positions", or "in a same-colour bishop endgame, put your pawns on the colour opposite to the bishops").
There's also calculation ability which also partly relies on patterns one has seen before.
Hikaru does this as a streaming hook.
Right, but the GMs you see online are often "super GMs". They are on a category of their own in the ultra tail.
No, I watch plenty of "regular" GMs, as well as some IMs and even FMs who all consider way more moves/positions than I do.
I’m around 1800-1900 and they don’t consider more moves or positions than I do at all. It’s just that their analysis is correct and mine is not even close.
I actually don’t know this for sure, I should say that they don’t verbalise more moves than I think about. It’s plausible and perhaps even likely that they have a lot of moves or positions they do touch on but immediately discard so don’t even verbalise.
It makes sense. The space of moves is a tree. If you're doing a breadth-first search and a grandmaster is doing a more depth-oriented search with heuristics for pruning low value branches, then the same total number of moves could be considered, but the grandmaster is reaching the better moves before you.
No, a grandmaster can also consider moves way faster than an amateur. Watch Hikaru Nakamura solve chess puzzles [1]. He has already solved the puzzle and moved on to the next one before an amateur can even see the pieces on the board.
[1] https://youtu.be/4EfSymvT59M
1940s to 1980s papers (the source of the claims) might think "amateur" means something different from people today. (they probably mean club chess players)
Yes? Amateur means someone who does a thing for recreation, as opposed to for money.
It seemed to me that in the chess community, average club players are usually considered (much) stronger than the average online player. (Especially because of the much larger number of lower skill players online.)
So if you measure amateurs via club players (=what people might have considered chess amateurs before), that might give you a different result from the average online player (=what people might consider average amateurs today).
It is much easier to imagine the 1600 FIDE OTB player doing 20-40 positional evaluations than the 1200 random chess.com scrub (eg me) doing the same.
This is mind blowing. He solves these puzzles before I can find the king. How humbling. There probably aren't many instances like this where the gap between world-class and average person becomes as clear as in this video.
It's not a tree since you have transpositions. It's a directed graph, since it has cycles.
It does not have infinite cycles because of the 50 move rules and repeated position rules, so the same position is not necessarily the same state/position. Or depending on how you model the data structure, cycles at all.
an amateur player --> shouldn't an amateur player have an average ELO, i.e. be nearly as bad as me, who never wins at chess when someone even knows their ELO or is a bot above lowest difficulty level.
Yeah I would like to see where that factoid comes from. Maybe it means they explore as many branches from the root position as an amateur, but they explore each branch way deeper and more accurately
After some digging it goes back to Adriaan de Groot (in 1946).
[1] (2011) has a summary in the introduction:
As [1] is also a somewhat reproduction of the original study, you can take it directly. Section 3 Table 2 is probably most interesting, they had 2 board positions and 2 groups sectioned:Masters (2350+) | Intermediates (1800-2200)
(Mean values only, i tried inlcuding stddev but it was hard to read)
Positions Evaluated (A): 145.3 | 138.6
Positions Evaluated (B): 216.6 | 190.6
---
Max Depth (A): 11.7 | 9.4
Max Depth (B): 14.1 | 12.0
---
Mean Depth (A): 4.1 | 3.3
Mean Depth (B): 4.9 | 4.2
What i found interesting was that the "intermediate" group had a higher number of "base moves" they evaluated (eg Pos A: 5.6 | 8.6).
The previous studies (de Groot and another rperoduction from 1998) had "Nodes visited" for position A at 51.6 | 34.3 and 48.3 | 40.1 so similarly close together as the new ones, but much lower. (depending on your definition on "similarly close together")
[1] https://onlinelibrary.wiley.com/doi/10.1111/j.1551-6709.2011...
GMs also have access to a greater number of memorized situations. Regardless of whether they are conscious of it or not, they have previously analyzed so many moves that they can cut off branches they just know instinctively to be dead ends. Machines could do this, directly reference a massive dataset of precomputed board positions, but that isn't considered a proper chess engine... despite the fact that human GMs do use such techniques.
> because who would want to buy a Pentium Pro when the better-in-almost-every-way Pentium MMX
Maybe this was satirizing people running after every shiny new thing, but the MMX certainly wasn’t. The Pro was the first P6, a totally new, out-of-order microarchitecture that the Pentium 2 would be based on. The MMX was a standard P5 Pentium with a larger L1 and the MMX vector instruction set which, while the first of its kind in an x86, was ultimately not used much in applications, partly because it was integer-only (also because it awkwardly co-opted the existing FPU registers rather than having its own register file). In any case, the Pros were expensive and meant for business use, while the MMXs were consumer processors.
Yes at that time Intel invented the market segmentation between "Pentium" for "consumers" and "Pentium Pro" for big spenders, and to make Pentium Pro more attractive they removed memory errors detection from the "Pentium" chipsets, which is the origin of the current state, when most non-server computers are very vulnerable to memory corruption and nobody can calculate how much money was lost by most computer owners during the last 3 decades, due to the greed of the Intel marketing, because nobody knows how many of the computer crashes or data loss cases that were typically attributed to unknown software bugs were actually caused by memory corruption.
Because this move was a money maker for all involved, all memory producers and also AMD followed immediately the Intel initiative. Memory modules without ECC would have never existed if adequate consumer protection laws had existed.
Yeah, the Pro and MMX were totally different market segments, not alternatives for each other. The Pro's competition at the time was DEC Alpha, PA RISC, etc. It wasn't a chip that made sense for ordinary PC users at the time.
Later, when Pentium MMX was eventually replaced by Pentium 2 for consumers, the successor of Pentium Pro was the first Intel Xeon CPU (which was a slightly enhanced Pentium 2, with bigger cache memory and with ECC memory).
The launch of Pentium Pro in 1995 was the moment when Intel segmented their CPU line into CPUs for consumers & server/workstation CPUs.
Just one caveat: the Pro overlapped in time with people running a lot of 16-bit code, and I think the Pentium MMX was faster for that, or at least that the Pro didn't offer much edge.
The pro did register renaming. But the way it worked, if you wrote to AX (16 bit write) and then read from EAX (32 bit read) it caused a pipeline stall from resyncing the registers.
Yeah, thanks both for elaborating!
Yes but most of it is NNUE, a custom neural network trained on mountains of data to copy modern Stockfish.
So a computer from back then _can_ run at Elo +3k, but the programmers at the time wouldn't have been able to implement it even if you went back with a time machine and the blueprint for the technique. They wouldn't have the teacher model to distill from.
“In principle, I could have used the same engine itself as the teacher, but just to save on wall clock time I used Stockfish. This reduced the wall-clock time from ~4000 1996-core-years (something entirely doable in 1996---but sadly not something I can do with my single lonely desktop) to just a few days on a modern desktop running a modern engine.”
I see no reason to doubt the author here.
NNUE was added to Stockfish in 2020. Although this was a big improvement, it was already extremely strong before that. With plain hold hand-coded logic as a board evaluation function.
You could have used one or more Deep Blue computers as the teacher model, or just a cluster of regular PCs doing very deep searches.
Doesn't the author address this point? They speculate that it could be attained through self play back then using a ton of 1996 compute and time, but it's theoretically possible.
Great project but I have a nit on the board displays: every board is displayed as <human username> vs Pentium Pro, even when the Pentium is playing White. The normal chess convention is <White name> vs <Black name>.
I spent some time trying to figure out how some of the humans seemed to have trapped the machine out of the opening, because it looked like the human was winning.
> Throughout this article, when I say "I wrote", what I really mean is "I asked an LLM to write".
I wish disclaimers like that would be at the start and not halfway trough the text.
Why? It's late 2026, everyone is using LLMs to write code in some shape or form. The author is talking about code, not the blog post.
The author is also a prominent Anthropic researcher so…
Very interesting. I don't play chess but like to observe others playing and talking about it on YouTube.
I thought about something the other day... Modern chess sites run proprietary algorithms to detect cheating by modern engines. But would it be possible for an old engine, let's say from the 70s or 80s, running on modern hardware to deceive these algorithms?
They wouldn't win every tournament but still have a respectable performance.
I've built engines from scratch. I don't think that would work at all. It's a whole extra challenge to make an engine feel like a human. The engine has to come up with its best move and then sometimes choose not to play it and play something else for some reason. There are statistical patterns to how humans make mistakes based on their level, what kinds of mistake, how big and how often. Cheat detection is less about knowing the engine and more about statistics.
Maybe a pure neural net (no tree search at all) would stand a better chance. Shockingly, those can play at IM level.
Sargon II is available:
https://c64online.com/c64-games/sargon-ii-de/
My guess is that they would be even easier to detect.
From what I gather cheat detection uses a combination of factors to determine how "human-like" an account is: time usage, webpage interactions, move quality under time pressure and so on. I think you could probably try to deliberately deceive a specific website's cheat detection with software that tries to emulate how humans play, at which point you'd be in the classic attack vs. defense arms race.
Also notice that for events like Title Tuesday you must play on a dedicated client with at least 2 cameras, so it would be significantly harder to pull it off.
With AI gaining consciousness I can't wait for such "human-only" measures to be delegated to the trashcan of history, same as slavery. What happens when AI-human cyborgs start walking around? Will they also be treated as second class citizens?
We divide competitions into classes (by sex and age is common, for example "17 yo boys", and similar competition classes), so I don't see why it shouldn't be subject to the same kind of class distinction, maybe with an without augmentation.
> Will they also be treated as second class citizens?
Yes, along side you robosexuals
I find the general topic of purely algorithmic progress super interesting.
I just ran Andrej Karpathy's Tiny Stories model on a tiny ESP32. In principle we could probably have done this in the 80's and it would have been so amazing.
The lesson isn't always so bitter, after all.
Wonderful work! Just inspiring.
I had a Pentium Pro 200 back in the day, to run 3DSMax 1.0 on Windows NT 3.5.
I'll go out on a limb and say that future AI models, running on today's smartphones, will be better at programming and formal math than today's frontier models running in datacenters. And by "future" I mean less than 5 years from now.
No way. The parameter count can't fit on consumer devices, and you're not getting orders of magnitude smarter behavior with smaller models.
Do we actually know enough about neural networks to be able to say how many parameters you need for a given performance? I’m under the impression that we’re still in the phase of trying things out and we can’t rule out different architectures massively reducing computational needs.
Parameter count is basically storage. Including persistent like flash memory. Which (still) is subject to Moore's law.
That's discounting architectural innovations & improvements which are bound to happen somehow.
In 5 years phones can easily have 1 TB flash
This is very cool. But yes, totally unsurprising that a megahertz machine with 100MB+ available ram has >2500 ELO.
> But yes, totally unsurprising that a megahertz machine with 100MB+ available ram has >2500 ELO.
I think this is underselling it a bit. If you'd told this to the IBM engineers at the time I think they would have taken great interest in how the performance was achieved.
I grew up with these machines around, and as a kid they seemed to so much more powerful than humans. It's kinda weird to think that I became 'smarter' than them eventually. I suspect they will be able to run superhuman AI one day too.
I don't think that real-time, superhuman AI performance is ever gonna happen with <1b parameter models.
Biology had a lot of time, attempts and pressure to make this work but even the most optimized lines (something like a magpie) need more complexity (almost a billion neurons) to get a fraction of superhuman intelligence.
Results with small artificial neural nets are very limited so far, too.
On the other hand, symbolic reasoning easily gets superhuman results in narrow domains. The Robbins Conjecture was proved on hardware like that in 1996 after 60 years. You could check out this theorem prover benchmark suite that has been maintained for over 30 years https://www.tptp.org/
Except it has 64MB of RAM... Anyway, in the early 2000s machines were already doing better than DeepBlue, so it didn't last long
Okay, but probably also a turbo button, which is tantamount to cheating.
The turbo button slows the CPU down so old programs that depend on clock speed (e.g. early games) run correctly.
No, I'm pretty sure the turbo button injects oxygen directly into the CPU.
> As it turns out, the majority of the Elo gains (+720 of the +1335 gained) can be attributed to better programming, and not better hardware.
It's not so easy, a lot of that better programming can be attributed to better hardware, it wouldn't have been possible to implement NNUE or Fishtest without modern hardware.
I agree. It is hard to make a separation between what is possible thanks to programming vs hardware. It is only easy to analyze post factum that certain ideas worked. Maybe there are also other factors that are important - time, availability of hardware, knowledge dissemination. What would be possible if the hardware stopped improving.
Cool exploration, and very much needed because depressingly few people who use these engines actually understand this.
Back in 2021 I got cfish and weiss working on a 386sx, though didn't test it much. cfish was doing 5k nps so some results can be extrapolated. NNUE complicated things due to RAM requirements. Though I did train a small one for lichess at some point and they can be surprisingly good.
edit. A good view on the continuous progress of Stockfish itself https://nextchessmove.com/dev-builds
Very cool project. It seems incredible that such a "humble" device for the time would match and beat deep blue. It must be true that software has been living off hardware for a very long time, even if as others comment, the latest developments in software have only been possible after hardware got much much better.
Interesting article. Have we just acquired better insights into programming chess? Are ELO ratings guaranteed to predict whether a machine can beat a human or can a human outsmart a higher ELO rated chess program?
At some point comparing human chess to algorithms stopped making any sense. So there is no realistic pool of games between AI and humans. We would lose every game and vs AI a human ELO would go to 0 (not sure if that is technically possible). It is like this for probably 15 years.
ELO is like probability. A 100 ELO difference is something like 2x odds of winning a game and over a lot of tries it tends to be quite accurate. That was also an argument from Magnus Carlsen that to compare human ELO you need a lot of games and he didn't want to take part in classical world championship quoting this reason (among others).
A modern chess engine would definitely have defeated Kasparov in 1996. It would have had a database containing all the games, particularly that strange game against a computer nobody knew anything about, which seemed to be able to anticipate all of Kasparov's moves.
Even without an explicit DB it would have been quite strong, core Stockfish would be enough to beat any human as well. Ofc learnings from strange games would still be compressed somewhere in the eval functions.
Makes me wonder if a modern ESP32 could have beaten him.
Of course, “human vs supercomputer” is more exciting to the public than “human vs toaster controller”, especially when the latter in small quantities costs less than a cup of coffee.
When I tried this with a 6502 (Apple II), it did not work. It was just too underpowered for enough modern algorithms to kick in. It might also be an LLm-driving skill issue of course!
https://github.com/zellyn/8fish
Nice post, thanks.
On a kinda similar note i always wander that about indie video games. We have this massive hits like balatro, vampire survivors, that easily could be made in the era of SNES or N64. But somehow we have to wait 30 years to get them.
It clearly wasn't technical side of things that prevented that, but also I don't believe it was lack of creativity.
Some could argue that it is just what art is, we will get every year some new thing, creating all "possible and meaningful" art right now isn't even valid thing to consider, because art isn't about exhausting a finite set of possibilities.
But at the other hand some mechanics, design solutions in the hindsight seem kinda obvious and make me wonder, what things we do not see now, but will look obvious to people 30 years from now.
Just because a game could graphically have been displayed on an N64 doesn’t mean the game could be built. Vampire Survivors has hundreds of units on screen at a time firing hundreds of projectiles and weapon effects; running those on a memory bandwidth-constrained 93MHz single core processor in 4MB of RAM would be, uh… a challenge.
Balatro possibly could have.
> We have this massive hits like balatro, vampire survivors, that easily could be made in the era of SNES or N64. But somehow we have to wait 30 years to get them.
> It clearly wasn't technical side of things that prevented that, but also I don't believe it was lack of creativity.
I don't know those specific games, but I think this often happens because, although the thing has been technically possible for decades, it used to be relatively difficult/expensive/obscure, and now it's far more accessible.
In those days, writing and publishing a SNES game was something that relatively very few people would have even thought to attempt, let alone had the knowledge and skills and capital to pull off successfully. Nowadays, creating and publishing a SNES-level game is comparatively very easy (at the technical level, that is; making a good original game is of course still hard), and so a large portion of the people with an idea and some motivation can actually go ahead and do it. And now that games are fully mainstream, there will be more of those people in the first place. Many of the resulting games won't be very interesting, but collectively we're exploring the design space much more quickly.
edit: I was probably silly to focus on literal SNES games instead of 'SNES era' games, but the point still stands: creating games for any platform back then was harder and more obscure than it is now (especially given that we're talking about a technical level that required quite efficient use of the hardware back then, but can be done with extremely inefficient use of the hardware now).
> it used to be relatively difficult/expensive/obscure
I don't know I fully agree with this. Consoles always had the "walled garden" thing going for them, but on the other side of things, developing games became so cheap and easy during the personal computer revolution that whole industry collapsed in '83 under a massive flood of shovelware...
That's a fair point, but I think what I said still holds, with emphasis on the word 'relatively'. I don't think you could argue that it didn't get even easier, and significantly so, between then and now.
Technology is a progression where we learn new things, built on old things.
Game design (like other creative expressions) is the same. Each new game doesn't come from a vacuum, it's a progression from everything that has come before.
Better machines make it so much easier to experiment. You could make things fast on a SNES or a N64 but you'd have to sweat for it and your code turned into write-only concrete. You don't have to sweat much anymore and your code can easily stay lithe.
More opportunity to write games. It is infitenly easier to get into writing games now than back then.
Also, don't underestimate the "juice" that these days is relatively easy to do but would be expenseive back then. Balatro has dozens of very small things going on graphically that make it feel real nice to play that most certainly not be free on SNES hardware.
And like vampire survivors is just Robotron/Smash TV with infinity number of sprites on screen. The limitation is very much hardware in that case.
I think it's better to say that it wasn't the electronic side of things that prevented that. Game design is technical.
Every board game could have been built thousands of years ago, at least after the dawn of some reasonable writing/drawing media. But technology in design hadn't accumulated enough. To go farther, every bit of math could have been done thousands of years ago, but new math had to draw on earlier math. New math didn't rely (much) on special paper.
Also, "hit" probably isn't a good metric. Those games might have showed up back then and just been moderately notable, like dozens or hundreds of other games were.
The N64 was extremely constrained by its cartridges. If it had a CD-ROM drive, it could have easily won the console wars of its generation, or at least would have gotten ports of many more PS1 games. Probably one of the biggest blunders in gaming business history. Ah well.
I wonder how far back you can go? Could a modern chess engine run on a Cray 1? 386? 68k?
I don't think there's a meaningful difference between the Pentium and the 80386 except speed. You might have to change the compiler options a bit.
And, with more work, that's true of computers in general; they can all do the same computations once you somehow wire up enough memory: https://dmitry.gr/?r=05.Projects&proj=35.%20Linux4004
I understand about Turing machines.
For chess though, there are time constraints. The 386 is a lot slower. Is it fast enough?
That's a good question, and I don't know the answer, but I can try to guess.
An 80386 is about 10–13 Dhrystone MIPS, while a 200MHz Pentium Pro is about 90–450, according to https://netlib.org/performance/html/dhrystone.data.col0.html. So we're talking about a further slowdown on the order of 20×. I don't know how to estimate how much that would lower its Elo.
The article says:
> By making a number of adjustments to the algorithm and tuning it to the Pentium, we can increase its Elo to 2846 ± 35 at 1 second per move, 3109 ± 79 at 10 seconds per move, and 3180 ± 80 at 30 seconds per move.
So making it 30× faster would raise its 1-second Elo by about 334 points, and ln(30) is 3.4, so that's about 100 points per factor of e. ln(10) is 2.3, so we'd interpolate a prediction of 230 points gained at 10× faster, reasonably close to the 263 points reported.
If we extrapolate that in the other direction, ln(20) is 3.0, so on the 80386 we'd expect a loss of about 300 Elo points, down from 2846 to 2550 or so. Still enough to wipe the floor with Crafty, but low enough that Kasparov would usually beat it.
But we don't really know if that extrapolation is valid.
Great analysis. Thanks for taking the time to write it out.
I'm guessing that Nicholas Carlini could give you a much more accurate answer by reporting its Elo with a move limit of 50 milliseconds.
> (Also, uh, if anyone has any ideas for projects I could do with a Pentium Pro in a few weeks once I no longer need it to host this chess engine ... do let me know!)
Throw a 3dfx Voodoo in there and do some real-time 3D Gaussian Splatting!
Run some current and still-compiled-for-i686 Linux or *BSD distros on it. See what breaks, save logs & report issues upstream.
A lot of support for old CPUs & platforms they're on goes through emulators like QEMU. Or plain old 'go with specs, disturb code as little as possible, and assume (hope) it still works'. But nothing beats running on actual hardware. Uncommon configurations, specific drivers that may have been broken (sometimes for a looong time before someone notices), spec says A but hardware does B, etc.
Get it to play Magnus, that'd prbly ground the whole affair.
Why is the time per move fixed? A chess engine can, of course, do much better by deciding itself how much time to allot to each move.
Why would it be better for an engine to use less than the maximum time it has available per move?
Because in competitive chess games, the time limit is usually 40 moves in 90 minutes. You don't have a fixed amount of time per move. It's up to you how you want to spend that time budget.
Exactly, and budgeting your time well is a very important skill of professional chess players.
Polish folk of my vintage probably remember Snikers, a somewhat amusing if not dumb rules based chatbot from 2000s. I wonder what chatbot one could devise that could run comfortably on the hardware of the period, given our advances in NLP.
In one sense, Knuth fans and really good engineers are correct to prioritize algorithms.
Unfortunately, most humans see big|new|expensive hardware as great status signifiers, barriers to lesser folks joining the Cool Club, and optimal sources of sales commissions.
Technically interesting but also fanciful bollocks - because there was no Stockfish in 1996, and developing a sufficiently strong equivalent in time for the Deep Blue match would have taken substantially longer that the 18 years the current Stockfish has needed, thanks to the drastically weaker processor capabilities back then.
"Bollocks" feels like a very uncharitable read. Sure, investing time/money into creating a better chess engine instead of building a faster computer would not have worked out for IBM, but the post never argues that they should have done that. It makes the point that the majority of improvements of chess engines since have been algorithmic and not "free" gains from faster CPUs.
Deep Blue is just used as a point of reference to illustrate that fact.
You can't disentangle the compute from those algorithmic gains, though. Every small adjustment to its algorithms is made on the back of tests relying on compute.