I've refreshed StarSkirmish Bench, a benchmark where LLMs play StarCraft BW by writing code, with results from the flagship models of OpenAI, Anthropic, Google, and Muse
I noticed Pluto is running on a simple CPU? If so, that's very nice for future RTS games. You just ship (or use a server-client arch) a bunch of models of various difficulty levels to let players practice. Could we even get models tuned to specific campaign levels? I wonder how costly the creation of Pluto was.
At least when I first heard about trying to make AI play StarCraft against humans (not the built-in AI of course, which the pros wouldn't have any trouble with), there was a lot of talk about limiting their "actions per minute" to the same level as high-level human players. What confused me was the premise that someone hitting 200 APM would be doing 200 useful things per minute; as far as I was ever able to observe, a huge portion of those actions were either just to look at different things by hitting a hotkey for a unit or building in a different area or even just keeping their fingers in motion so that when they did need to do something it could flow from the movements they were already doing rather than having to actually think through starting and stopping.
People have been arguing about how useful apm actually is since roughly 5 minutes after people started measuring it.
That being said, there is a substantial difference between humans playing at anywhere from 50-500 apm and a robot that can literally micro every single zergling on the map at the same time and do things like avoid ever taking the splash damage portion of a siege tank shot.
Fair enough, that's a pretty stark difference. It sounds like the exact precision of the actions that a bot can do is pretty much beyond human abilities.
The top players were losing a bunch to Pluto when it first came on the scene because they didn’t realize it was a human and were playing normal strategies against it. Once they realized it was an AI bot they developed strategies to exploit it and can now win very reliably against it.
Pluto is very good at assessing relative army strengths based on units within its vision and will always engage when it judges that its army strength has an advantage, and it uses superior micro and targeting to win battles efficiently. The key weakness of Pluto is that it has no memory of units that are not in its line of sight, so it can be easily exploited with fast skirmishing and harassing units, drops, and spreading out artillery units to deny it from seeing the full army strength all at once.
Its other big weakness is that once you get an army advantage you can force it into a turtling position where it will never try to attack you and you can safely colonize the entire map with expansion bases, giving yourself an insurmountable snowballing resource advantage.
With respect, the wins you're describing are exploits that the next iteration of bots will rub out. The "AI forget the Reaver exists", "AI doesn't understand how dark swarm works", "AI doesn't know what nukes are" are bugs that can be corrected, not big gaps in reasoning.
So the reliable wins rely on exploits, not superior gameplay. The pros I've seen fight it without exploits, lose pretty bad. just in the last few hours: https://www.youtube.com/watch?v=xga636MBOns
This is an interesting hypothesis but I'm not sure it's proven.
Obviously chess bots can beat anyons but this involves a situation where you're making 1 move at a time in a highly constrained environment: exactly 64 squares and a set number of pieces and so in. You also have, relatively speaking, near infinite time to plan your next move and as a result AIs can use a combination of databases and simulation algorithms to derive the best next move.
Almost none of that applies to starcraft.
We can talk about "teaching" the bot to handle other situations but that word covers a gigantic realm of complexity, like how do we encode this information in a way that the ai can actually draw from to apply to its current situation? If people talk about "more chess positions than atoms in the universe" or whatever, this is so many orders of magnitude smaller than the number of valid starcraft positions that we need to invent new numeric notation to accurately describe it.
StarCraft is in between, where faster actions can be a big advantage but choosing good enough actions for that to matter is hard enough that it took many years to achieve.
It's beyond insane that Astra achieves above 50%. Are models allowed to "play against" the specific bot they're up against? What explains the fact that it drops to single-digits against strong bots?
How do they avoid the LLMs simply working with pre-training data? Which apparently they do, since the human written bots win. But it's a mystery, is what it is.
Better. I don’t see the human bots in the v0.2 bench but in v0.1 they didn’t beat stardust, a human written bot, which in turn doesn’t beat the best human players
Is there anything interesting to learn from the agent written bots? Or are they just inferior versions of the best bots because the time limit is so short? Is there any indication that they’re trained on the code from the human written bots (like, do they magically “guess” creating functionality that requires experimental feedback or domain experience?) How much are the bots on rails applying the same techniques every time?
I think those kinds of questions would be more interesting than the raw win/loss ratios.
none of these games really require a lot of experimental feedback, the goals are well stated and known; experiments may drive optimization if thats the question.
as far as the on rails comment : there's no good way to know the provenance of an LLM output, but the LLM made bots generally look like very small, modular, sometimes functionalized versions of the human equivalents; the difference being that the LLM is calling the funtions often rather than a conditional loop in a main().
a factorio bot I worked on when Astra was first getting previewed was essentially a directory of very small argument-receiving functional and self-supporting programs that had very limited scope, and an agents.md file that directed the LLM to where such capabilities existed to preempt a directory search every compaction/context-swap.
so you'd have things like "python walk.py N 40" for 40 units of northern travel. a watchdog that waited for HP to drop or movement to become inhibited, etc.
My very first experience with LLM driven bot development in the GPT3 days created huge unmaintainmable monoliths that kind of worked due to TDD but were essentially spaghetti. LLMs have been trending away from those paradigms into small self-encapsulated programs. probably because those are less abusive to ctx count and produce better results since they lose/compact less.
I would disagree that creating a StarCraft: BW bot doesn't require experimental feedback. The way a human author typically improves their bot is by having it play opponents and A/B test tactics and strategies.
There's also a pretty high profile StarCraft: BW bot right now called Pluto trained exclusively by self play RL (AlphaStar style).
Strong agree. Even if you understand StarCraft well and are an excellent programmer, and anticipate and correctly handle every situational edge case, it is hard to know the dynamics of how your behaviors will pan out without experimentation.
Jay Scott put it well: Adding features lowers your winrate. Only iterating on them increases it.
Dan Gant (author of Purple Wave) has written about some of this.
I did a 48 hour test with GPT-6 Astra and Claude Opus 5.5 and we saw some interesting uses of Psionic Storm.
I've open sourced some of GPT-Astra's work (below). It actually did an interesting amount of changes in strategy over time as it's opponents became more difficult.
To echo this, this is where Astra begins to stand out as a larger model and Opus begins to show certain cracks. I find Astra more adaptive, even as Opus is more capable in most coding/execution/tool-use domains. Fable-5.1 would be interesting here as it would presumably have a similar "large model" expansive reasoning and adaptation.
Anecdotally, in game benchmarks I've run, OpenAI models seem particularly adept at finding the meta game and exploiting it...
The unsexy perspective on StarCraft is that it's a game about logistics. You need to keep making things, not get supply blocked, fit all your buildings, not get your units trapped, and then bring all your units to a fight at a good time and at the same time.
Astra and Opus both cleared this bar pretty quickly, both settling on one-base Dragoon + Observer play, taking expansions pretty late, on 3 Gateways. This strategy is fine against everything and great against nothing, letting you win against anyone that you can out-execute. StarSkirmish gives agents access to the FastApproximation combat simulation library, which makes better combat predictions than the B-tier bots do, and let their bots win by trading armies better against them.
But the A-tier bots (Locutus and BananaBrain) are already strong at execution, and Astra wasn't making progress against them. It switched to a proxy 2-gate Zealot rush. This worked well on a map pool with mostly 2-player maps (so you can place the Gateways very close to your opponent). But it's also an astute choice for a new bot because it doesn't require you to be good at anything in StarCraft that happens after the first four minutes!
It's also smart as a variance-maximizing strategy. In many games, if your opponent is superior, you want the game to be decided by as few events as possible. Because with a bad coin, you're more likely to win 1 of 2 flips than 10 of 20. Proxy 2-gate shortens the game and the number of interactions required to win.
It's also advantageous given the format, where you get unlimited tries to win a series against each tier's opponents. If you can make the games a coin flip, and then take more coin flips, you can clear a tier even with an overall poor winrate. I do think this aspect of StarSkirmish is worth addressing in the future.
Astra's proxy gates beat the A-tier bots, and it moved up to StarSkirmish's S-tier (Stardust and PurpleWave, though really I'd consider BananaBrain completely our peer in ability).
A brief aside about bot tournaments: The AIIDE conference and historically the COG tournaments had every bot play hundreds of games against one another, and ranked then on overall winrate. I have never liked this because it does not resemble human StarCraft (where players play a best-of-N series), unbalances the races (Zerg vs. Zerg is more volatile and winrates should be closer to 50%), and creates some twisted incentives. One of those incentives is ensuring you beat weaker opponents 100% of the time instead of merely 95%, which means defaulting to reliable-but-weaker strategies to beat weaker opponents, and having a large bag of available strategies to find something which exploits trickier opponents.
And in this iteration of StarSkirmish, where benchmark bots are mostly unable to store data between games, Stardust ran one basic conservative strategy almost every game. This weakens Stardust, but meant that Astra's proxy gates were unlikely to win. So Astra had to do something very uncomfortable for a coding agent: Go back to the drawing board and try something completely different.
Astra completely switched gears, landing on a strategy that PurpleWave also likes to use against Stardust: Use Dark Templar to secure an expansion. This is particularly effective against Stardust, who does not use Shuttles + Reavers, and thus does not gain utility from the Robotics Facility it requires to make Observers.
From there Astra progressed slowly, but discovered a shared weakness of PurpleWave and Stardust: Both of us are designed to dodge Psionic Storms, but that feature was broken in both our bots simultaneously (for unrelated reasons!).
So Astra's Dark Templar opening transitioned smoothly into a two-base timing attack with Dragoons and High Templar. It moved out ...
I am not sure I like this benchmark, but mostly, because I am not sure what it is really evaluating. It is not an idle dismissal.. multiple factors seem to complicate what it is supposed to look at ( just top of my head ):
- code generation speed
- logic analysis
- understanding of the game platform and how use of C may influence it
- considered approaches, tactics and strategies
Don't get me wrong. It is is neat idea. I am just not sure what it measures.
All of the above, to the extent that they can be measured from a simple to verify end condition (stats, win rates, etc). Adaptability, ability to understand and progress on a task, and so on.
Yeah giving a 1 hour limit for code generation makes the whole thing a bit incoherent. Plus, performance is just going to depend on how closely the models have memorized the best open-source Starcraft bots and can reproduce them.
> Plus, performance is just going to depend on how closely the models have memorized the best open-source Starcraft bots and can reproduce them.
If that's the case then these models are failing extremely hard. 7% win rate against human written AIs that they should be copying from memory? A copy paste command should do 50%!
Also: given that the human written bots are almost certainly in the training data, the winning rate against human-written bot seems to be not well-calibrated. You would expect at least they are 50% or better, given that they can "memorize" the best human-written bot.
Worth digging into the protocol and see what went wrong.
Top bots are in the ballpark of 50kloc. Even large LLMs are not so large that they can memorize all the world's data like that. And all the fine-tuning that compression loses comprises most of the work in the first place.
Wow this is interesting to me from another angle. A friend of mine created a benchmark to evaluate LLMs on 9x9 Go (using KataGo opponents as Elo anchors) [1]. The relative rankings of this (StarSkirmish Bench v0.2) benchmark (GPT-6 Astra > GPT-6.1 Sol > Gemini 4 Argon > Claude Opus 5.5 > Claude Fable 5.1 > Claude Sonnet 5.5) matches the order of his GoBench rankings exactly, even though they are different problems at surface level.
Unsure, or as LLMs say, I’d gently push back on that. In my opinion, the jaggedness has increased. While this type of capability has improved, I still see longer running Astra agents making the assumptions that any employable person at any level wouldn't make. Also, generally, all models feel more robotic and less pleasant to talk to (subjective) while still lacking judgement.
So I have a hunch about this: I think it makes sense that they become less pleasant as they become more capable, because LLMs inherently do not have feelings, or any faculties/organs that could generate them. The pleasant tone was just an artificially encouraged artifact of the fact that LLMs approximate reasoning probabilistically, with mimicry. Of course, if you mimic things thoroughly enough, with sufficient granularity, you start to approximate the real thing and become robot-like, but maybe then there's also less room for more pleasant disclosure that was the result of less rigid mimicry.
So it almost seems like a situation where you cannot really have your cake and eat it, at least not in the way it currently works.
Vals IOI, olympiad programming problems solved in C++, and WeirdML v3, unusual machine learning coding tasks also mirror StarSkirmish Bench.
I did a StarCraft: BW hillclimb experiment with GPT-6 Astra and Claude 5.5 Opus. Peter Gostev did a Chess hillclimb saw similar behavior as I did with GPT-6 Astra.
Might want to reach out to the author of the above benchmark. I think having a working group of multiple distinct benchmark owners might help quickly evaluate new model developments
Is this meaningful? Aren't frontier models MoE anyway? If you issue a StarCraft query they will probably just route to their StarCraft optimized model.
MoE is a misleading term, there is no "starcraft expert". There's not even a python expert. MoE just means a subset of blocks of weights are activated per query, nothing more. In so far as they might have legible functions they're like "balancing parens"
MoE doesn't work that way. It's not about increasing performance of a task, but speeding it up. To get a faster model, you need to make it smaller. If you make it smaller, the less it can remember. MoE tries to get faster responses without reducing the capabilities by running a smaller part of the model, and constantly swapping the parts used, so that the whole model gets to influence the output collectively, but each token is created by a smaller set of weights.
Anyone else feels like they can tell when the underlying model is MoE? Dense models are more concise in their answers. MoE seems to stall for time, writing vaugely until a right set of weights gets picked and commits to an answer. MoE also backtracks a whole lot more.
Maybe because Protosses have been doing bad lately and need some help improving :) (won only 3 out of 22 ASL championships in the past 10 years)
On a more serious note: all three races play quite differently so it makes sense to focus on just one. And protoss is mechanically the easiest and most straightforward race: fewer but stronger units, a single worker can be used to build multiple buildings at once, versatile basic static defence
I can tell you why NOT Zerg. Zerg you have to do too many actions. Select the hatchery, select larvae, choose which unit to build. That extra select larvae step is a problem.
Even the most powerful frontier models on SOTA hardware are not fast enough for starcraft, which generally involves multiple informed decisions per second at peak moments.
You can train a specialized model for this, but a frontier model is ill-fitting.
If you are talking about take a screen shot, analyze move the window a bit, take another screenshot, move the window, repeat until they have seen enough of the map to make a decision, then decide what to click on, issue order.... this could absolutely be done. It's very similar to how LLM's have driven a car on a safe track.
But right now the response time would be so slow that the bot would not be close to competitive except with the most low level players. Especially because its hard to click on things that are moving at the same time as well.
Sometime soon, I'd like to see an LLM be competitive using only computer vision, mouse movement, and locked into a human possible number of actions per minute.
Doesn’t this suggest that the results seen in the OP are merely copying strategies it has seen in other bots (code in the training data), and not actually the agent developing its strategy?
Basically yes, but the space of "AI for RTS and videogames" likely goes well beyond Starcraft, so there's probably a lot of unique programming techniques these frontier models can apply for writing AI bots that haven't been done yet.
I think we want to move beyond code based bots asking the LLMs to try to train a custom neural net, in a limited time + compute, to play StarCraft BW is probably the most promising direction
I am considering an experiment where the task is to try to win by adversarially targeting Pluto (a high profile self play based RL model)
I assume lots of bot code is part of the training data? So it's basically a version of how will they can improve upon those solutions public out there?
This kind of makes me think about making better bots for Counter Strike based on human top level players, which i think its a very undervalued category of software. You can even forget about about CS and name any FPS and you'll be hard pressed to find a game with seriously good bots.
I think you are thinking about insta-killing machine bots set to detect a head through a wall and shoot it. I am talking about properly trained bots that while playing pretty good are also making mistakes and taking actions that may or maybe not suit the team strategy. A bot with the randomness of a human.
This is somewhat interesting to me as I’ve recently come to a different conclusion. I’ve been populating a ModernUO server with bots, and have found Opus 5.5 to be substantially better at understanding and building it. I actually started with 6.1 Sol and 6 Astra but pulled them after a week of little progress, had Opus 5.5 take over and it rebuilt everything properly in a night and has carried it forward quite a way since.
> ... runs each model in its lab's 1st party CLI harness at the highest reasoning setting it offers.
Anthropic (Claude models): Claude Code
OpenAI (GPT models): Codex
Google (Gemini models): Antigravity CLI
Meta (Muse models): Muse Code
> The CLIs' web search and web fetch tools are turned off, and the agent has no other internet access. The CLI is installed before the clock starts, and the hour counts from when the agent starts. There is no submit tool: the harness collects the bot when the time runs out.
One thing we're learning is that harness is HUGE. The difference in performance for the same underlying model can change dramatically across:
1. harnesses
2. allowed tools (no web search/fetch - how does the model refer to documentation?)
Once again, a page with a Pareto curve that is not a Pareto curve. Claude Haiku 5.5 is not on the curve, but inside. Seems like people don't really know what a Pareto curve is.
100 comments
[ 1.8 ms ] story [ 33.8 ms ] threadThat being said, there is a substantial difference between humans playing at anywhere from 50-500 apm and a robot that can literally micro every single zergling on the map at the same time and do things like avoid ever taking the splash damage portion of a siege tank shot.
Pluto is very good at assessing relative army strengths based on units within its vision and will always engage when it judges that its army strength has an advantage, and it uses superior micro and targeting to win battles efficiently. The key weakness of Pluto is that it has no memory of units that are not in its line of sight, so it can be easily exploited with fast skirmishing and harassing units, drops, and spreading out artillery units to deny it from seeing the full army strength all at once.
Its other big weakness is that once you get an army advantage you can force it into a turtling position where it will never try to attack you and you can safely colonize the entire map with expansion bases, giving yourself an insurmountable snowballing resource advantage.
So the reliable wins rely on exploits, not superior gameplay. The pros I've seen fight it without exploits, lose pretty bad. just in the last few hours: https://www.youtube.com/watch?v=xga636MBOns
Obviously chess bots can beat anyons but this involves a situation where you're making 1 move at a time in a highly constrained environment: exactly 64 squares and a set number of pieces and so in. You also have, relatively speaking, near infinite time to plan your next move and as a result AIs can use a combination of databases and simulation algorithms to derive the best next move.
Almost none of that applies to starcraft.
We can talk about "teaching" the bot to handle other situations but that word covers a gigantic realm of complexity, like how do we encode this information in a way that the ai can actually draw from to apply to its current situation? If people talk about "more chess positions than atoms in the universe" or whatever, this is so many orders of magnitude smaller than the number of valid starcraft positions that we need to invent new numeric notation to accurately describe it.
It would be interesting to see the reverse, how long does each model take to get >50% WR against bots of various tiers.
For Brood War they're at https://www.basil-ladder.net/
I think those kinds of questions would be more interesting than the raw win/loss ratios.
as far as the on rails comment : there's no good way to know the provenance of an LLM output, but the LLM made bots generally look like very small, modular, sometimes functionalized versions of the human equivalents; the difference being that the LLM is calling the funtions often rather than a conditional loop in a main().
a factorio bot I worked on when Astra was first getting previewed was essentially a directory of very small argument-receiving functional and self-supporting programs that had very limited scope, and an agents.md file that directed the LLM to where such capabilities existed to preempt a directory search every compaction/context-swap.
so you'd have things like "python walk.py N 40" for 40 units of northern travel. a watchdog that waited for HP to drop or movement to become inhibited, etc.
My very first experience with LLM driven bot development in the GPT3 days created huge unmaintainmable monoliths that kind of worked due to TDD but were essentially spaghetti. LLMs have been trending away from those paradigms into small self-encapsulated programs. probably because those are less abusive to ctx count and produce better results since they lose/compact less.
There's also a pretty high profile StarCraft: BW bot right now called Pluto trained exclusively by self play RL (AlphaStar style).
Jay Scott put it well: Adding features lowers your winrate. Only iterating on them increases it.
I did a 48 hour test with GPT-6 Astra and Claude Opus 5.5 and we saw some interesting uses of Psionic Storm.
I've open sourced some of GPT-Astra's work (below). It actually did an interesting amount of changes in strategy over time as it's opponents became more difficult.
https://github.com/llmskirmish/starskirmish-hillclimb
Anecdotally, in game benchmarks I've run, OpenAI models seem particularly adept at finding the meta game and exploiting it...
The unsexy perspective on StarCraft is that it's a game about logistics. You need to keep making things, not get supply blocked, fit all your buildings, not get your units trapped, and then bring all your units to a fight at a good time and at the same time.
Astra and Opus both cleared this bar pretty quickly, both settling on one-base Dragoon + Observer play, taking expansions pretty late, on 3 Gateways. This strategy is fine against everything and great against nothing, letting you win against anyone that you can out-execute. StarSkirmish gives agents access to the FastApproximation combat simulation library, which makes better combat predictions than the B-tier bots do, and let their bots win by trading armies better against them.
But the A-tier bots (Locutus and BananaBrain) are already strong at execution, and Astra wasn't making progress against them. It switched to a proxy 2-gate Zealot rush. This worked well on a map pool with mostly 2-player maps (so you can place the Gateways very close to your opponent). But it's also an astute choice for a new bot because it doesn't require you to be good at anything in StarCraft that happens after the first four minutes!
It's also smart as a variance-maximizing strategy. In many games, if your opponent is superior, you want the game to be decided by as few events as possible. Because with a bad coin, you're more likely to win 1 of 2 flips than 10 of 20. Proxy 2-gate shortens the game and the number of interactions required to win.
It's also advantageous given the format, where you get unlimited tries to win a series against each tier's opponents. If you can make the games a coin flip, and then take more coin flips, you can clear a tier even with an overall poor winrate. I do think this aspect of StarSkirmish is worth addressing in the future.
Astra's proxy gates beat the A-tier bots, and it moved up to StarSkirmish's S-tier (Stardust and PurpleWave, though really I'd consider BananaBrain completely our peer in ability).
A brief aside about bot tournaments: The AIIDE conference and historically the COG tournaments had every bot play hundreds of games against one another, and ranked then on overall winrate. I have never liked this because it does not resemble human StarCraft (where players play a best-of-N series), unbalances the races (Zerg vs. Zerg is more volatile and winrates should be closer to 50%), and creates some twisted incentives. One of those incentives is ensuring you beat weaker opponents 100% of the time instead of merely 95%, which means defaulting to reliable-but-weaker strategies to beat weaker opponents, and having a large bag of available strategies to find something which exploits trickier opponents.
And in this iteration of StarSkirmish, where benchmark bots are mostly unable to store data between games, Stardust ran one basic conservative strategy almost every game. This weakens Stardust, but meant that Astra's proxy gates were unlikely to win. So Astra had to do something very uncomfortable for a coding agent: Go back to the drawing board and try something completely different.
Astra completely switched gears, landing on a strategy that PurpleWave also likes to use against Stardust: Use Dark Templar to secure an expansion. This is particularly effective against Stardust, who does not use Shuttles + Reavers, and thus does not gain utility from the Robotics Facility it requires to make Observers.
From there Astra progressed slowly, but discovered a shared weakness of PurpleWave and Stardust: Both of us are designed to dodge Psionic Storms, but that feature was broken in both our bots simultaneously (for unrelated reasons!).
So Astra's Dark Templar opening transitioned smoothly into a two-base timing attack with Dragoons and High Templar. It moved out ...
- code generation speed - logic analysis - understanding of the game platform and how use of C may influence it - considered approaches, tactics and strategies
Don't get me wrong. It is is neat idea. I am just not sure what it measures.
All of the above, to the extent that they can be measured from a simple to verify end condition (stats, win rates, etc). Adaptability, ability to understand and progress on a task, and so on.
If that's the case then these models are failing extremely hard. 7% win rate against human written AIs that they should be copying from memory? A copy paste command should do 50%!
Worth digging into the protocol and see what went wrong.
[1] https://rolandgao.com/blog/gobench/
So it almost seems like a situation where you cannot really have your cake and eat it, at least not in the way it currently works.
Vals IOI, olympiad programming problems solved in C++, and WeirdML v3, unusual machine learning coding tasks also mirror StarSkirmish Bench.
I did a StarCraft: BW hillclimb experiment with GPT-6 Astra and Claude 5.5 Opus. Peter Gostev did a Chess hillclimb saw similar behavior as I did with GPT-6 Astra.
Anyone else feels like they can tell when the underlying model is MoE? Dense models are more concise in their answers. MoE seems to stall for time, writing vaugely until a right set of weights gets picked and commits to an answer. MoE also backtracks a whole lot more.
On a more serious note: all three races play quite differently so it makes sense to focus on just one. And protoss is mechanically the easiest and most straightforward race: fewer but stronger units, a single worker can be used to build multiple buildings at once, versatile basic static defence
Protoss vs Protoss only also helps downscope the problem given the time constraints
You can train a specialized model for this, but a frontier model is ill-fitting.
But right now the response time would be so slow that the bot would not be close to competitive except with the most low level players. Especially because its hard to click on things that are moving at the same time as well.
Sometime soon, I'd like to see an LLM be competitive using only computer vision, mouse movement, and locked into a human possible number of actions per minute.
TLDR they suck and many default to doing probe rushes.
I am considering an experiment where the task is to try to win by adversarially targeting Pluto (a high profile self play based RL model)
I think with the abilities of AI improving, playing games would be a good benchmark as long as they're not trained on them.
One thing we're learning is that harness is HUGE. The difference in performance for the same underlying model can change dramatically across:
1. harnesses
2. allowed tools (no web search/fetch - how does the model refer to documentation?)
3. thinking levels