Wow this is interesting to me from another angle. A friend of mine created a benchmark to evaluate LLMs on 9x9 Go (using KataGo opponents as Elo anchors) [1]. The relative rankings of this (StarSkirmish Bench v0.2) benchmark (GPT-6 Astra > GPT-6.1 Sol > Gemini 4 Argon > Claude Opus 5.5 > Claude Fable 5.1 > Claude Sonnet 5.5) matches the order of his GoBench rankings exactly, even though they are different problems at surface level.
Is there anything interesting to learn from the agent written bots? Or are they just inferior versions of the best bots because the time limit is so short? Is there any indication that they’re trained on the code from the human written bots (like, do they magically “guess” creating functionality that requires experimental feedback or domain experience?) How much are the bots on rails applying the same techniques every time?
I think those kinds of questions would be more interesting than the raw win/loss ratios.
I am not sure I like this benchmark, but mostly, because I am not sure what it is really evaluating. It is not an idle dismissal.. multiple factors seem to complicate what it is supposed to look at ( just top of my head ):
- code generation speed
- logic analysis
- understanding of the game platform and how use of C may influence it
- considered approaches, tactics and strategies
Don't get me wrong. It is is neat idea. I am just not sure what it measures.
Even the most powerful frontier models on SOTA hardware are not fast enough for starcraft, which generally involves multiple informed decisions per second at peak moments.
You can train a specialized model for this, but a frontier model is ill-fitting.
How do they avoid the LLMs simply working with pre-training data? Which apparently they do, since the human written bots win. But it's a mystery, is what it is.
I noticed Pluto is running on a simple CPU? If so, that's very nice for future RTS games. You just ship (or use a server-client arch) a bunch of models of various difficulty levels to let players practice. Could we even get models tuned to specific campaign levels? I wonder how costly the creation of Pluto was.
It's beyond insane that Astra achieves above 50%. Are models allowed to "play against" the specific bot they're up against? What explains the fact that it drops to single-digits against strong bots?
Yes during the 1 hour window they are allowed to play practice matches against opponents they are graded against - though they do not have access to their opponents source code.
They drop to single digits against the A tier and S tier bots because those bots are just generally harder opponents to beat.
There is currently an AI bot playing on the ladder in Korea and essentially it will nearly always win because it can micro every unit individually across the entire map. If a normal human can max out at hundreds of actions per minute, the bot can do thousands. Broodwar is so insanely difficult mechanically that there is no way for humans to compete. I imagine in a few months, maybe a year, it will be completely unbeatable.
[1] https://rolandgao.com/blog/gobench/
I think those kinds of questions would be more interesting than the raw win/loss ratios.
- code generation speed - logic analysis - understanding of the game platform and how use of C may influence it - considered approaches, tactics and strategies
Don't get me wrong. It is is neat idea. I am just not sure what it measures.
You can train a specialized model for this, but a frontier model is ill-fitting.
It would be interesting to see the reverse, how long does each model take to get >50% WR against bots of various tiers.
Although I should caveat the format advantaged the LLM bots compared to traditional tournament rules.
They drop to single digits against the A tier and S tier bots because those bots are just generally harder opponents to beat.