HN

Comments (69)

• Menu

  GodelNumbering
  Wow this is interesting to me from another angle. A friend of mine created a benchmark to evaluate LLMs on 9x9 Go (using KataGo opponents as Elo anchors) [1]. The relative rankings of this (StarSkirmish Bench v0.2) benchmark (GPT-6 Astra > GPT-6.1 Sol > Gemini 4 Argon > Claude Opus 5.5 > Claude Fable 5.1 > Claude Sonnet 5.5) matches the order of his GoBench rankings exactly, even though they are different problems at surface level.[1] https://rolandgao.com/blog/gobench/
• Menu

  cgearhart
  Is there anything interesting to learn from the agent written bots? Or are they just inferior versions of the best bots because the time limit is so short? Is there any indication that they’re trained on the code from the human written bots (like, do they magically “guess” creating functionality that requires experimental feedback or domain experience?) How much are the bots on rails applying the same techniques every time?I think those kinds of questions would be more interesting than the raw win/loss ratios.
• djoldman
  > ... runs each model in its lab's 1st party CLI harness at the highest reasoning setting it offers. Anthropic (Claude models): Claude Code OpenAI (GPT models): Codex Google (Gemini models): Antigravity CLI Meta (Muse models): Muse Code > The CLIs' web search and web fetch tools are turned off, and the agent has no other internet access. The CLI is installed before the clock starts, and the hour counts from when the agent starts. There is no submit tool: the harness collects the bot when the time runs out.One thing we're learning is that harness is HUGE. The difference in performance for the same underlying model can change dramatically across:1. harnesses2. allowed tools (no web search/fetch - how does the model refer to documentation?)3. thinking levels
• Lucasoato
  They don’t stand a chance against my cannon rush, I reached platinum with 10 APM and that strategy ahaha
• MichaelNolan
  So this benchmark has the LLMs writing a bit to play StarCraft. Somewhat similar there was a post here a while back that had the LLMs playing StarCraft directly - https://broodbench.com/ and https://news.ycombinator.com/item?id=49766966
• Menu

  BeaverGoose
  They should benchmark it against Pluto. It's a artificial neutral network bot that can easily take games off the best broodwar players.
• Menu

  iugtmkbdfil834
  I am not sure I like this benchmark, but mostly, because I am not sure what it is really evaluating. It is not an idle dismissal.. multiple factors seem to complicate what it is supposed to look at ( just top of my head ):- code generation speed - logic analysis - understanding of the game platform and how use of C may influence it - considered approaches, tactics and strategiesDon't get me wrong. It is is neat idea. I am just not sure what it measures.
• t1amat
  This is somewhat interesting to me as I’ve recently come to a different conclusion. I’ve been populating a ModernUO server with bots, and have found Opus 5.5 to be substantially better at understanding and building it. I actually started with 6.1 Sol and 6 Astra but pulled them after a week of little progress, had Opus 5.5 take over and it rebuilt everything properly in a night and has carried it forward quite a way since.
• aiiotnoodle
  Would love to see an LLM written bot play Red Alert 2/Red Alert.I think with the abilities of AI improving, playing games would be a good benchmark as long as they're not trained on them.
• Menu

  m_w_
  This is an interesting benchmark - I like that time is limited, as it often isn't in other benchmarks.It would be interesting to see the reverse, how long does each model take to get >50% WR against bots of various tiers.
• Menu

  explodes
  The video shows GLM but it's not in any graphs. Are there more complete charts anywhere else? Ones that include the 45 LLMs?
• Menu

  pelagicAustral
  This kind of makes me think about making better bots for Counter Strike based on human top level players, which i think its a very undervalued category of software. You can even forget about about CS and name any FPS and you'll be hard pressed to find a game with seriously good bots.
• jstummbillig
  How do they avoid the LLMs simply working with pre-training data? Which apparently they do, since the human written bots win. But it's a mystery, is what it is.
• donutdonot
  I assume lots of bot code is part of the training data? So it's basically a version of how will they can improve upon those solutions public out there?
• tweakimp
  If you want to see human written bots in action or compete in the bot ladder yourself, try https://aiarena.net (SC2)
• Menu

  qsort
  It's beyond insane that Astra achieves above 50%. Are models allowed to "play against" the specific bot they're up against? What explains the fact that it drops to single-digits against strong bots?
• rdudek
  Does anyone know if there was a project having LLMs play Cities Skylines and see how well it manages to design and build a city?
• Menu

  rafiyashaheen11
  Oh my gosh! It feels so cool trying this that is played by LLMs. I am sooo eager to try argon 4 and build many robotic projects.
• xyst
  No mention of actions per minute? Clearly the best indicator of performance missed in this experiment
• amishacreates
  Benchmark vs intelligence which considers more?
• Menu

  brap
  What if we let the agents play directly instead of writing bots?
• Cynddl
  Once again, a page with a Pareto curve that is not a Pareto curve. Claude Haiku 5.5 is not on the curve, but inside. Seems like people don't really know what a Pareto curve is.
• nozzlegear
  This thing would beat me, but I'm ass at StarCraft.
• Menu

  caughtinthought
  How do the best humans do against the LLMs?
• 0ddj0b
  Any thoughts on building this for Starcraft 2?
• aleksiy123
  The site overall looks nice too. It’s fun not to see slop for a ui for this kind of project.
• Menu

  brokensegue
  why protoss?
• jehykyle
  [flagged]