HN

Need help?
<- Back
Brood War Bench

Brood War Bench

benswerd

Comments (63)

  • pelagicAustral
    Unrelated to the benchmark...I love StarCraft. I started playing it right from the beginning, most of my friends right now are from that era. I literally met people that have spread to almost every continent when I was in my early teens. We played at internet cafes and did not have access to the internet, that was priced differently...I miss those days so much.Everybody was from a different background back then, and nobody was anything other than a guy that plays StaCraft at the cybercafe... And now, we are in our 40's and I know Math teachers, history teachers, oil rig operators, software programmers, professional gamers, lawyers and more... hahah So crazy to think about it... and I know them, we talk, what a world.
  • AntiRush
    Back in 2010, during the early days of bwapi, there was a Brood War AI tournament held by the Expressive Intelligence Studio at UC Santa Cruz. It's interesting to see how different the approaches were back then, vs this or Deepmind's SC2 work.https://web.archive.org/web/20091124210529/http://eis.ucsc.e...There's a great contemporary Ars Technica piece by a competitor:https://arstechnica.com/gaming/2011/01/skynet-meets-the-swar...As an undergrad I did a project using genetic programming. It was not very successful, but it was a lot of fun.https://tomisin.space/archive/starcraft-genetic-programming/
  • mcteamster
    I love this. Funnily enough StarCraft has influenced how I approach AI at a meta levelProtoss: powerful and expensive frontier coding agents you directly micromanage for the toughest tasksTerran: versatile team comps of dedicated agent roles you can delegate well-defined tasks toZerg: massive swarms of specialist custom agents inside your apps that you evolve and optimise for speed and costKnowing every faction has its strengths and weaknesses helps me decide which tools to use for the job.
  • Barrin92
    Game performance is one of those topics that makes it so abundantly clear how limited these systems still are. StarCraft is predominantly a mechanical game so the horizon of what you need to do is quite short and tactile, and even then without advantages no system has come close to beating a human.I saw someone recently try to get an agentic system to play Final Fantasy and it did about as well as a Roomba.
  • faeyanpiraat
    There is currently a bot beating everyone on the ladder. Just watched it today on Artosiscasts yt channel.
  • GodelNumbering
    A friend of mine created GoBench[1][2] that evaluates LLMs on 9×9 Go using KataGo opponents as Elo anchors, you see real capability differences there, like Astra Max substantially leading all other models. I think strategy is a generally interesting area to evaluate LLMs on[1] https://rolandgao.com/blog/gobench/[2] https://rolandgao.com/gobench.pdf
  • tweakimp
    If you want to see human written bots in action or compete in the bot ladder yourself, try https://aiarena.net/
  • gadtfly
    Did it play by looking at screenshots and sending clicks, or was there other mediation/symbolization?It sounds like it might have been actually played in real time, which would be very important to distinguish.I have recently seen other harnesses letting agents play real-time games in what seems like discrete time slices, turning eg Portal into something turn-based https://www.youtube.com/watch?v=ruuGXFAmiOE
  • therealdrag0
    Would Jev be good for this?
  • minimal_action
    I think we're on the early days of games you connect with your agent to. Human + AI units one versus the other. Like knights with their horses. Not sure which is the horse..
  • malfist
    This is a great idea for a benchmark. Something all the benchmarks seem to be missing is strategy, tactical solutions in most of the benchmarks are all thats required but here requires actual long term thinking and tactical thinking, balancing and orchestration.
  • Game_Ender
    Any details about the harness the agents were given? I am curious what representation of the screen and world state was provided to the agents and what tools they had available.
  • moomin
    Wondering what it would look like if you allowed them to write scripts. You could throttle the number of clicks to make it interesting.
  • mococa
    +1 because StarCraft
  • c7b
    At last something that feels properly orthogonal to pelicans on bicycles.
  • benswerd
    + Playable Agent driven Starcraft
  • bee_rider
    A ton of conversations about the game must be in the training set. I wonder, is there any way just from watching how they play, of telling if they tend to pick strategies that people complain or meme about online?
  • winwang
    Would be interesting if you could team a fast and slow agent together -- slow model can either act directly or maybe just communicate to the fast model.
  • anon
    undefined
  • chaostheory
    Gemini wasn’t included, but I’m guessing its performance would have been similar to Grok’s performance despite having roots in DeepMind.
  • dschuessler
    Somewhat related: In 2018, Google DeepMind had already created AIs that were capable of beating professional gamers in StarCraft 2 (the sequel to Brood War): https://www.youtube.com/watch?v=cUTMhmVh1qs