HN

Comments (169)

• Menu

  mwt
  > The rocket launch was only achieved through significant ‘manual’ effort on Sol’s part as it papered over its late-game failures to connect certain production systems by repeatedly ferrying supplies between areas of the factory.Robots: they're just like us!
• Menu

  minimaxir
  If you haven't been keeping an eye on LLMs playing games, GPT 6.1 Sol can new beat games 100% vision-only (without harnesses that give the agent extra information about the game). Pokemon is no longer a difficult challenge for it, even with additional difficulty restrictions such as hard mode and level caps. Recently, GPT 6.1 Sol beat Final Fantasy VII and its bonus bosses vision-only surprisingly quickly.I suspect Opus 5.5 can do even better but there aren't as many people testing it.
• Menu

  mickeyp
  Watching AI play games is fun, but I can't wait for someone to crack having small LLMs in games to enrich them in useful and interesting ways.
• Menu

  program_whiz
  One limitation of these type of demos (and similar with Minecraft, etc) is that: 1. Putting into "peaceful" mode makes response time a non-factor 2. These models have read and have access to every guide and video ever published on the subject. Most humans, especially on first play, learn game systems and interactions from scratch, then develop novel strategies and plans and "hypothesis test" their way into beating the game.This is still impressive compared to where we were a few years ago, and being able to synthesize all existing information about a game and then play it to completion in a semi-realistic setting is great.At a quick glance someone might mistake this for "brilliant AI solves Factorio", when in reality, given a similar but different game where it wasn't gifted all the information or training data (e.g. massive videos and screenshots) it might fare terribly (as others have pointed out in comments).
• Majromax
  > The rocket launch was only achieved through significant ‘manual’ effort on Sol’s part as it papered over its late-game failures to connect certain production systems by repeatedly ferrying supplies between areas of the factory.Looking through the video, around 12m one can see the Sol trace report the cancellation of a conflicting 'courier' (hand-feeding) script.Rather than implement 'manual' effort, it seems like Sol considered the Engineer to be just another unit of automation!That's not exactly within the intent of the challenge, but it's not too far out of bounds either. It's also a hard loophole to close if the agent interacts with the game through a scripting layer.
• Menu

  fasterik
  It doesn't seem to understand the core mechanic of Factorio, which is putting outputs onto belts and routing them to other inputs. It does this in a few places, but for the most part everything is put into chests and moved around by the player.
• Menu

  VoidWhisperer
  Factorio is a much easier game without the bugs being a pain in the arse as you expand - i wonder how Sol would handle the bugs
• Menu

  kanemcgrath
  It is weird and almost alien how AI designs factories. When Claude announced Fable 5 they also did a Factorio demo that was the same kind of designs and a lot of hand feeding and direct insertion. In some sense it proves to me that llms are not just copying and are actually creating a bespoke solution. But I honestly expected more from frontier models
• jart
  Those biters make life nasty brutish and short for the robot. https://youtu.be/6ZdsmWfKNJU
• Menu

  CSMastermind
  I tried a few recently on Civ 6 Deity, and they all lose pretty quickly.
• Havoc
  I bet it helps that Factorio attracts exactly the kind of obsessive players that'll leave a digital trail of training data documented in precise detailBit like warthunder guys can't help but leak classified info
• damowangcy
  I don't have the budget to do so, but if I do, it would be interesting to put different models in the same game with the same prompt to observe the behaviors that different models have when told to problem solve.
• juanpabloaj
  it could be interesting to have a player vs player mode in Factorio, and see two AI playing against each other
• Menu

  ArcHound
  Honestly, I'm more interested in the harness. Having an LLM gardener that could expand outposts to connect to the main network, or just pop in to observe if there's any major problem to report could be a cool project for long-running multiplayer bases.I see it at https://github.com/ryanmadden/autorio and https://ryanmadden.net/claude-plays-factorio/, but I'd like to ask - any plans to support blueprints? Or push it further into Space Age or dare I say.. pyanodons?No matter the answer: very nice!
• blobbers
  This is a much better benchmark as a concept (without knowing the details of how the problem is framed to the agent) than some of these ridiculous things the frontier labs are reporting (vending machine simulator? come on).Can you report more in depth at stumbling blocks? It's a surprisingly linear curve.I'm going to try it on a local run.I wonder if there is room for "recommendorio" where it gives you AI advice on your current factory.Is the model pre-trained with any knowledge of the game?
• Olscore
  Automating fun was not on the top of my priorities but sure.
• LaGrange
  Wow, that's the saddest headline I read in the week Patrick Stewart died.
• jquery
  Space Age or classic? Space age is much harder.
• Menu

  sdwvit
  how much did it cost I wonder
• onehair
  is that spaghetti factory i am seeing? :O
• Menu

  bosshawg
  interesting artifact of over investment and subsidies of AI and datacenters
• Menu

  ang_cire
  Another thing AI can do that I can't...
• Menu

  dfedbeef
  Good to know it's good at something, I suppose? Sol seems like it always just wastes my time and writes terrible code.
• marsven_422
  [dead]
• naillumanist
  [dead]