HN

Comments (35)

• amarble
  I did something similar in March using Opus 4.6 (iirc) on google's "Parameter Golf" challenge, "a challenge to train the best language model that fits in a 16MB artifact and trains in under 10 minutes on 8xH100s, evaluated by compression on the FineWeb validation set (tokenizer-agnostic, bits per byte)."I never ended up writing it up, but you can watch me spend $600 as it explored different experiments: https://github.com/rbitr/parameter-golf/blob/main/BUDGET.mdI found, similar to another comment, that it got in local minima very easily and continued to pursue loosing ideas instead of exploring (despite being prompted to do so and being aware of how much budget it had left). I also found it tended to ignore instructions. And one example, when I fed it a better solution that had come along from the public leaderboard, it ignored everything it had done and started exploring locally around that new solution, which wasn't very interesting or productive.Would be interesting to re-run with a newer model but it's hard for me to justify the money again.
• lhl
  Pretty interesting to see on the training front. I've used most of these models to grind semi-autonomously (days at a time) on kernel optimizations (except for Fable - it kept triggering guardrails almost immediately and bouncing me down to Opus 4.8 at the time). I think for a lot of people that might be the biggest problem, although it looks like Opus 5 still does well.I found that if you leave them alone undirected, the models (especially GPT models) will rathole, but with the right scaffolding it seems to work pretty well. My general loop is to start with ideation and profiling phase, limit # of runs before forcing moving on to the next item down the list, and then iterating, potentially mixing models with "fresh eyes". This is probably something that could be fully automated, but I like checking in once a day or so and seeing what's happening and redirecting.
• vibe42
  "Almost every model finds the same winning ideas. What separates the best traces is what an experiment leaves behind. They preserve weak signals long enough to validate them, but they also have a better understanding of the results."Curious if a harness that helped preserve signals in some history log would change the outcome.Also curious if different goal prompts would have changed the outcome. Not a bunch of prompt engineering; small diffs like "consider novel solutions, keep track of weak signals".IMO they allocated quite a bit of GPU time to the same goal prompt.
• nsingh2
  What's going on with sol here? The note says it spends a lot of time waiting, did it just not effectively use time (i.e. something like parallel runs) so it's graph ends up being stretched in the time axis?I'm also seeing notes like on Opus 5 saying it was a run with a older serial version of program.md, so the graphs aren't complete apples-to-apples comparisons?Edit: the blog seems to address these https://www.primeintellect.ai/blog/measuring-autonomous-rese...
• Menu

  totetsu
  “We ran 153 autonomous runs across 18 frontier models on the nanoGPT optimizer speedrun.”Uh.. okay.. but whats a run… read blog“We want to measure how well frontier models can conduct research….””we ran 153 autonomous runs on the nanoGPT optimizer speedrun across”Okay but what is a optimiser run and what connection does it have to being good at research?“For comparison, Anthropic's internal automated AI R&D evaluation optimizes a model on a CPU node,”So I should go look what Anthropic was doing to understand?Why not just explain what it means in their blog..
• espadrine
  I would be interested to have a third X-axis with dollars.Time is sometimes more about inference infrastructure (especially with systolic chips) than model quality (and providers tweak knobs to support higher batches at the expense of latency).Tokens are not always fully equivalent between models.
• Menu

  c0rruptbytes
  auto research the new cool kid on the block - look at https://mlx.fast
• Menu

  bastawhiz
  This is a pretty embarrassing showing for Grok. I wouldn't trust xAI models as far as I can throw them, but I am interested in how much of this is deficiencies of the model and how much is their harness just terrible. Not that it makes it better, a good harness is far easier and less expensive to make than a model.
• JSR_FDED
  Huge uplift with Kimi K3 by using the Prime Agent coding harness. Hadn’t heard of this harness until now. Is it worth looking at for regular coding sessions or is it only worth it for very long horizon tasks?
• ninjahawk1
  I might’ve missed it, but why was Fable 5 tested on high while Opus 5 was tested on max? Seems like quite a few of them aren’t on the same effort setting as well. Although effort doesn’t really matter anymore since they can change it dynamically, seems like that might be viewed as an experimental error to some.
• throwa356262
  "We were again surprised by the lack of novelty. The models clearly understand the objects they manipulate at a deep level, and yet very few genuinely new ideas emerge..." That was disappointing. I guess 2026 will not be the year of AGI.See you next year, guys?
• Menu

  narush
  Cool post!I'm a co-author on a recent blog post from METR about the NanoGPT speed-run here [1]. I think it'd be of interest to anyone who enjoyed the original post. Appreciate the good beefy runs and spend here, it's a (from my experience) not super easy to do!(Also: just to label this comment clearly: it's written hastily from a car, and based on lighter skim of the original blog post [2] than would be ideal. Please correct any mistakes or misinterpretations I have here!)A few callouts:1. If I understand the experiment correctly, they start the models at the original baseline. If this is true, I have some worries about contamination. Appendix C [3] has some notes on likely contamination we observed in recent models. This makes interpretation a bit harder.2. If you look at the token scaling plots in the original post: not all models are hitting a performance plateau. This is an important point: we shouldn't treat these results as a full upper-bound on capabilities, but rather some bound on model performance @ cost (assuming good scaffolding, etc).3. Our post is mostly about how to _interpret_ the results given here. Quoting from our post: "If we can estimate performance as a function of cost for both humans and agents, we can measure the “expenditure horizon” as the point at which those curves cross: the budget at which humans become more cost-effective than AIs. "Feedback appreciated. I think you can see expenditure horizon as a sibling methodology (that is much less validated) to METR's time horizon work [4] - roughly, instead of baselining against the time it takes humans to complete tasks, you baseline against cost. This may be better suited to some types of problems similar to NanoGPT.[1] https://metr.org/blog/2026-07-21-expenditure-horizon/ (Most of this work was my coauthors listed on the post, not me. I'll claim credit for any mistakes though :) ).[2] https://www.primeintellect.ai/blog/measuring-autonomous-rese...[3] https://metr.org/blog/2026-07-21-expenditure-horizon/#append...[4]https://metr.org/time-horizons/(Edit: METR is hiring. Email is in bio if you're interested in helping AI companies and wider society understand the capabilities and risks of AI.)
• Menu

  nl
  How good is GPT-5.6 Luna!Such a cheap model, and Sonnet levels of performance.
• Menu

  henry2023
  Crazy to think that we have open weights capable of challenging trillion-dollar SOTA models.Interesting times ahead.
• skybrian
  Neat!The graphs show the "best validated result" for each model. I wonder how much variation there is between runs for a model?
• bee_rider
  Why is the human score such a round number?
• logicallee
  I'm highly interested in their Grok 4.6 run which is currently running. I think it is a very good fit for doing really well on this benchmark. Will have to check this page again in a couple of days.I wrote a quick review of Grok 4.6 here: https://taonexus.com/publicfiles/aug2026/grok-4-6-review/
• Menu

  ninjahawk1
  I misread the graph and genuinely thought you put NanoGPT where Fable is.Lol.
• moffers
  “We put liquid bandage on a bee, and it died”
• amelius
  [dead]
• luciana1u
  [dead]