HN

Comments (133)

• taintech
  I think currently AI models did incredible improvement against 2025 OpenRCA research paper with 11.25% success rate.Key questions, are they topped in performance? Is there some next leap?
• Menu

  springtimesun
  I built exactly this using my own codebases. The setup isn’t that complex; split the git history to just before the change, sandbox the agent with everything they will need at that commit and lightly modify rules so they don’t go searching outside the box. Then they get the same prompt (usually the ticket that began the work) and are graded against the accepted PR.The thing that takes the most time is finding the examples. In my real dev flow it’s rarely ticket -> PR -> merge, things bounce around a lot more. So, even though the stated goal is to get away from one shots, that is basically the environment you have to set or else test for specific other outcomes (e.g. agent stopped and raised a question when it realized x).It takes time to do, but I would really recommend it. Now I can push new open models through the batteries and see how they line up to past ones in a few days (I run them locally, it’s slow). It moves my sense of x model is good at y and bad at z to from vibes to a better heuristic (these still run at temp 1, heuristic is the correct way to think about outcomes IMO). It grounds it in your actual code and problem space.My takeaway from my testing: in Rails or front end codebases, most models I test are competent and with a human in the loop they would accomplish their goal of getting to a mergeable PR. They are not as good as Claude and since I pay subsidized rates via subscription Claude still gets first pass. They are very worthwhile to layer in as reviewers and catch many issues. My anxiety about a rug pull by the frontiers has been turned way down. I would have to adapt to a local only flow, but it wouldn’t be much adaptation and the opens can deliver in their current state.
• Menu

  glub
  I'm surprised Sol and Astra are leading "Unverified assumption" metric and Fable is better there.I run Fable as my main model with Sol as advisor that watches every turn. Fable likes to throw around assumptions that it didn't check that are simply false, and Sol always goes to actually verify them and then alert Fable it's assuming things.I've tried reversing this pairing with Fable as advisor. It'll just sit there going "sounds good"
• Menu

  prometheus1992
  Does this mean they ended up sharing those private codebases with OAI, Anthropic etc? Also, the ~30% number tracks with my experience. I thought I was going insane for expecting too much from the models but they are still bad, including astra. This morning it messed something pretty trivial while fixing an issue which I was shocked to see. Also2, benchmarks don't mean much these days.
• Menu

  lmeyerov
  My intuition is that many of the better & bigger 'private' code bases, at least in terms of claude code and codex... are not in fact private at this point.One lesson of running botsbench.com, in a slightly different domain, is to measure for model contamination every time.
• Menu

  jstummbillig
  I am trying to estimate if my reaction to seeing GPT-5.6 Sol last on that list is reasonable or or mostly emotional and find that I have no way of telling.
• Menu

  skhameneh
  Some things in this seem reasonable, but others just don’t make sense and there’s crucial details missing (like reasoning levels and what harness was used).For example, I found Kimi K3 to use more tokens than some other models, which caused it to cost twice as much purely because of the token volume. This experience lines up with ArtificialAnalysis’s benchmarks, but not these.There’s a number of other comparisons here that don’t match up with my experience or other benchmarks. By many accounts, this is the outlier.I could attribute the differences to harnesses used or something like reasoning levels, but none of those details are published.While this seems interesting, I can’t take this seriously.Correction: The harnesses are listed as a column, I missed that. My other concerns and questions still remain, it’s unclear why some of their results are the outlier that does not match my experience, ArtificalAnalysis’s benchmarks, or some of the experiences of others commenting.
• chandureddyvari
  It also depends on your skills and tooling (test execution and verification- agent browser, functional/unit etc) GPT 5.6 Sol lagging behind Kimi, GLM 5.3 is surprising to me.IMO Fable 5.1 ~ Astra > GPT 5.6 Sol > Opus.
• barbegal
  Without a human to benchmark against it's really tough to gauge how good these models are vs how good the task definitions and existing codebases are.My intuition from the example full instructions are that the tasks are poorly specified which results in ~60% failures due to bad assumptions and missing requirements.
• matheusmoreira
  I used a similar methodology. Code review is my most requested action, so I used blind code review results to compare the frontier AIs.Even posted an article about it:https://www.matheusmoreira.com/articles/code-reviewing-lone-...Unlike TFA, the lone lisp code is public. I suppose the models could have been trained on my codebase. Still, I think it produced some interesting results.Took months and loads and loads of tokens to do this, so I'm not gonna repeat this study as new models come out. It did anchor all of my future expectations, though. OpenAI is winning as far as I'm concerned, and their cybersecurity program is the only remaining pain point.
• Menu

  visiondude
  this is the closest benchmark to my experience using the model harness combo. Astra for as great as it is falls slightly behind Fable 5.1 for me for large feature work (although it comments code much better). in particular, Fable is able to assess priority better than Astra (meaning Astra sometimes does things that aren’t worthwhile while missing things that are clearly important, particularly on possible ballooning scenarios- fable catches “this works for x amount of data but if we run this on y way greater than x amount of data we’ll run into issues). Gemini 3.8 is under appreciated, use Google Stitch to see it in action if you haven’t used Agy yet.
• janaksunil
  i'm janak, cofounder of Specific Labs (YC F25) and one of the authors of Real-SWE.if i can help answer any questions please feel free to email me at janak@withspecific.com, happy to send over my phone number as well :)
• Menu

  ttul
  We built a “code atlas” that provides the LLM with a semantically queryable map of how things connect and relate in a very large and sprawling codebase that evolved over 15 years. It tends to dramatically reduce the length of time models have to spend reading code while also making sure they are aware (within their context window) of nuances that are important that might be missed were they forced to just rely on reading the code in hundreds of repositories.I strongly recommend trying this approach out yourself. The recipe is not rocket science. Get your coding agent to take a first cut at building the atlas itself, and then manually correct it. Once you’re happy that it got things right, put an MCP on it or a CLI or whatever. And your LLMs will know what to do from there.
• option_greek
  A lot of this tracks but misses the variations in what the models in general are good for.
• hollars
  The high score of Gemini 3.8 Flash vibes with my experience anecdotally. While it often goes off the rails with open-ended questions (which is a strength if taken with care), it is also a good at solving issues in a well-defined environment like an enterprise codebase.
• Menu

  bel8
  I'd love to see these:- DeepSeek V4.1 Flash- Kimi K3- GLM 5.3 (and flash)- hy4-preview- Grok 4.6All of these can be acessed using a $10/mo OpenCode Go subscription.
• Menu

  freakynit
  This is the first set of benchmarks which match my observations around gemini-3.8-flash perfectly. This model is a true hidden gem.
• obilgic
  Gemini 3.8 flash has been incredible for our agents. For us, It performs better than any other model except Fable.
• Menu

  finn888
  Averaging pass@1 across eight runs per task is useful; it exposes harness consistency instead of letting one lucky resolution dominate.
• Menu

  demibabs
  > Each task comes from a private production codebase that we licensed from a real-world companyHow does that work?
• Menu

  arshxyz
  Dreadful color-coding on the output tokens table
• Menu

  dgellow
  A bit of a meta question: what are the most relevant benchmarks by now?
• matt3210
  These'll be part of the training set eventually.
• Menu

  m3kw9
  what if the production code base was made by mostly by Anthropic models?
• Menu

  jcmontx
  I’ve been able to offload most tasks (coding or eles) to Codex since 5.3-codex with extra high thinking
• andai
  AGI 38.8%
• Menu

  traceroute66
  So TL;DR benchmarking in a completely non-reproducible manner ?"Model X performed great, but we can't possibly tell you anything about the code it was looking at apart from it was a large code base from an unknown company".So basically pinky-promise benchmarking ?I'm not sure I follow the value here ?
• anon
  undefined
• cute_boi
  This benchmark is shitty because it puts Gemini in 3rd position.I tried gemini on simple code base and it invoked 210 tool calls just to update 3 lines of code.
• Uptrenda
  real software engineering benchmark is how much stress you can take at work. Everyone knows this, bakka.
• Menu

  IshKebab
  I think these benchmarks are not that useful, e.g. this suggests Fable is better than Astra, but in practice Astra is waaaaaay faster (like 5x; it's not even close), and also waaaay less annoying to talk to.There's only two or three sane options here - you can easily try them all and pick yourself.
• songhonglei1985
  [flagged]
• hefu_hk
  [flagged]
• aryansingh9034
  [flagged]
• paidx
  [flagged]
• skilledDevelope
  [dead]
• ahmetaytar
  [dead]
• bix6
  Wake me up when September ends or when I can do this locally.
• Menu

  bdlowery
  The fact that gemini 3.8 flash is so high up there just tells you this is an awful benchmark.Try and use gemini 3.8 yourself for any real world work and you'll see it's terrible. It'll just go in circles reading the same file 20 times for no reason making hundreds of tool calls for a simple change.EDIT: I was using gemini cli... it's not a harness issue lol