HN

Comments (33)

• travishcronin
  Interesting to see harness benchmarks for coding agents. The same problem exists for conversational and data agents but I dont see anyone benchmarking them yet. Seems like manual-spot checks are the norm
• Menu

  nsingh2
  One issue I see with including something like Pi is that it's intentionally bare bones. I don't think anyone uses Pi without some basic custom extensions (subagents, check lists, etc), so this benchmark may not be representative of a realistic setup.Would be interesting to see some sort of ablation test too. E.g. what parts of Codex contribute most to the perf, and can they be recreated more minimally in something like Pi.
• vidarh
  Testing it against Kimi is potentially skewing the numbers massively. Kimi has a number of quirks that requires behaviours that e.g. Claude or GPT doesn't.Harnesses that are built around needing to work with "weird" models will need to deal with that, such as Kimi's tendency to get stuck in tool-call loops.Harnesses built to deal with e.g. Anthropic's models primarily, do not need to deal with that.Claiming on the blog that this gives Kimi Code no home field advantage seems like a dicey assumption. I haven't dug into the newest Kimi Code much, but the older Kimi CLI included several tools that were clearly specifically aimed at working around that behaviour - when I copied their checkpoints and "dmail" mechanism into my own harness, the performance with Kimi improved dramatically, but it made zero difference against Anthropic models.That doesn't make the data worthless - it's clear you shouldn't use Clade Code to work against Kimi. But it does significantly limit the utility of it.
• markbao
  This is great. People always talk about how important the harness is and yet we have so few harness benchmarks. Agree with sibling commenter that ‘harness x model’ is needed.People always talk about how Cursor harness has some secret sauce; would like to see how that one stacks up.Thanks for making this and filling a real gap!
• yorwba
  There really should be error bars on those measurements. With just 30 samples, the 95% confidence intervals for accuracy should all be more than 35 percentage points wide, so comfortably overlap. For cost it's harder to say, because outcomes aren't constrained to {0, 1}, but I also expect a lot of variability there.
• Menu

  knombertus
  Is Github Copilot (integrated in VS Code) not a thing? I use it all the time and don't know what more I could wish for.Disclaimer: I don't let agents run on huge tasks for hours. Almost all tasks I give them are done in under 30 min.
• kaishin
  Putting only the tasks and results in the repo is a poor decision. These conclusions would be far more credible if anyone could re-run the benchmark.
• Menu

  kakugawa
  With all the hype around the latest Gemini 3.8 Flash/Cyber release, will Antigravity CLI [1] be supported?1/ https://antigravity.google/product/antigravity-cli
• TheJCDenton
  Incredibile result for pi. The ratio quality / complexity of the harness make me think that all other harnesses are very bloated.
• fhn
  I just started using Hermes and it's pretty good. Guess I'll try Pi. I'm hesitant on DSH.
• Menu

  GodelNumbering
  Do you have instructions on how to run a custom harness against this? There are none on the linked page. I want to run Dirac (https://github.com/dirac-run/dirac).
• netcyrax
  Very interesting. As models become commodities, the harness will be the next optimizing game.
• EFLKumo
  I've been confused a lot why there isn't a benchmark to measure a harness's performance rather than the model's one. Now there it is.
• Menu

  xnx
  Good start, but needs to be harness x model to be useful. 3 top harnesses x 3 top models would be more interesting.
• monster_truck
  It's nice to see time reflected here.Deepseek is fuckin fast! Seems like the bigger the task, the faster it gets, which is kind of unfortunate because nobody is going to be doing 17 benchmark passes on a $50-100 task. I'm assuming the brief pause before it avalanches out 16kb of text at upwards of 250tk/s (multiples beyond anything resembling a comfortable reading speed) is some sort of workload evaluator directing sessions to individual/multiple cards, occasionally waiting for what it thinks is best to become available.I couldn't believe it at first, it shit out a damn fine multithreaded physics simulation fabric (integrated into a massive codebase, tests passing) in under an hour. Anything I could find online says they average like 80 but my logs average ~3x that.
• RadixIO
  Software is constrained when you write it. Agents have to be constrained while they run.
• grigio
  Where is jcode?
• Edward40
  The Exo harness is the most interesting one in the benchmark, as it has the potential to complete tasks with fewer turns and cost.
• tamimio
  Harness wars are the next browsers wars!!But wow, opencode is that bad?!
• Menu

  jonstewart
  Forgive my stupidity but how do you run Claude Code with non-Anthropic models?
• aquadros
  [flagged]
• Finance-broski
  [flagged]
• anon
  undefined
• ADD-SP
  Codex is the best harness to me because of its GUI and subscription.