Need help?
<- Back

Comments (103)

  • seizethecheese
    The headline here is confusing. They ran a bunch of harness and agent combinations on terminal bench tasks, and 64% of the spending was on tasks that didn’t pass terminal bench verifiers.It’s unclear whether that’s supposed to be surprising against the selected models for the benchmark. I suspect it’s not and the headline is written to pander to “wasting money on AI” upvotes.Edit: they changed the title. The one used to get to the front page was something like "Across N agent runs 64% of agent cost went to failing runs".Edit2: since the link is poor quality, I tried to answer some of these questions myself using 927k agent runs vs. this blog post's 27k: https://micjm.com/blog/on-model-harness-interactions/. The upshot: there's not a lot of harness x model interactions. OpenAI models do seem to have a preference.
  • Zigurd
    I find myself pulled in two directions by the performance of coding agents: First, keeping them on a short leash seems sensible. I don't get failed results very often. When I first started using coding agents, this seemed not just sensible but required in order to not deal with a lot of hallucinated weirdness. Secondly though, it's pretty cool to vaguely wave your hands in a certain direction and get both user interface and business logic that's objectively nicer than I would've gotten from the effort I would've put into manually coding. Whenever I cautiously use a coding agent, I wonder what I am missing by not letting it cook.
  • xyzzy_plugh
    I don't like this framing.For a given model, the best harness depends on the problem. As models become increasingly capable, the harness appears to matter more than the model. This is perhaps controversial but allow me to explain.Most of my problems are in a single domain, using specific tools and programming languages. I give whatever model de jour the specific tools it needs to solve the problem. I don't let it execute arbitrary scripts or programs (though it can, if it really tries). I give it a restricted number of calls and I feed that information back into it (budget). I let it farm out work to other models, using the same harness. I give everything a budget, which is tracked and enforced, but it's not real, and refills at different times.This bespoke, specialized harness absolutely smokes any other mainstream harness for these tasks. Significantly faster, cheaper, more correct.But I can't build a mobile app with this harness, or make a movie.These tests are doing the same thing. They measure a handful of scenarios. I can't easily tell if those map to my needs or not. In my experience, very few of them do.Can these general purpose harnesses solve my problems? Yes, if I'm willing to let the model churn for a long time and burn a lot of tokens. But that doesn't feel like a sustainable practice.Maybe I'm too early, maybe it's premature to optimize the harness this way, but I'm spending fewer tokens than my peers and getting a lot more done, and that matters.
  • cpeth
    Harnesses are all fairly simple loops from the ones I have looked at; there is not much to them.As someone who regularly uses Claude Opus 5.5 from Claude Code and from Copilot, I would say harness barely matters in terms overall performance and behavior. Opus is nearly identical in both harness. The opposite is also true, if I switch Copilot to something like Gemini 4 Argon it's completely different even though I'm in the same harness.Just some andecdata.
  • bisonbear
    These are cool results and I'm glad that tokens are being invested to eval harness capabilities.However, I am hesitant on the correlation between pass/fail in evals which are a binary 0/1 determined by tests, and real world usage, where users can steer, run code review, and care about more than just test pass/fail.I wrote about the problem of determining "success" for coding agents, and a lot of it boils down to the fact that agents can write dumb code that passes tests, and write smart code that fails tests, and we need to shift to rewarding quality of code and alignment with intent rather than just signal on held out tests (which is still important!)https://www.stet.sh/blog/cost-per-successful-task
  • ford
    To be clear - "Failed" here means "Did not meet all pass criteria on a given benchmark task", not "Experienced a runtime failure"
  • zodwick
    I wonder if there is corelative improvement between the same lab's own harness and model.For example I remember Codex having a different line number format than CC and every other harness had to have a two format tool to maximise performance.I wonder if sucha thing exists still or do they RL on all possible combinations now
  • alok-g
    Genuinely asking: Why is Kilo code (https://kilo.ai) not present in this and most such analysis? It does not also get talked about much on HN in general.
  • scosman
    We need this benchmark for non-coding harnesses. Harnesses with custom knowledge, custom tools, specific guidance, etc. A ton of the task-specific performance comes down to the skills/tool-set.I've been building a tuner that tunes the harness (prompts, skills, tools, etc) instead of the weights of models: https://github.com/Kiln-AI/harness-tuner -- Might try to publish some "untuned" vs "tuned" comparisons, the numbers are dramatic.
  • scubazealous
    While this could be better, I wonder how much time the average developer spends on runs that fail? How much time do students spend on runs that fail?
  • franzb
    Original title: "In 27,280 coding agent runs, 64% of inference spend went to runs that failed"
  • fennecfoxy
    Yeah, developers sometimes go down the wrong path as well. It's a non deterministic system exploring a huge problem space...
  • elAhmo
    So, on my daily coding tasks choice of provider, model family, model version, effort/thinking level and harness all have an effect. Some of these are multiplicative (like harness), so this is all extremely confusing for the end user about what the hell should I use do do my tasks.
  • evalstate
    Just use a harness that has been properly integrated and tested with each model, with published traces and comparisons (author of) https://fast-agent.ai/benchmarks/GitHub and Copilot CLI have started to do the same btw (not sure if they publish the traces/reward hack results). But we should normalise this as consumers shouldn't be forced in to playing this game.
  • d-sanoj
    I think the harness matters less when we are measuring the work done. There are all kinds of benchmarks in differ cases showing one model is better than the other. The existing ones like Claude Code or codex work well with their models.
  • cronin101
    I'm far behind the curve with local/open weight models, but I thought it was common practice that you should do some last-mile fine-tuning on the particular harness that you're using and would get a reasonable return on investment for doing so (particularly on quantized models)?Can anyone confirm/deny this is the case?If so, it seems a shame not to include this practice into the reporting methodology for full transparency.
  • SAI_Peregrinus
    When a task passes all tests, it's done. More tokens should be spent on failing runs, because most runs where tokens are used should fail. For the same reason that your lost keys are always in the last place you look for them.
  • Majromax
    Saying that 'steps per pass' explains cost seems nearly tautological. Run costs are proportional to the number of tokens produced, and the number of tokens is strongly correlated to the number of steps.The 'steps per pass' criterion is also not something under the harness's direct control in the way that its tool collection and system prompt (/start tokens) are. Even the cache hit rate is hard to control beyond a limit, since modern harnesses rarely rewrite history.I'm also suspicious about the run-to-run variance between models. Given the high variability within the presented results and the observation that failed runs (which can spin for a long time before delivering a wrong answer) account for a majority of the cost, I'm not sure that many of the conclusions here would hold up under a second sample.
  • samueloph
    Am I'm missing something or at this point the distinction between "harness" and "agent" doesn't exist anymore because too many people use them interchangeably?Is this post mixing them up or is my understanding of these concepts off?
  • bensyverson
    This is interesting and valuable, but I'd love to see a follow-up that tests a more parallel or subagent-heavy workflow.
  • augment_me
    This is completely expected. Agents are pseudo-greedy search functions that brute-force solutions using their context.If you want perfect search, you need lossless compression, which defeats the point of the current systems that are lossy L2-optimized representations of all information.
  • bob1029
    The entire point of this AI thing is that the cost of failure is now ~zero. You can try again in a few minutes / hours. A traditional software business might only complete one iteration before time runs out.
  • jrmg
    "In 27,280 coding agent runs, 64% of inference spend went to runs that failed" (the title this submission was submitted with) is a pretty editorialized title.The page's title is: "Introducing FrontierHarness Eval v1.1"
  • andai
    >Through our experiments, we found that you should consider switching harnesses when switching models to maximize your experience.I've seen this for years, and seen several papers that studied it: the same prompt will help with one LLM and harm performance for a different LLM on the same task.And on benchmarks which test LLM x Harness, they find that LLMs tend to do way better with their "own" harness. (Or in the case of Chinese LLMs, the harness of the LLM whose training data they borrowed :)
  • colinmarc
    How are people even using Baseten for coding agents? I am using Pi (so very small requests) in a "normal" single-agent way and I hit the rate limits on throughput immediately.
  • WorldPeas
    evolving harnesses are also a cool development I've seen recently though I feel they'll have diminishing returns https://www.alphaxiv.org/abs/2609.01437
  • justincormack
    You do need to optimise prompts too, eg I found that openai models just gave up in Pi, but adding a line to the prompt matching Codex eg "Once implementation is authorized, the agent should persist through the bounded change, tests, review and final validation. Complexity, expected duration and context compaction alone are not reasons to stop." fixes that. Otherwise it would look at tasks and say they were too hard.
  • winddude
    Reading a lot of the comments everyone is talking about state actors, etc.Let's not forget multiple departments like the Pentagon, FAA, Homeland Security etc, have been raising alarm that things like this were becoming a heightened risk since Trump and Musk cut ~300k jobs with DOGE, and slashed civilian contractors across nearly every department.
  • BugsJustFindMe
    This is nice and all, but I really want to see the best harness for the actual frontier models ChatGPT and Claude, not just open weight models.
  • worik
    Given the variation between results for different, but similar capability models it is hard to take this seriously
  • maerF0x0
    So? I'm sure exceedingly high amount of EC2 cpus are idling, or lost to discarded branch predictions etc.We waste whatever is cheap to gain what is dear.(Of course we iterate towards better, but that doesn't mean we delay shipment/usage once it's profitable / better than status quo)
  • 0xbadcafebee
    The choice of agent/harness is, as people frequently point out, just one of many variables that changes agent performance. Skills, MCPs, Extensions, system prompt additions, often have drastic effects, making things better and/or worse. It's interesting to compare them side by side, but probably won't reflect your own unique experience.Second thing to consider: ZCode was sending everyone's code to Alibaba... not sure I would use their harness just to save a few cents on coding.
  • jedisct1
    It would have been nice to see swival.dev in that list.But unsurprisingly, Codex works pretty well, too (and not just with OpenAI models), and Claude Code is just a waste of tokens and money.
  • brookritz
    The TLDR saya: Basesten, Harneas and Leaderboard and still have no clue.
  • shevy-java
    Well, it is a milking system. You have to pay for those tokens anyway.
  • underdeserver
    Wait till you hear what the human success rate used to be.
  • aleqs
    Another bullshit result obviously meant to promote a specific company/product. Codex is garbage btw.
  • guanlan
    Hi HN, I’m Guanlan, founder of Runta. The eval runs on our runtime, so weigh this accordingly.The short version: when you change the model, think about changing the harness too. On any one model, cost per pass varies by up to 4.4xWhat we ran: 22 harness configurations × 5 open-weight models × 62 terminal tasks × 4 attempts. 27,280 runs, 83B tokens. Each harness ran with default settings at maximum reasoning effort, each trial in a fresh sandbox. Costs are Baseten list prices times reported tokens.What we found: The lowest-cost harness changes with the model. Codex CLI ranks 1st on cost with GLM-5.3 and 15th with Kimi-K3. Paying more doesn’t buy passes. The harness with the highest pass rate ranked anywhere from 4th to 19th on cost. The model still sets most of the bill (93% of cost variation). The harness is the second lever. Starting prompt size doesn’t predict cost. Output tokens do. 64% of inference spend went to runs that didn’t pass.
  • johnrc
    [flagged]
  • amarinhodev
    [dead]
  • anon
    undefined
  • shiqimei
    [dead]
  • Olscore
    [flagged]