HN

Comments (78)

• Menu

  kfsone
  I feel we are caught in a "this is fine, pay more and we may turn down the fire" situation.The LLM itself produces one token. Some tool adds that token to the input and runs it again, flogging the horse. Downstream another tool, some kind of harness, tries to control this stream by injecting tokens into the context and then sending it to the inference tool, and then trying to pattern-match the output.Finally, there you are on CodePorn.yata paying for an agent to generate code, paying for an agent to tell you what's wrong with it, and paying for an agent to make it differently bad, and hopefully move on to the next task.If it still hasn't dawned on you that this isn't just a bubble, but a snake-oil-bubble-bath, just try to imagine the paradigm shift whereby you go on github.com, assign an issue to an agent, the agent fixes it by rewriting the application in Pascal but a reviewing agent catches that you wanted it to print a measurement in Pascals (pa), and you don't pay for the work or the review, you only pay for work that one or two reviewing agents determine is up to par.Nobody is going to do that because as soon as they test it they're going to have to do some math that won't make sense without admitting/realizing it's not some near-sentient, AGI rating 0.9 intelligence, it's just a text prediction algorithm that can pull out entire sentences when you use it to infer output on topics it trained on.
• Menu

  Lerc
  Cost per tokens is as valid as price per unit volume of fuel.Changing the fuel type, efficiency of your vehicle, driving distance, or driving conditions will all change how much it will cost you.Fuel cost per unit volume does not become meaningless just because you are neglecting all of the other factors involved. That would be throwing away the only data point you have been using.This is just asking for someone to amalgamate all of the factors involved into one simple, easy to game, index.
• Menu

  yreg
  Cost per token doesn't say a lot, but "Cost per benchmark task" is also meaningless if your task is difficult enough that the cheaper model has no chance of cracking it.
• SwellJoe
  Efficiency is the next frontier in LLMs, and I'm not confident the American companies are taking it seriously enough. DeepSeek, even in a naive API-calling loop, serves something like 80-90% cached tokens at an absurdly low price per token. Using an agent harness tuned specifically for their caching (Reasonix) pushes the cached tokens to 97-99%. DeepSeek is consistently among the cheapest models per task in my benchmarks, while also performing quite well. I'm still almost always using Claude for work, but for side projects, small stuff, etc. and anything better served by an API rather than starting up Claude Code (or `claude -p`) I'm using DeepSeek pretty often.Anthropic models also shut down on a lot of security-related work, which is what I've been spending a lot of time on lately. I expected Fable to refuse this kind of task, but even Opus 4.8 refuses to build a verification harness for security bugs, as that involves exercising a discovered bug to prove it's been fixed in an automated red/green way, which looks like exploit creation to Opus' guardrails. So, I have to use other models for that work, now, though most of the original benchmarks I built were built with Claude.
• Menu

  kpw94
  In the context of local LLMs on limited hardware I've ran to the exact same conclusion: "tok/s" isn't the most useful metric when my personal North star metric, given my fixed hardware is: Model smart enough to execute my goals _in the minimum amount of time_.Some models I tried (Mistral I think) had better tok/s, and roughly same billion parameters / scores on various benchmark... But they were _so_ verbose, that they generated many more tokens compared to a Qwen model of same caliber to answer the same thing.So even though it had better generated tok/s, because so many more were generated, the clock time was longer.And this compounds over mutli-turns: more generated token means more context used in the next turn (until some compaction or something runs)
• Menu

  Jcampuzano2
  I keep trying to convince directors and executives at my company to look past the cost per token amount but they refuse to do so. Those are the only things that actually give any sort of measurement of the monetary value of a token by these labs, and so its what many go by.For example there's some benchmarks that show that Opus for any task that requires a higher than `high` level of effort, may have actually been cheaper to use Fable on low even though the cost per token is drastically higherSimilarly with GPT 5.5 vs Opus. They simply look at the dollar amounts the labs assign to each model and run with it.But part of the issue compounds on the fact that there are many people who simply default to the smartest model/effort and don't actually vary their model per task. So in some sense I don't actually blame them very much.
• usef-
  Sonnet 5 makes more sense when you pretend the higher thinking efforts don't exist. (His test was on xhigh)Anthropic's own release announcement mentioned that it's less cost competitive per task than Opus at higher thinking levels. It's significantly cheaper at lower levels though.I'm wondering if this is going to be a universal pattern of smaller models: they're less smart, so to achieve the same benchmark results they have to think a lot more and hence become expensive.Benchmarks force models to solve the problem entirely by themselves, requiring thinking. But if you pair them with a smart model (who thinks and solves beforehand) they won't need to solve the hard parts and can run on low/med. I suspect that was Anthropic's intention.
• Menu

  janalsncm
  Pricing based on tokens always seemed a little weird to me.“Tokens” was and still is an engineering concept. The fundamental unit of transformer encoding and decoding.But I have a sinking feeling that many AI developers think “tokens” got their name from the same idea as “virtual tokens in a casino” which is more related to product pricing and business.
• Menu

  pier25
  On top of that isn't it strange that if the LLM makes a mistake you're still charged for those tokens?They're selling "intelligence", automation, etc but if the service doesn't work as expected the user has to pay for that.
• Menu

  dools
  It’s not meaningless at all: every query returns usage and I can calculate the cost.EDIT: this is like saying hourly rate or salary is meaningless. Different people have different output. You have to evaluate performance.EDIT2: just pray the LLM providers don’t start taking Patrick McKenzie’s advice and start charging based on “value delivered”
• Menu

  dbuxton
  As well as cost-per-task I think it's worth thinking about speed, especially in non-coding contexts that benchmark less cleanlyWe've started trying to do some comparison videos to capture more of the UX vs speed vs cost stuff e.g. https://www.linkedin.com/feed/update/urn:li:activity:7479891... which one of my team did for my LinkedIn account (disclaimer: marketing)(In this particular case Deepseek was way slower than GPT 5.5 but I think that's because it installed Libreoffice half-way through the task!)
• Menu

  nathanyz
  The Sonnet 5 comment is spot on. Even Anthropic's own graph initially showed lower performance at higher costs. Only thing I notice about Sonnet 5 is that it does appear to hand off tasks to agents more frequently similar to Fable, but of course nowhere near the quality of Fable. My guess is that Opus 5 will do similar but just isn't ready yet.
• vfalbor
  I believe the future lies somewhere in between. I'm working on a hybrid application to reduce our company's token consumption. It runs on our data center's computing infrastructure and on laptops in our community. You might be interested; you can check out the code if you're interested: https://github.com/vfalbor/hibrid
• anon
  undefined
• tidbeck
  Related to this, for our use case, setting thinking to high instead of low made tasks complete faster and cheaper (Gemini 3.0 flash).Other aspects are caching, often at 0.1X cost, where providers really differ in how efficient they are (Anthropic really good, Google not so much) and how chatty a model is (costing output tokens).
• BugsJustFindMe
  Price per token is meaningless for more reasons than this, because all of the provider monthly subscriptions price tokens _extremely_ differently than their per-token billing rates. It's stupid to look only at what you get when paying more than you need to for a given service.
• shireboy
  Yup. I’ve been evaluating several on openrouter and find token cost meaningless for my work. I haven’t found a great alternative, though the “cost per task” he uses makes some sense.
• koolba
  This reminds me of cpu benchmarks vs actually running games and measuring FPS.
• danielmarkbruce
  An LLM is an extremely complex thing used for all manner of purposes. The hope that there would be some simple pricing construct that would map nicely to value provided is a pipe dream.Pricing per token is at least reasonably straight forward. If you aren't getting value, you don't use the service. One doesn't buy a Ferrari and then complain that in their town Ferrari doesn't help them pick up women and hence it should cost less.
• mikebs1
  The variable missing from cost-per-task: which tasks shouldn't be hitting an external API at all.
• Menu

  bArray
  The only metric that really matters is 'profit per amount invested'. This is very difficult to quickly evaluate, and therefore we resolve to use simplified metrics such as cost per unit tokens.The point at which the metrics become meaningless is when others become aware of them, and begin to optimise for them. Lines per code is is not a bad insight for development activity, only when the developers are not aware of the metric. Price per 1M tokens became meaningless when LLM providers started to optimise for it. It seems to be that Sonnet 5 is optimised to score well on AA intelligence whilst seemingly having a lower price per 1M tokens.I think generally we are in an AI bubble, and it will at some point pop. The numbers simply don't make sense. I would gamble heavily on local cost per task to survive the LLM winter. Given that hardware is pretty much a fixed overhead, you probably want to optimise for task per kW - that's where I'm betting.
• zeroonetwothree
  Well, not totally meaningless but certainly can be misleading.
• yalogin
  Another thing I noticed is the llms perform efficiently/effectively only under the optimum circumstances. May be this just a Claude issue, but when the session goes on for very long the effectiveness drops drastically and I start getting bad answers. This is especially true for design and debugging. Wonder how that ladders up to the token usage
• shay_ker
  cost per benchmark task is definitely interesting!i've always wanted cost per prompt, but even that has too much variation.
• teravor
  tool use is another factor, every time the agent uses a tool the entire context is priced at cache rate on top. the same happens when it asks you for input.
• cyanydeez
  locally, im about 80k every 30 minutes for a project. can run deer flow to pump them tokens.but yeah, its the 80s LOC metric since quality isnt captured
• dandaka
  Another important benchmark would be — cost per benchmark task using subscription tokens. Since most of us are using subscriptions and cost per token there is quite different from API costs.
• sleepybrett
  would be nice to have these benchmarks so they can be run against models like the qwen family, gemini, etc.
• Cappybara12
  [dead]
• Menu

  lifeisstillgood
  My advice to any CEO / individual - throw your hands in the air and bring it in-house. Yeah the performance can dip depending on what GPUs you can salvage these days but the uncertainty over price is almost nothing compared to the uncertainty over the effective use of AI. It’s not just coding (do I go partly agentic or all out Steve Yegge). This is all over the enterprise - do we parse every email, rewrite PowerPoints? Or just stop using PowerPoints at all. Do we throw LLMs at the mess of wikis and word docs, do we pretend that the policies no-one has ever read actually are how the LLM should think or is it how the work actually gets done - barely documentedThe uncertainty of how to use this vastly vastly outweighs the price in a data centre - so buckle up, buy enoughbGPUs to experiment at a known cost and one day you will find the approach that gives you 10x returns - at that point pay any price per token but not till then