HN

Comments (49)

• Menu

  vova_hn2
  > The same model will benchmark very differentlyProviders probably serve quantized versions without disclosing it. Which is a real shame, because for certain tasks I would be perfectly willing to trade accuracy for cost. But, unfortunately, it is impossible to explicitly choose how quantized do you want your model to be, unless you are running it yourself on your own (or rented) hardware.BTW, does anyone knows if LLM Gateway suffers from the same issues? Currently looking at trying it, but haven't got to it yet.
• Menu

  joshstrange
  This squares with my, much much, smaller OpenRouter usage. It’s just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found.OpenRouter sells the idea of swapping being commodity providers but it couldn’t be further from the truth. Provider A is often not swappable for B or C (again, as this author found). It can be crazy-making as you sit there thinking “OpenRouter has no clothes right?! Am I the one that’s wrong?”.I love the _idea_ of OpenRouter and maybe Stripe can improve this situation but the only sane way I’ve found to use it is to tightly pin providers to the point I wonder if I should just use the providers directly.Without pinning you are in for a world of hurt and unreliability (varying model capabilities, speed, etc).
• dangoodmanUT
  > Reasoning models sometimes put everything in the reasoning field and hand back content: null, finish_reason: "stop". 345 completion tokens, HTTP 200, nothing to show the user.This is actually expected behavior. No content and no tool call is the same as content-only: the agent decided it's done. Anthropic has done this for a while.
• Menu

  bsaul
  That looks absolutely horrifying. What are the alternatives ??
• Macha
  I do use OpenRouter for my personal use and this matches with my experience. I still use it because of it's top up model providing a way to not get surprised by out of control costs and being able to switch models with one account, but it's very noticeable at times.The models I've mainly been using recently are GLM 5.3 Flash and GLM 5.3. While obviously all these models have some variability, GLM 5.3 Flash feels like it oscillates between "I can't believe it's not Sonnet", but it costs a fraction of that and "This feels like I'm back using GPT-4, why am I even bothering with an LLM?".
• mesmertech
  Yea I can believe all these. I've personally have been having issues on these points:"200 OK, no answer" - insane that openrouter's main feature is literally a fallback and streaming doesn't support 200 no content to fallback to another provider or smth."rate-limit by IP"... now it kinda makes sense why deepseek v4.1 flash rate limits me on prod but never seems to happen on local. Makes you have to basically pin Deepseek as provider, since I've never had 429 error on them
• seanieb
  I keep trying to sign up but they ban my account before I can use the service, and their support team doesn’t respond to my emails.
• Menu

  srcreigh
  Without pinning providers, you pay let’s say 2x more for agentic coding, since that many input tokens aren’t cached as you bounce between providers.
• fl0id
  God that was painful to read.
• ghm2199
  > The tool call is in the textNoob question: do good harnesses automatically optimize for bad tool calling behavior automatically?
• ofisboy
  I use Fireworks for a production app. Much lower volumes though. My only gripe is their serverless offering for reranking has only 1 model which is Qwen 8b and it's quite expensive.
• rpjt
  I never tested this but always suspected it. When I learned that providers differentiate themselves on how they optimize and host the model (otherwise, why would you choose one over another?), I figured some were less "give me the best possible experience" than others.
• Menu

  _ink_
  Wow, that explains a lot. I was using OpenClaw with open router, but stopped because how inconsistent the results were over the same prompt. I thought OC is at fault, never would have guessed that there are differences between providers.
• alexcz
  could not agree more one pet peeve of mine is that using strict json output also does not work on all the endpoints of the models so I have to go through them one by one try it out and then only whitelist them.
• CubsFan1060
  Another thing to be aware of -- apparently their billing limits don't work. Somehow someone was able to use my key from Singapore. It had a $10 daily limit and they were able to spend $100. Not only that, it got me blocked from all models for the frontier labs.On top of that, near as I can tell, there are no protections for your API key. No restrictions by country, IP, etc...
• Menu

  kinard
  great post, I just put $100 in credits on open router to try different models, I think maybe using the "real" provider is the way forward once I've spent my credits and decided which one I want (for now).
• Menu

  joelthelion
  Excellent post, thanks for the article. This is very relevant for anyone using openrouter and similar services seriously.I feel there is still a lot of progress to be made before we can really trust LLM providers.
• Menu

  jwrallie
  I noticed some strange behavior when I was setting it up to avoid providers that collect data, hence blocking default providers. Definitely it is worth filtering well tested providers, which is a feature OpenRouter provides.
• anon
  undefined
• Krisso
  Wish this had more insights on 'cost' - I found OpenRouter credits to burn up fast.
• bbor
  Wait WTF?! I thought performance was, well, performance, not efficacy! OpenRouter's UI in this section is remarkably broken and unclear at the same time, and I have no idea where the author got those clean charts. "AutoExacto" numbers (completely meaningless name) are available for 6 providers, two of which are the same provider, and none of which are even in the top half throughput-wise. When you click "+28 more providers", it just shows a clearly broken modal. And the the next two graphs are even worse. Really all of these graphs are pretty and completely 100% useless.I feel like it's absolutely insane that some providers serve the same model with much less efficacy. That doesn't make sense to me. What's going on?! I'm suddenly feeling intense shame for having routed all my non-subscription usage through them so far, and honestly some white hot anger that they would blatantly lie about something so important.What am I missing? Is this really true?
• anik200
  Good job
• abuds
  [dead]
• hadeer626
  Yes
• Menu

  npn
  As expected vibecoding bros cannot even read the manual properly.It is pretty trivial to pin a single provider for a model. Better yet, instead of calling the model directly, use presets instead. You can easily change the setting on openrouter without having to update your app every time.