HN

Comments (86)

• Menu

  nikcub
  I'm glad there are more attempts at solving model routing, as costs (at API rates) has really become an issue. Some feedback:1. Reiterate the cache issue from other comments already here. there is a lot of optimisation in harnesses around caching and a proxy model blows that up2. Coding agents are model aware - they already route code discovery to mini / flash models, planning to heavy models, workflow design to ultra, implementation to mid / high etc. They know when they're exploring, planning, implementing, reviewing etc. and which model class to select and when it fails.With a proxy you're breaking this control loop and feedback. It doesn't know, for ex. that it just attempted with deepseek v4 and it failed, lets try Opus?3. How are you going to RL improvements and prevent the router becoming stale? You only have access to your own internal prompts and ~thousands of samples.This is RL'd on one orgs codebase. There are going to be a lot of prompts you haven't seen before and have no insight to on how to route correctly, and you have no insight into users HF to improve your own model. Orgs aren't going to share their traces with you, so you need other sources to train on and improveThere are also new model releases every week that you need to keep up with - whats the story going to be here4. Publish evals by running terminalbench / deepswe bench. Show us the performance / cost / time chart vs the other agent and model sets. If you can show gains there, you have a very simple value prop to sell where you can charge for a % of the saved costs
• Menu

  jakozaur
  It's rather hard to do at the proxy level with agentic coding, such as Claude Code or similar. These are long-chained sessions of tool use that heavily rely on prompt caching. Changing mid-flight is costly.It looks like much more context is required to decide on the best model (e.g., summarizing logs might use a cheap model, whereas you likely want Opus/Mythos/GPT 5.6 to debug multithreading logic). In an agentic system, a decision about the model may be embedded in the decision to orchestrate the model.
• Menu

  stpedgwdgfhgdd
  The thing I do not get with these routers is that you will have more cache misses (5min ttl). And if there is one thing i’ve learned; using the cache is crucial.How does this router translate to $$$ when developing?
• Menu

  g00k
  Man, I'm not so sure if I'd use something like this because the way I prompt already changes based upon what model I am using. I'm not convinced it would route to the right model based on my diction or whatever.
• Menu

  GodelNumbering
  This would not work in the way that shows any significant genuine benefit IMO. Caching and optimum routing of a single request are at odds with each other. Higher the distinct model count in a conversation, more cache misses you accept.Based on what OP said elsewhere in the discussion "threshold to switch to another model will be higher" means that essentially you reduce the workflow into two models at most. The two model primitive, one planner and one executor, is already sufficient for such a use case.For lower than 2 models, it's just a simple single model cache-preserving conversation which arguably doesn't need another layer. For larger than 2 models, you are likely paying a large aggregate cache penalty that negates most of the gains
• Menu

  nativeit
  This would have been neat back when I could afford enough tokens to even set it up properly. Now I’ve had to increase my GH Copilot subscription just to cover the bare minimum updates to a few websites every month, and I no longer do any test driving, or even recreational coding projects. I don’t have hundreds of dollars a month to plow into these products, so I’m rationing use, looking for better local options, and being much more discerning about where these tools actually save time. Precarious time to be alive…
• jpease
  Is this noticeably different than having your implementation planning phase break a larger task into sub-tasks, and recording the ideal model to use based on scope as part of the task definition?
• Menu

  peterbell_nyc
  I auto tune my prompts to a locked model version based on production data used as evals with holdback data. I think the use case for this would be one off interactive prompts? For now I just run those all against an Opus 4.8 MAX and I'm sure I could downtune, although for interactive my opening prompt isn't always reflective of my overall goals for the multi turn session.I'm just trying to figure out why on the fly routing would beat testing and tuning and locking models and versions for each class of call, with evals and auto tunes running to explore more possible models for commonly run classes of prompt over time . . .
• Menu

  matt_d
  Looks interesting!Out of curiosity, how does it compare with vLLM Semantic Router?For reference:https://vllm-semantic-router.com/https://github.com/vllm-project/semantic-routervLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Models, https://arxiv.org/abs/2603.04444https://github.com/vllm-project/semantic-routerFor instance, does it offer similar algorithms:- vllm-sr/auto: efficient, fast, balanced routing, similar in spirit to Fugu // Sakana Fugu — Multi-Agent System as a Model: https://sakana.ai/fugu/ - vllm-sr/fusion: panel-style multi-model reasoning and synthesis. - vllm-sr/flow: router-native workflow orchestration - vllm-sr/remom: multi-round reasoning over one or multiple models.FWIW, it does look good on https://routeworks.github.io/leaderboardRef.RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers, https://arxiv.org/abs/2510.00202, https://github.com/RouteWorks/RouterArena
• Menu

  lubujackson
  I notice Cursor already does something similar. Even if I have Opus 4.8 selected, it will trigger subagents using Composer 2.5. I like using Auto personally because it is pretty effective and deeply discounted, but at work I YOLO Opus high.I imagine a solution like this will eventually be an enterprise-forced solution because there is no reason right now for individual developers to be selective about model pricing. Even more important is non-tech users who do stuff through MCPs like "give me a full overview of all analytics" and let it chug for half an hour.
• Menu

  spqw
  This + making sure common requests are saved as reusable skills and scripts would probably save a large part of my token usageAs prices increase we will see more of these tools to optimise and make the best use of token budget
• Menu

  notatoad
  Is this talking to claude code, or to claude api (and paying api rates)? programatically routing requests through claude code sounds like a good way to get banned, just like the opencode and openclaw users.
• Menu

  jmalicki
  > with no noticeable differences in quality or velocity.Have you done any A/B tests on this with evidence? (That's one thing I'd be very interested to see for claims like this - I'm not necessarily doubting you, it just seems like it could be useful to understand claims of quality/efficiency)
• Menu

  pradeep1177
  So, how are you handling read/write caching? I mean, if I keep routing the next prompt based on the task weights? How about if I'm sending every 5th query to opus, which do expensive write cache?
• Menu

  asdev
  Large model companies will likely build this and make it better. It'll also be cheaper overall since they'll be subsidizing token cost if you use them directly vs third party router paying API costs
• Menu

  thandv
  This might be a stupid question, but can a extra added local llm help with the caching problem?
• zcw100
  Can't really win can ya? Scarce? They're driving up prices! Plentiful? It's all a big bubble!
• Menu

  k9294
  What about request caching? If you swap to a cheaper model mid execution it might cost more that to make multiple requests to the already cached provider?
• Menu

  reliablereason
  Wont this kill the kv cache?Also i am pretty sure neither open ai or anthropic leets you seed the agents own tokens.
• Menu

  alansaber
  "We reward the routing model when it selects an LLM that achieves the task successfully" sounds pretty oversimplified
• Menu

  treexs
  Ahh been working on the same thing for a while now but haven't launched yet
• Menu

  gautam_io
  This is cool!Will this use my Claude Pro/Max subscription? Or will it always use the API billing "pay as you go"?
• Menu

  suyash
  I would rather just use OpenCode - leverage AI models, even can host locally or paid ones with ease.
• Menu

  Reuben_Santoso
  this is impressive. genuinely better than most people appraoches with using LLM as another judge to help route. which just uses more tokens than saves
• Menu

  _pdp_
  Cool.. but I still don't get how this is going to save money. It seems to me that it might actually burn more money just because the whole system now seems to be coming from different LLMs.Also, small LLMs are prone to stop before completion, throw errors and produce loops. Is this factored in the design of the tool? I am not sure.edit: spellcheck
• Menu

  arendtio
  What is the difference from Cursors 'auto' mode?
• Menu

  debarshri
  It is funny. We are building something similar.
• Menu

  mkagenius
  We have created Murmur[1] which kind of works with your existing subscription (having API key is not mandatory). You can just tag @copilot @codex from claude code to delegate work to them. (it can also do it on its own too btw)1. https://github.com/instavm/murmur - Murmur
• Menu

  emilio_srg2
  but this means you work with API pricing rather than subscription pricing. Isn’t it better to use claude or codex CLI etc directly in terms of cost?
• Menu

  slopinthebag
  > At Weave, we write ~all our code with AIThis is probably not a very effective way of marketing imo. At least, it turns me completely off.
• Menu

  ai_slop_hater
  Isn't this more expensive than always using the same model, since, as I understand, by routing to different models you give up on cache?
• james-mxtech
  [dead]
• barmazoid
  [flagged]
• kumiko_studio
  [dead]
• randomuser558
  [flagged]
• gmziven
  [flagged]
• Menu

  iluvcommunism
  This is basically what I need, a router. I’m tired of changing intelligence & speed levels manually.
• Menu

  bijowo1676
  How come data privacy and confidentiality is not an issue with services like these?Do people voluntarily use these proxies/routers, knowing their prompts, outputs and code will be seen by other people ?I get it might be ok for personal projects, but for anything that makes money and is a part of business... this must be big no-no ?