HN

Comments (60)

• Menu

  mmastrac
  Amusingly constrained decoding is the same approach we use to turn an AR model into a /v1/systemone endpoint (I worked on the vLLM feature).Tool calling is fraught with peril, especially because there's nothing that necessarily says that a models tools are fixed in place for the life of its run. In some cases, tools are added mid-run to avoid a giant prefix miss and (at least last time I checked!), the industry has not settled on OpenAI-style chat message formats for tool call updates.I would absolutely prefer to run in a fully-constrained tool call environment, and would love if we hadn't YOLO'd the whole thing! The PTC-style tools sort of fix this by punting tool calling to JavaScript, but that's even harder to enforce, and models not trained on the style might take a turn of failed tool calls to get it right.
• Menu

  Aurornis
  Constrained decoding has been in vLLM for a long time, and constrained tool call decoding has been in vLLM for a while, too.The blog post is rather confusing because until you understand the entire situation it appears to imply that they fixed vLLM (what the title implies) but really they just needed to configure it correctly.To be fair, the course of constrained tool calling configuration in vLLM changed over time in ways that changed behavior, which is the real problem. The Ubicloud team deserves a lot of credit for having benchmarks in place to track performance and catch regressions for investigations. Some of the providers you find on OpenRouter have inexplicably bad performance relative to others and clearly do not care this much.However, if this article left you confused because constrained decoding and constrained tool call schema have been features in vLLM for a long time, jump to the end where they show what configuration changes they made.
• Menu

  hansvm
  I wish they'd published something about the actual impact on model quality. Schema-constrained decoding changes the distribution the LLM will output. I'd be curious how a back-up-and-retry method for tool calls (which also doesn't pollute the context and also has 100% tool-call accuracy) would compare. In their example stats (23/25 success rate), it'd be +9% tokens used for the tool call attempt itself, but that's +9% on just a small fraction of the total tokens, so if a retry solution gave any better accuracy I think it might win out.
• docjay
  That is bending the barrel to align the scope. What’s interesting is that the authors mention the cause of the failed calls, “…a model may generate a tool or field name that it has seen during the training, even if it does not exist in the current production environment. For example, it may generate search_web when the available tool is web_search, or use city_name when the active schema expects location”, but then failed to see that it’s also the solution. If you call it “web_search” but it calls it as “search_web” then just rename your function to “search_web” and let the problem resolve itself.Things get more interesting if you don’t give it a call syntax and just see what it does based on the name alone.
• bob1029
  > For example, when we served Kimi K2.6 with vLLM version 0.28, we got the expected tool call schema only 92% of the time. In other words, 2 out of every 25 attempted tool calls failed.100% success is critical for tool calling.The probability that the overall agent iteration is successful would scale very poorly even if we achieved 95%. With p95, if you need 50 tool calls that must execute sequentially to achieve a result, the chances that you won't have an error are only ~8%. There are ways to recover from bad tool calls (error feedback helps a lot), but if the agent is persistently using the wrong tools with the wrong args the overall system will not work very well.
• Menu

  jesdo
  I'm one of the authors of this blog post. Figuring out how to integrate the LLM, the inference and serving engine, and the constrained decoding implementation such that you get the intended output and overall quality definitely requires careful engineering and testing. Please feel free to AMA. :)
• phegler
  Okay, this explains the feeling that I had that vLLMs tool parsing is much more precise than for example llama cpp or Ollama, I still use ollama due to mlx which of my understanding is not offered on vLLM unfortunately.
• sroussey
  Constrained decoding came to JS land transformers last month for in browser structured output as well, also used for tool calling.https://github.com/huggingface/transformers.js/blob/main/pac...
• Menu

  draxen
  I really appreciate this. I hope LLM inference providers and Chinese LLM labs start using this latest vLLM version ASAP.I do things that require ~300 tool calls per prompt and I have been stuck with OpenAI and Anthropic since 2024.If people still wonder what moat OpenAI and Anthropic have: They got 100% tool-calling accuracy (using constrained decoding) and prompt caching right all the way back in 2024.
• max-privatevoid
  llama.cpp has done this since forever. I've used other engines including vLLM here and there, and I was quite surprised that many of them don't have this feature. Glad vLLM has it now.
• Menu

  roger_
  Does this work with Qwen 3.8?
• Menu

  creativeSlumber
  Can the same mechanism be used to make sure it outputs syntactically correct code?
• greeks_lab
  [flagged]
• sarmedshafi
  [flagged]
• arbor-group
  [flagged]
• michaelpo
  [dead]
• yaml_tinkerer
  [flagged]
• CodexOfWeights
  [flagged]
• d-ramdass
  [flagged]
• fr2029r
  [dead]