HN

Comments (151)

• tarruda
  There's quite a few tangential features that must be implemented correctly or risk affecting the LLM output in significant ways.Parsing/encoding is one example: A couple of months ago I've debugged a reasoning loop bug in Step 3.7 Flash on llama.cpp that was caused by the parser capturing an extra `\n` as part of a reasoning block. It was something that only manifested at longer multi-turn agentic sessions, and the extra linefeed was steering the model into making reasoning self corrections that only got worse with longer sessions (more details about this issue: https://github.com/ggml-org/llama.cpp/issues/24181#issuecomm...)No inference engine is perfect, but I feel that llama.cpp is the most reliable way to run language models locally.
• Menu

  big-chungus4
  I saw some guy streaming how he was deploying qwen3.8 37B on his local setup. Well, he was asking Claude to do it. It took him two hours of passing errors to Claude for the endpoint to start working, he then started testing it against DS4 Flash when Qwen had thinking disabled and Claude messed up sampling parameters, it was an absolute pain to watch
• Menu

  jonplackett
  I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.
• ThouYS
  It is good that someone is having such a deep look. This is not exclusive to LLMs in the least. Every non-trivial program depends on hundreds of little details being correct.That is also why I recommend including health checks in such programs. Some function that checks that all assumptions hold. That could be an endpoint, an automated test, a periodic diagnostic job, etc...
• a11r
  Even a 4-bit quant of Qwen3.8 27b is indistinguishable from Gemini 3.7 flash in our internal tests. With an RTX5090 card and ninfer, you can get ~800 TPS token generation (c=8) and ~140 Tokens per second single stream.
• utopiah
  Comments are mostly showing off M5s and 5090s without addressing the article.
• Menu

  runeks
  Somewhat off topic, but I've started wondering if we can actually make local LLMs feel smarter than the frontier closed models, by post-training it for your specific use case.Say Company X has a software product which consists of a million lines of code, including a ticket for every bug and new feature for this piece of software. Then wouldn't it make sense to try using an open weight model but post-train it on that specific code base while using the tickets to teach the model about past bugs and features. Not sure exactly how, but it could involve doing reinforcement learning solving a past bug on that historic version of the code base, and rewarding the model if it comes up with the correct solution (as defined by the linked PR which fixed the bug).So essentially post-training your local open-weight model using reinforcement Learning with Verifiable Rewards (RLVR) on your software products history of bug reports and their ultimate solution. And the same for new features.
• walrus01
  Much of this is why I stick to the rule of:a) Don't quantize your KV cacheb) Don't run quantizations of the LLM that are worse than the best available Q8 (the largest possible file size unsloth GGUF for a given model like qwen 3.8 27B as an example). I would rather things go slowly but I have confidence that it's doing things more accurately.
• synthrakx
  I have been using open source LLMs locally for the past 5 months like Qwen, Llama, DeepSeek and others, and I have also noticed that current local models feel significantly dumber than closed source commercial models like ChatGPT, Claude, and Gemini. One main reason I think is the amount, variety, and quality of original authentic data on which they are being trained on, and also the training method plays a significant role in the performance difference between local open source models and closed source commercial models.
• Menu

  InvertedRhodium
  I’m running Qwen3.8 aggressive uncensored Q4_K_P on a 4090 in a loop against the 2026 CrackMe CTF challenges.Using oh-my-pi in a prebuilt environment that I let Qwen build too.Codex wouldn’t even look at the files - literally, as soon as it read something with CTF it shut down. Didn’t even offer to fall back to a dumber model.
• nullpoint420
  At least I'd be in control of model quality vs. when Anthropic decides to randomly drop the quality of their offering
• djoldman
  "I can't wait to run this new sota model locally. I'll just use the quantized version that is certain to be better than [other model I'm running]."This is fast becoming one of my top old-man-yells-at-clouds pet peeves.Reported performance metrics are ONLY good for the exact model weights.Quantizing a model, or changing it in any way, requires new evaluation to know how well it performs.Quantizing a good model doesn't mean the quantized version is good.
• heywoods
  So to what extent does this apply to cloud hosted LLM’s? Are there benchmarks that score models across cloud providers? My experience using LLM’s during day time vs evening sessions has felt “night and day” and I’ve chalked that mostly up to it must be my imagination or just the general indeterministic nature of LLM’s. Sessions resumed after a day away also feel “dumb” sometimes so I can see an aggressive kv cache eviction policy playing a role if it’s reasonable to extrapolate what the article is saying about local inference.Are things like kv cache eviction policies and memory budgets shipped with recommended configurations based on the hardware and software serving the inference requests? or are they configured dynamically by the cloud provider hosting the model to manage multi-tenant load?
• Roark66
  The problem is benchmarking. Not everyone has a 500k token workstream of the model they are setting up for the first time to run it against 10 different config and compare differences.And if you download benchmarks from the net they are likely poisoned by models being trained on them.
• Abishek_Muthian
  After 3 years of running local models I think the model which fits unquantized (BF16) in the VRAM is the best model for general purpose tasks; fine-tuned SLMs or utilities based on non language models for solving a specific problem (e.g. TTS,STT,RMBG etc.) have been the best use of local AI for me.Local models for coding, is just not worth the effort IMO; unless of course you have the hardware to fit it unquantized in your VRAM.
• Menu

  catlifeonmars
  > I will make you read the really long unpleasant version with math.This is the version I want to read :)I assume it is unpleasant in spite of the math, not because of it?
• Menu

  JacobJack
  > And the comparisons in this post are not going to be running some 2.58-bit-gguf-in-ollama with a couple test prompts.Genuine question : is there something fundamentally wrong with Ollama ?I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly).I thought the main advantage of VLLM was better concurrency management (better batching).But if the quality of the interference itself is an issue, then maybe I should reconsider my choice.
• Menu

  throwdbaaway
  > Both the NVFP4 and AWQ W4A16 failed to properly close their tool calls ...If I understand correctly, this failure mode is just not possible with llama.cpp / ik_llama.cpp, which enforces token generation to follow the grammar once a tool call is detected.> ... and botched Cisco command line syntax (the correct command was ‘show arp’, while they executed ‘show run’)But this failure mode can still happen.Anyway, NVFP4 and AWQ W4A16 are generally regarded as low quality quants. IQK/Trellis quants from ik_llama.cpp and EXL3 quants from exllama should work better.So, perhaps the lesson here is "don't use vllm at home"?
• mkhalil
  "Why LLMs ARE dumber than they appear" is much closer to the reality I live in.
• fenestella
  The section on system prompts and context window management is spot on; most people don't realize how much the default quantization in popular runners degrades logic compared to full FP16. I'd be curious to see if the author has benchmarked the impact of KV cache compression on longer context reasoning, as that usually seems to be where my local Llama 3 setup starts to fall apart.
• OsamaMustafaa
  I believe whoever lays down the best structure around LLM will take the lead. Proven in Anthropic vs OpenAI.
• anon
  undefined
• IronWolve
  sglang, 150+ tok/s on a 5090 in ubuntu 26.04 via wsl. gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090, dspark, medium reasoning, 96k context.Using opencode and it built a old fashioned arcade vertical shooter with no issues.Images are ok'ish, just had grok create updated images, and it came out great.
• arcanemachiner
  Jeez, I thought I could get away with q8_0 KV cache. Guess not.
• Menu

  happybox2016
  Rate limiting on free LLM APIs is usually where the pain lies. I've seen 5 concurrent reqs hit 20K/day limit in under 2 hours. Does anyone know a free API that still allows some reasonable concurrent requests?
• mrgaro
  Any DGX Spark users in this thread? What's your favourite model to run on it?
• redbear2026
  And here i am with a unsloth UD Q4_K_XL quant. Its a good model still.
• woadwarrior01
  RTN quantization of weights
• giuscri
  what a beautiful non-slop article!!! (i’m not ironic)
• shevy-java
  No. They are dumb.
• CodeWithLeo
  [flagged]
• Menu

  anotherCodder
  most of the time when a local model feels dumb its not the quant, its the chat template. a lot of gguf mints just drop the template from the metadata and the runtime silently falls back to chatml. model still talks fine so nobody notices, it just gets noticeably dumber. got burned by this myself serving qwen, now i grep the gguf for the template tokens before i blame anything else. second place is sampling, people run whatever defaults their ui ships instead of what the vendor recommends and then compare that to benchmark numbers that were run greedy or with the official settings
• futurist_hp
  [flagged]
• koba3
  [dead]
• luciana1u
  [flagged]
• deadcatfound
  [dead]
• goglidesdev
  [flagged]
• Menu

  paulyy_y
  [flagged]
• nineteen999
  [flagged]
• Menu

  PrinceNaroliya
  [flagged]
• Menu

  luciana1u
  The punchline nobody wants to hear: your local model isn't dumber, it's just finally talking to you the way you actually sound.