<- Back
Comments (185)
- hecturchiRight...- Tiny context size or hours to load it - Forget about thinking and preserve thinking and the benefits because well, that uses tokens. - Low quants reduce accuracy heavily - Flash attention with who knows how much draft ensures it makes the same mistakes all the time and cannot call tools, format output or follow basic guidelines reliably. Otherwise 2x-4x slower. - K/V quants set to who knows what to make everything even less accurate.But sure it "fits" and hallucinates just fine.Useful would be combinations with:- Full context size so it can code and think a bit. - Draft MTP <= 2 so it doesn't trip - Q4 quants or better so its accurate - q8 cache or better so it stays accurate. - 20 token/s so it finishes while reviewing previous step. - 1000 tokens/s context load so compactions don't waste 10+ minutes. - And enough left RAM for 50+ context checkpoints so that it can progress quuckly.Closest you have is Qwen3.6-35B-A3B-MTP.Latest gens (Qwen3.8 and co.) are just too big for low specs. 27B dense models seem to be ok for integrated >=92 GiB RAM.Source: I have low specs and tried them all (to do useful stuff not to see if they "run").
- a11rI'm a little skeptical of going below 4-bit quants due to the potential for significant degradation in quality. I'm running 4-bit quants on an RTX Pro 6000 rented for approximately $1/hour and getting about 1.2 million tokens out and 40 million tokens in per hour with caching. The quality of 4-bit quant is good enough for difficult but well-scoped coding tasks. Here is the inference stack I am using: https://www.reddit.com/r/BlackwellPerformance/s/FrKwk3GoDK
- deadbunny> Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.And I thought piping to bash was bad
- sneheshtI tried it and it worked surprisingly well. On my machine (Nvidia 4090, 128GB DDR5, Ryzen 7950x3d) I'm getting 124 tokens per sec, thought to share it here.https://huggingface.co/Qwen/Qwen3.8-Flash-Next
- hemedanmertI need a version of this that runs 3.8 27B on 8 gigs of VRAMamazing project, congrats on the launch
- lxeWhy isn't this type of expert caching in the native llama.cpp yet? Why do we need a separate codebase?
- kamranjonDwarfstar already supports this, curious how it compares, but I use the q4 quant daily and it works really well.https://github.com/antirez/ds4/blob/main/docs/MODELS.md#qwen...
- SuperV1234We're getting closer and closer to the day we can have an Opus-like model running locally. The dream!
- sweetboyI think it would be great if you could try models with lesser parameters that could fit on 6GB VRAM-ish, which could work for "gaming laptops" as well.
- TepixAll headlines about LLM performance MUST have the quantization also mentioned in the headline.You know, so you're not wasting your time like in this post.
- mmaunderMore great work on local model but you’re still losing a lot. Down to 2 bit quantization and the coder model throws away half the MoE experts. In a world where anything is better than nothing, this is a net win. But we have a way to go still.
- Luker88Currently I am running llama-cpp with `Qwen3.8-Flash-Next-UD-IQ3_XXS` on an old ryzen 8845HS with 96G of ram (and no dedicated graphics card) at 7tk/s and ~60tk/s filling, max ~120K context window.Surprisingly useful as long as you can leave it running a couple of hours at the very least.While huge models will still be better I think the general availability of RAM might be the downfall of AI companies.
- TepixQ2 quantization. Not interested.
- prettyblocksI've been playing with this on a 3090 and it FLIES. Does a pretty good job too on the tasks I've thrown at it (php code base security audits).
- nialv7There are so many AI generated inference engine for local models now, each of them are generally narrower but they are all faster than llama.cpp. Maybe llama.cpp needs to rethink their strategies...
- b212I tried to run Qwen 3.6 27b locally a few months ago and all those synthetic tests do tell you something and quite a lot of people were very excited about that model but honestly? It wasn’t even close to default mode in Cursor or Sonnet at the time.I’m all for local models and I do want them to be the future but I wonder when, and if ever, we’ll catch up to a level of, let’s say Opus 4.6. I guess it’s currently doable but requires $50k hardware?
- piloochMy goto private setup, runs ~50t/sex on a dgx spark with sglang, nvfp4. Excellent model.
- fsiefkenI wonder if a higher Qwen3.8-27b quant could beat or match these lower < 16/24/48/64G Qwen3.8-Flash Next quants given similar quality.What speed are you willing the sacrifice to debug/program for more complex jobs faster?Then there are also these quants; https://huggingface.co/IsValorum/Qwen3.8-35B-A3B-Distill-MLX...
- paulezPretty impressive so far, but needs more testing.It is more useful than Qwen3.8:27b (which is already quite good) and runs faster on my 7900 XTX / 64 GB DDR4 system.Local LLM is getting more exciting every day!
- mark_l_watsonQwen 3.8 Flash Next is amazing. I only have a 64G Mac so I have to run Sushi project’s 3 bit quant. Amazing results with pi-dev. More for fun than anything else, but I am trying to do as much as possible with local models, now rarely falling back to a paid deepseek-4.1-flash API.Progress on running local models has been amazing.
- gdevenyiI had this working with the FreeToken inference engine a month ago when they launched.https://github.com/FlashML-org/FreeToken
- ryan_glassAnyone know how it compares to GLM 5.3 for real world use?
- cesarvarelaIt's funny that most of the AI industry is built around the assumption (which is most likely true) that it is not possible to run SOTA models on current consumer hardware.Imagine if someone managed to run an Astra- or Fable-level model on a 5090 at reasonable speeds.
- zkmon>> The model is a team of 24,576 small specialists ("experts")That's a neat number (576 is the square of 24). Ofcourse it must have come from 24 * 2^10.
- ai_ja_naiI am not getting it: I see a fp2 quantized model going on a 5090 with 64GB of RAM at 90 tops with -10% accuracy over original model. How is this supportive of the claims?
- bt1a80 t/s w/ 3090s and 3.05bpw exllamav3
- louskenlm studio bionic, unsloth, now this... it would be nice if it worked at least in one of those without installing another component
- hypferIs these another one of those repos where it turns out that claude decided to quant the KV cache to q4 or smaller?The Readme doesn't say, but it's all AI generated, so..
- Jeeetendragetting it to fit is impressive, but i'd want to compare the smaller quants on a real coding task before picking one. how much quality do you lose going from IQ3_S to Q2_0?
- NeywinyI don't like that some configuration is fine via arguments and others by environment variable. I've noticed LLMs like doing this. And even more, like hallucinating such things. To me the advantage of AI coding is that the boilerplate of command line arguments and passing them around becomes trivial instead of tedious.
- esafakHas anyone calculated the effective intelligence of these quantized models?I think publishing benchmarks with quantized models should become standard practice.
- shieldx0013That’s an impressive claim — running a 125B parameter model at ~100 token/s on a single RTX 4090 would require some
- 0xbadcafebeeLol, sure, if you quant it to hell (Q2) it'll go real fast...They even link to a Q1 quant (Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF) with half the experts ripped out. The idea is it'll go much faster and supposedly benches to not-terrible results. But the problem is you can't rely on it for real world long-horizon coding because that's where reasoning comes in, which is why you want the other layers.It turns out there's still no free lunch. Either get enough VRAM for a Q4, or use a much smaller model. Lobotomizing a larger model just to say you can run it fast isn't useful.
- quietFalconNice, though generation speed is the easy half for MoE offload, what's your prompt processing look like at say 16k context?
- pannyI'm far less interested in how good a big expensive model is on hardware 99% of people can't afford and would rather see what runs best on a chromebook or mobile phone with 8GB of RAM.
- lsbThere’s other slop projects to run of Qwen, like ds4, would be interesting to see a comparison
- apiContinued progress on these fronts is another reason I think the data center buildout is a bubble. It posits that AI use and growth will require an ever-increasing amount of power and floor space, which contradicts the entire history of computing. The high cost of data centers is largely electricity and floor space, which means there's a huge forcing function to make both the silicon and the software more efficient.
- srikanthbuilds[flagged]
- cyberboss[flagged]
- thinkthunkWhy would I want to use a model that has no knowledge of Tiananmen Square or Winnie The Pooh?
- tracerbulletxThe interesting thing here is that it's a model specialized fork of a generic inference engine that unlocks consumer hardware to run a bigger model with useable performance than it could before.