HN

Need help?
<- Back
Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU

Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU

joshcsimmons

Comments (67)

  • armcat
    Something to keep in mind with aggressive quantisation, is that it's been empirically proven [1] that (1) quantisation has non-linear performance behaviour, and (2) generally the peak size-to-performance ratio occurs at 4-bit; anything lower and you will see significant reduction in performance.Also, I've been playing with speech models for the past 2 years, and for me anything below 8-bit is a huge drop in quality; so quantisation for LLMs and for other bespoke modality models, is not 1:1.[1] "Reliability Scaling Laws for Quantized Large Language Models", TUM, https://arxiv.org/pdf/2607.10855
  • antirez
    To put this on perspective, consider that with 128GB of unified RAM it can run even fast: https://x.com/antirez/status/2119392658661749175 Like, coding-agent acceptably fast. See later in the reply for the prefill at ~300 t/s but can go even faster for larger prefills.
  • boredatoms
    The quantization is probably the most important thing to know and its not clear at all from the dense word salad
  • akie
    At which quantization? That's the only thing I want to know.EDIT: I think it's 2 bit? If so, would be curious to see the quality of the output. Not great, I think.
  • SillyUsername
    Interesting. There are quite a few forks of llama.cpp where streaming weights has been successful, or special quants / decompressing before processing.- Bonsai Qwen 3.8 27B- Strata- Pulsar- This...I just wish upstream Llama.cpp would accept the PRs faster - they build multiple times per day, but their PR backlog was 1.7K last I checked, which is a ridiculously impractical number to review with a few volunteers.I watched the Qwen 3.8 Bonsai feature and it took over 3 weeks from creation to be reviewed, and even then it doesn't include the patches for bugs, that they fixed downstream and simply gave up pushing to upstream because the process was so laborious.So they end up with delayed or missing model support, and half working implementations like Bonsai's Hadamard Folding, and new models lose out on this kind of optimisation.I suppose this is fine if you intend your project to be the reference project for forks and encourage this kind of specialisation, but I don't think that's intentional.At least I can use llamaswap to juggle the inference engines and models on the fly, and abstract the fact I need to install 3 or 4 different ones.
  • dgreensp
    How can CPU be fast enough? What does “modchip” refer to?
  • superkickstart
    I wish people would clean up the ai spammed emoji from the github readme. The readability is just horrible.
  • real_faxenoff
    This is a very impressive achievement in an unexpected direction. I'm really looking forward to trying out qwen on this system.
  • robotswantdata
    Prefill of this approach (prompt processing) is far too slow.The readme is practically unreadable ai slop as well
  • lastdong
    Strata, Project Maya, the FP8 3x speedup on memory-bound workloads on Ampere cards and now llama-mod chip. I’m amazed with all these optimisations and thinking outside the box. Well done, don’t stop!
  • spwa4
    Anyone else finding these AI generated pages really hard to read?
  • GypsyKing716
    Just because you CAN load it, doesn't mean you SHOULD load it. Sparse models n such ... but I digress.
  • anon
    undefined
  • intoXbox
    This is quite an obnoxious Readme.md with AI slop. It wraps around llama.cpp but without seeing what platform assumptions it makes its unclear for other readers whether this is worth testing.
  • throw10920
    "2-bit quant" is the "in mice" of "look at how big a model I can fit on this hardware".
  • ctrlaltdileep
    hm... this is the first time. TIL something from the HN comments more than the actual article. I'm relatively new to the AI space and the idea of quantization is something I'll have to take forward when evaluating future models.
  • walrus01
    I very recently spent a lot of time using deepseek v4 flash 0731 before it was superseded by newer more capable things (glm 5.3-flash, v4.1 flash 1017, etc) and I am exceedingly skeptical of anyone that claims to get good QUALITY results out of it at that size. Whatever they're doing to shrink it will make it worse than an unsloth Q2XS quantization of it or equivalent. That size of model needs something like 150GB of RAM to run properly with full unquantized kv cache and not with a ton of training data discarded.I remain exceedingly skeptical of people who are claiming to run fairly large models in ridiculously tiny sizes. For comparison a 'regular' version of v4 flash 0731 has file sizes more like this: https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUFEven the 97GB Q2XS will show very significant divergence from the full precision thing. Running a lobotomized model just to say you're doing it is really not accomplishing anything special.
  • outfinity
    It is 2 bit? anybody tested this? is any good?
  • yieldcrv
    all the pieces are coming into place for the Apple M7 chip, 2028 is going to be great for useful local inferencefrontier labs are going to need to have some new tricks up their sleeve that are inaccessible by prosumer level compute
  • junaid0016
    [flagged]
  • ankur_misra
    [flagged]
  • hb_b19yaati
    [flagged]