HN

Need help?
<- Back
Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

marcobambini

Comments (142)

  • fennecbutt
    Idk about you guys but I'd find 0.5t/s useless. Even for long tasks.I'd rather just shell out the money to offload as much as possible to say 2x 4060ti 16gb with tensor parallelisation. Anything but that low token rate.This is the sort of thing I'd expect in 20 years for some cyberpunk esque "turtlebot" that thinks at 0.5t/s, is solar powered and performs some menial civic maintenance background task like cutting grass, or scrubbing pavements. Or the "slowbot" that sits in the garden slowly pruning a bonsai, only just keeping up with the growth of the young plant.
  • ruler88
    Really cool!!For all of the other commenters - this project isn't about practicality today. Obviously this isn't gonna be as good as using a cloud provider. But the tool draws a line of what is possible. Combination of making the models more efficient, and making local machines more capable can one day get us to a world where very high quality local models are economically feasible.
  • ikurei
    They say it's a waste that you pay for the tokens and then the inference provider pays for the electricity. Isn't that how everything works? I pay cucumbers and the farmers have to pay for the water and the fertilizer...I hope that reasoning is an after-the-fact justification by the LLM that wrote this.It's a ver interesting idea and I wouldn't mind trying it out, but with a smaller model. At 0.5t/s and reading many gigabytes from the SDD every second... I wonder if this wouldn't be extremely practical if targeting a 500gib or 250gib model, something that is still outside most consumers' laptop.
  • bgirard
    Approximate calculation is putting the cost at ~$5 per million tokens (assuming 42W sustained, 20¢/kWh), and that's excluding hardware and other costs.
  • mappu
    Standard llama.cpp can mmap the gguf, so it'll stay on disk if it doesn't fit on memory, and the kernel page cache will ensure the hot parts ("resident trunk") stay resident.What's the benefit of a custom implementation at all?
  • hddambo
    Pretty soon we'll have 3T param models down to 1 bit. They'll be able to tell you whether they're off or on.
  • marcobambini
    Thanks guys for all the comments, I am going to rewrite the README (without using an LLM)
  • pja
    That README hits all my “this is authored by an LLM” instincts. I presume the codebase is also written by an LLM?
  • nialv7
    are they allowed to use the "SQLite" name?
  • bilsbie
    Claude might as well be .5 tok/s. I end up waiting several minutes and what it tells me could usually be summarized in under 100 words.So I could potentially live with this if it was concise.
  • JakaJancar
    Love that it’s embeddable, I needed exactly this for the upcoming app I’m shipping!
  • tim-projects
    Once the tech catches up to the point that we can accurately select the right model for the task, then this ends up becoming a valuabke thing to have. You would spin it up sparingly as part of an automated discovery process maybe for 30 mins a day. And the rest of the time is spent using tiny models.I could see a future like that.
  • theanonymousone
    For me at least, it makes way more sense on a DSv4 Flash size and ~10 tps and I would definitely try (or try to try) such a thing.The concept and the proof of it is great, of course.
  • roundup
    How does this project compare to https://github.com/gavamedia/deltafin ?
  • herf
    So if this Mac uses 30-50W, that's 40-60 tok/Wh...vs maybe 80k for a modern GPU cluster? So that's about 1000-2000x more power for the SSD streaming, unfortunately.
  • Catloafdev
    Neat! But, what do you do with a 0.5tk/s LLM?Have you tried running it via llamacpp or other software that supports naive SSD offloading to compare speeds?
  • OutOfHere
    Be advised that the firm behind it ("sqliteai") had a nasty history of using non-open source licenses, e.g. Elastic License. I advise against using anything by them for this reason even if this project currently has an open license.
  • righthand
    I couldnt find anything explaining the name of this company on their website but is it okay that they’re riding on the name of an open source tool?SQLite code itself is public domain but I’m not sure about the name.
  • SSilver2k2
    This sounds a lot like what the colibri project did for GLM-5.2. I'm a fan so keep at it!justvugg.github.io/colibri
  • 1vuio0pswjnm7
    28 Jul 2026 21:35:19 UTCRunning Kimi K3 on a M1 Maxhttps://github.com/gavamedia/deltafinhttps://news.ycombinator.com/item?id=49090233[ok]29 Jul 2026 02:01:04 UTCKimi K3 running on any device locallyhttps://github.com/RightNow-AI/local-kimihttps://news.ycombinator.com/item?id=49092591[dead]29 Jul 2026 13:49:32 UTCShow HN: Waste Run the full 2.78T-parameter Kimi K3 on a laptophttps://news.ycombinator.com/item?id=49097512[flagged] [dead]29 Jul 2026 14:38:35 UTCSelf-hosting Kimi K3: 20% more hardware cost, 20% better task resolutionhttps://aistack.imec-int.com/blog/gpu-self-hostinghttps://news.ycombinator.com/item?id=49098130[ok]29 Jul 2026 14:57:14 UTCRun Kimi K3 on a local computerhttps://github.com/sqliteai/wastehttps://news.ycombinator.com/item?id=49098395[ok]29 Jul 2026 15:41:50 UTCShow HN: A new engine to run Kimi K3 on a laptophttps://news.ycombinator.com/item?id=49098966[ok]29 Jul 2026 19:54:13 UTCKimi K3 for local use (1.56TB > 594GB) compressed by Unslothhttps://huggingface.co/unsloth/Kimi-K3-GGUFhttps://news.ycombinator.com/item?id=49102192[ok]29 Jul 2026 20:02:40 UTCKimi k3 now runs on one consumer GPUhttps://twitter.com/Akashi203/status/2082555972380401852https://news.ycombinator.com/item?id=49102291[ok]30 Jul 2026 06:12:55 UTCShow HN: Run Full Kimi K3 with 29 GB of RAMhttps://github.com/sqliteai/waste/https://news.ycombinator.com/item?id=49106591[ok]30 Jul 2026 13:05:09 UTCKimi k3 run on RTX 5090https://github.com/RightNow-AI/local-kimihttps://news.ycombinator.com/item?id=49109455[ok]30 Jul 2026 14:01:44 UTCA new inference engine to run Kimi K3 2.78T parameter with 29GB of RAMhttps://marcobambini.substack.com/p/the-waste-inference-engi...https://news.ycombinator.com/item?id=49110183[ok]30 Jul 2026 16:52:58 UTCRunning Kimi K3 on a local computerhttps://github.com/sqliteai/wastehttps://news.ycombinator.com/item?id=49112587[ok]
  • anon
    undefined
  • cadamsdotcom
    Dear creator: you didn't ship the first draft of your code - why did you ship the first draft of your README??
  • logicallee
    Interesting project. The headline number (29 GB of RAM) is for 4k context.From what I've read elsewhere, Kimi K3 is quite verbose in its thinking. At the quoted rate, it would generate only a total of 1.8k tokens in 1 hour. Is that enough for it to get any thinking done and produce output on more complicated prompts?
  • nharada
    It's too bad Optane PMem is dead
  • whatsThisBtn4
    People who bought CPU instead of Nvidia are coping with this reality.I'm not sure how this happens. Reality distortion field?Like, it is common knowledge at this point right?
  • cjbprime
    Does it not use Metal, on macOS? Would it be faster if it did?
  • ElectricalUnion
    Ok, this one will take just 30h (compared to that other project that would take 6.25 days) to start writing output tokens after you say hi in Claude Code.
  • broadsidepicnic
    Waste is/was a p2p client back in the 2000s
  • jpecar
    Where can this 1tb k3.waste be downloaded?
  • yieldcrv
    looks helpfulyour readme is overly verboseagents don't need that and its extremely low signal for humans tootell your language model to get it to the point
  • mrdootdoot
    Fun. I see the novelty.
  • brcmthrowaway
    How does it compare to dsv4?
  • anon
    undefined
  • albertomr3
    [flagged]