HN

Need help?
<- Back
Running Kimi K3 on a M1 Max

Running Kimi K3 on a M1 Max

tito

Comments (68)

  • knighthacker
    Local AI is going to win. Not because it's cheaper btw.
  • antirez
    SSD streaming on an M5 Max 128GB: https://x.com/antirez/status/2082136334160818528Soon decent speed across two Mac Studios with 512GB of RAM.
  • Azantys
    0.01 tk/s is unusable for anything, you would wait a whole day for just 1000 token of output, what is the point of projects like this?
  • throwitaway222
    how much money in hardware would it cost to get 100tps?
  • acmnrs
    The title should probably be edited to specify "M1 Max" instead of "M1 Mac". You aren't running K3 on a base M1 anytime soon. Either way, still a very impressive project.
  • ALLTaken
    Exactly my machine 64GB M1 Max So happy about this! ♡idk how people access (soldout) and even afford 512GB RAM MacStudio's. Isn't it $40k or so?
  • mips_avatar
    Would be interesting to see how fast it would be on 4x mac studio 512gb machines.
  • nlessard
    Anyone who knows the state of NVMe hardware more than me know if this would obliterate the lifespan of your drive? Seems like the biggest limitation to me (some people are probably fine with letting their Macs churn over the weekend).
  • jrhizor
    Super cool, and I appreciate the upfront speed disclaimer
  • mindwok
    Does anyone else feel like the writing is on the wall for a future of local models? Spamming data centres everywhere, powering them, having to commit insane capital to hardware, all the effort to serve inference over a network reliably - when here we are with a frontier model nearly running on a laptop.Local AI on your device seems like a much more likely future to me than datacenters in space. For inference at least, training is another story.
  • hmokiguess
    Now set it up with an agent and a permanent `/goal` to say it cannot stop until it has solved for speed, then leave it on and livestream so we can all see when it becomes exponential. Could have the Eternal Jukebox playing in the background!
  • denysvitali
    60s/token - if only there was a way to drop that "s" this would be amazing
  • tjwebbnorfolk
    > ~60–76 s/tokenI don't know if I'd call this "running"
  • walrus01
    tokens/second, no, more like seconds/token
  • als0
    Says it requires a 2TB disk? Must it be internal NVMe?
  • lostmsu
    under 0.02 tok/s
  • piterrro
    Will it fit on ESP32??
  • onesandofgrain
    Cool gimmick
  • brcmthrowaway
    Why not train another smaller LLM to give the same answers as Kimi K3?
  • fenestella
    [dead]