HN

Comments (104)

• Menu

  matthewfcarlson
  I have been bothering the VM team for years for VM GPU pass through. I worked on the Apple Silicon Mac Pro and it would have made way more sense if you could run a linux VM and pass through the GPU that goes inside the case!Sadly, as you can tell, they have not taken me up on my requests. Awesome that other people got it working!
• Menu

  Aurornis
  Excellent article.The game benchmarks are fun but the LLM improvements are where this gets really interesting for practical use. I love Apple platforms as an approachable way to run local models with a lot of RAM, but their relatively slow prompt processing speed is often overlooked.> Here you can see the big issue with Macs: the prompt processing (aka “prefill”) speed. It just gets worse and worse, the longer the prompt gets. At a 4K-token prompt, which doesn’t seem very long, it takes 17 seconds for the M4 MacBook Air to parse before we even start generating a response. Meanwhile, if you strap the eGPU to it, it’ll only take 150ms. It’s 120x faster.The prefill problem goes unnoticed when you’re playing around with the LLM with small chats. When you start trying to use it for bigger work pieces the compute limit becomes a bottleneck.The time to first token (TTFT) charts don’t look bad until you notice that they had to be shown on a logarithmic scale because the Mac platforms were so much slower than full GPU compute.
• geerlingguy
  I came into the post thinking it would be running a VM through the slow tinygrad driver... but this is much, much better.It'd be amazing if Apple would provide better support, and allow more than that 1.5 GB window to make this easier. Arm overall has some quirks with PCIe devices, but at least in Linux, it's gotten so much easier since most modern drivers treat arm64 as a first class citizen.
• Menu

  djmips
  > Because OpenGL is not well-supported anymore on macOS, the game is completely unplayable there, even with CrossOver. Ironically, it plays totally fine on a Windows PC, but this is a game you literally can’t play on Mac without this eGPU setup.I understand that this is true it seems that Doom does support Vulkan but you would need to add VK_NV_glsl_shader to MoltenVK. Probably much less work than what went into hanging an RTX 5090 off of a M4. Still, kudos to the scott and the local AI Inference speeds are pretty cool. What a crazy project! <applause>
• divbzero
  This is pretty impressive. My impression was that eGPUs simply do not work with Apple Silicon.(EDIT: Apple agrees with my impression. “To use an eGPU, a Mac with an Intel processor is required.” And, on top of that, the officially supported eGPUs were all AMD not NVIDIA. https://support.apple.com/en-us/102363)
• Menu

  mywittyname
  > As much as I hate to admit it, step one in most of my projects now is to ask AI about it. Maybe it’ll tell me something I don’t know.Or, more likely, it will tell you something it doesn't know.Reminds me of yesterday, when I was arguing with ChatGPT that the 5070TI was an actual video card. It kept trying to correct me by saying I must have meant a 4070ti, since no such 5070ti card exists.
• lenerdenator
  The lack of native games on Apple Silicon is one of the greatest crimes ever committed against computing.I got Fallout 3 working on my M2 MBP as well as it did on Windows back in the day. Temps were cool, battery was decent. If they sold my college years gaming collection (15-ish years ago) in a way that ran natively through GoG or Steam, I'd buy every single title.
• swiftcoder
  This is proper mad science, love it
• delbronski
  Nicely done! Glad to see real hacking is still alive in the age of AI.
• Menu

  arjie
  Wait, this is incredible. I have a spare 5090 lying around and run a claw-like on my M4 Mini. Just plugging it into some sort of 3D print frame for stability and plugging it into the TB port might get me a pretty viable tool for local inference. Would need something neat to ensure the power etc. is well fed.The problem is `max-num-seqs` and `max-model-len` fight each other, and unless you're in the pure single-client mode you'll need multiple slots so to speak.
• SamiahAman
  Very nice effort. This has incredible technical depth, particularly in the DMA and QEMU sections. I also like that you didn't oversell it as the ideal Mac gaming solution. I found the AI inference results to be the most fascinating. Overall, it was a great read.
• coder68
  This seems pretty useful for AI inference if it can pass Apple approval. I've wanted to use my Nvidia GPUs with a Mac Mini, this would enable it to run CUDA directly. Very cool!
• Menu

  frollogaston
  I'm guessing the x86 emu is cause Windows games are rarely built for ARM, right? Was kinda curious how an ARM VM would fare. Anyway awesome article.
• anon
  undefined
• Menu

  anon
  undefined
• semiinfinitely
  where did you get a 5090 I will buy it from you
• Forgeties79
  > step one in most of my projects now is to ask AI about it. Maybe it’ll tell me something I don’t know.Bingo. This is exactly how I use LLM. I like getting a gut check, seeing what the first recommendations are or if there is some deep flaw in what I think the approach is, and I almost never copy/paste whatever it spits back or just follow its instructions.
• Menu

  zer0zzz
  Once egpus work on Apple Silicon there will be little reason to own a pc
• anon
  undefined
• sharathdoes
  damn
• null-phnix
  i mean porbly
• sharathdoes
  lol, is there a list of games tho, which mac pro's can support
• Menu

  moralestapia
  Wow, phenomenal project and write-up, thanks for sharing it."no - not in any practical sense today, and "maybe" only in a very deep, borderline-impractical research sense."This is why humans will always rule over crappy LLMs.
• Menu

  nothinkjustai
  > As much as I hate to admit it, step one in most of my projects now is to ask AI about it. Maybe it’ll tell me something I don’t know.It’s these people, not the ones who refuse to use LLMs, who are as they say, “cooked”.