HN

Comments (149)

• Menu

  Grombobulous
  I’m curious to know if these local AI setups are legitimately useful compared to cloud. I’ve struggled a lot to get something useful out of the hardware I have.I realize I’m somewhat limited (16GB RX 9070), but still, it seems really far off from the kind of experience even a basic $20/month subscription gets me.Any tips anyone might have are appreciated! I’d love to be local first and would be willing to buy hardware to get there.
• Menu

  setgree
  It's fun to see that even an extremely large company can find unexpected product market fit [0]. Per this article, "The company reportedly did not possess an engineering team dedicated to business customers or staff focused on developer relations, and lacked an enterprise AI strategy." That sounds insane in retrospect, but I think there's just inherent uncertainty in what people actually need and will use things for.[0]https://pmarchive.com/guide_to_startups_part4.html: "In a great market—a market with lots of real potential customers—the market pulls product out of the startup... The product doesn’t need to be great; it just has to basically work."
• Menu

  AdmiralAsshat
  Mac Mini's were really nice HTPC candidates, too, before the AI boom. Like all things genuinely useful and affordable, they were snatched from the hands of normal consumers by a bunch of schmucks chasing the latest gold rush.
• Menu

  paxys
  I really hope with Ternus taking the helm Apple starts to remember that it has products outside of iPhone.
• Scubabear68
  Not just the high end stuff. The Neo is sold out until late September on the budget end, it seems like it is a smash for HS and college kids.I hope Apple can take all this cash and do some stability releases like they used to do, bugs around things like Family Sharing, the painful "update" to Settings App, etc could all use a lot of love.
• ChrisMarshallNY
  Sounds like people want those bespoke servers that Apple has been rumored to have developed.
• Menu

  imagetic
  No they weren’t.
• ceejayoz
  Time to bring back the Xserves, I guess.
• Menu

  Xeoncross
  If I had to pick a product, I'd say an affordable 32GB mac would be the sweet spot for running local models that function well like Qwen 3.8.It's true, most people don't run models, but being the default platform for running open weights seems like it has plenty of advantages right now. Just like sales benefited from developers defaulting to MacOS for most open source languages like Ruby, Go, Rust, and TypeScript.
• compounding_it
  More customers is generally a good problem to have in most businesses. Just that the situation is very paradoxical given the supply shortages.
• jmyeet
  So for people who don't understand, there are two markets for Apple hardware in this space:1. Running an agent like OpenClaude. The $599 Mac Mini was an insanely good deal for this. I happened to buy a M5 Pro Mac Mini for $999 last year for other reasons. The equivalent is now almost $2000; and2. Hardware for running inference on local models. This to me is the far more interesting market because Apple has a real opportunity to disrupt NVidia's stranglehold on the market.With current architecture, the largest model you can reasonbly run is the amount of memory on the GPU and is a function of the quantization (eg int4, int8, fp8, fp16, etc) available and the number of parameters. NVidia aggressively segments the market. The most VRAM on a "consumer" card is 32GB on the 5090, which allows you to run ~31B parameter models.In comparison, the RTX 6000 Pro has only slightly more CUDA units than a 5090 but has 80GB of VRAM. A few months ago they were $10-11k. Now they're ~$16k.Macs use a shared memory architecture. Apple has previously sold Mac Studios with up to 512GB of RAM. Almost all of that memory can be used to hold much larger models without taking a penalty for interconnections between different GPUs or machines. Plus Apple interconnects between computers are actually relatively good by chaining TB5. It's still slow but it's about the best non-enterprise option available.But the previous Mac Studios just didn't have the raw FLOPS and memory bandwidth. The M5 Ultras are up to 1.2TB/s of memory bandwidth. M3 Ultra had ~900GB/s. RTX 5090s and RTX 6000 Pros are 1.8TB/s. The current best HBM3 NVidia DC GPUs are at 3.2TB/s IIRC. But the M5 Ultra has a claimed ~4.5x the FLOPS of the M3 Ultra.We don't have our hands on these yet but it probably means they are going to be much closer to a 5090. I expect ~50% of a 5090's inference speed. That may sound bad but it's actually really good because a 256/512GB Mac Studio can probably locally run the best Flash models. With NVidia hardware you'll need to spend many tens of thousands for that.We'll see what the inference speed is but I expect it to be usable. DeepSeek v4 Flash, for example, will be entirely runnable. We're not at DeepSeek v4 Pro local yet.
• anon
  undefined
• wseqyrku
  is this a manufactured demand meme
• Menu

  comrade1234
  I wish they sold something that could go in a colo - redundant power supplies, lights out management, etc. you know they have them internally...
• evanjrowley
  I imagine Apple could also grow their business in the EU by marketing to companies who want powerful AI features but can't leverage 3rd party AI services due to GDPR. If only they could come up with iOS, App Store, and developer policies that respect the sovereignty of the EU.
• Menu

  Whatarethese
  There are so many used M4 Mac Minis for sale on marketplace for $600+ now that the people I guess got bored of the local models and decided to see if they could make their money back as they probably bought them when they were $399 on sale earlier this year. I lowball them every time.
• jshier
  Now if only they hadn't discontinued the Mac Pro. Could be quite the AI machine with multiple compute GPUs at higher bandwidth than an external Thunderbolt enclosure.
• Menu

  hzwanip
  Off guard? :'D
• martythemaniak
  This may sound a little wacky, but one potential use case I'm considering is robotics. Say you want to use a fine-tuned mid-size VLM model right on the robot. You pretty much have to use the Jetson line, which has a great ecosystem and everything (depth cameras, lidar, SLAM, small segmentation models, etc) runs on it, but it's gets very expensive very fast if you want to run LLMs on it. The Jetson Thor lines are 3-5k depending on memory etc.One very efficient option today is to have the cheapest Jetson (Orin Nano) run the classical robotics stack, then have a base mac mini run nothing but the VLM. The Mac mini is considerably cheaper and faster at these workloads than the mid-range Jetsons.I think this wonky situation is because Apple us under immense consumer pressure to absorb the ridiculous memory prices, while the Jetson is aimed at "business" and much more likely to fluctuate with the market. Last year I bought a Jetson Orin Nano 8GB for $375CAD, today that official nVidia Amazon page is out of stock and other sellers have it listed for $900-$1100CAD. Absolutely bonkers pricing.
• Menu

  api
  Apple has a huge opportunity here to lead the market for machines to run local models if they step into it. Their stuff is already better than what nVidia is offering with stuff like the DGX Spark.It's a niche market but it's a market that overlaps heavily with professionals in the AI space and lead developers, so it's a market that gets them customers in those roles.If I were running Apple I'd call the RAM price bubble for what it is and temporarily eat some margin to offer machines with more RAM than competitors, especially these models that are great for edge AI, and capture market share.
• Menu

  taskoutputs2k
  just a real bummer that they raised the prices so much
• Menu

  jmclnx
  I am missing something in the article. From what I am reading, AI companies are so desperate for memory they are buying Apple Systems and other hardware and striping them for RAM and maybe other components.Is that what others read ?