HN

Comments (51)

• Menu

  amanzi
  No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?
• brainless
  I experiment a lot with local LLMs, particularly small ones like Qwen3.5 4B and 9B. I have build multiple experiments to make harnesses that use these models for code generation, planning, local search, etc.These are really good models but the harness has to be built around them. I have a ton of generated system prompts for specific purposes. Even parts of a SolidJS stack, for example Route management, has its own prompt. These are experiments but the results are real. If we build harnesses around small models, we can build a locally running WYSIWYG editor which works on plain text prompts.The performance, in simple tokens/second, is not the most important factor. For many private data points, like emails, I would rather have a local graph based search and LLM on top where the harness is specific to problems like calendar, contacts, finance, etc.I run all experiments on an 16GB M4 Mac Mini but coding agents building the harness are a mix of Codex, Claude Code and opencode.
• Menu

  jumploops
  My biggest problem with running local LLMs on my M4 Max/128GB RAM is the prefill latency.I've since acquired two DGX Sparks, and it feels so much snappier.
• crossroadsguy
  > <a href="https://omlx.app">oMLX</a>Is that supposed to be hallucination? The human or other kind. Feels like a made up URL. It's .ai, isn't it?
• mkagenius
  I tried the 1 bit model of Qwen3.6 27B on my M1 pro (16G) and got 13 tok/s with only 5G of ram usage.https://x.com/mkagenius/status/2093730391429685732(xcancel seems to have received a cease and desist)
• whatsThisBtn4
  Apple did great work convincing people their unified memory was good at AI. Even AI says Apple is the best of all time at marketing.Meanwhile the stock market has Nvidia at the top... Until everyone gets cuda.
• alexgoodhart
  I have an m1 Mac 64gb and look forward to trying this outNot many people share setup with actual setup handholding so that was very G of you
• Menu

  miles_io
  M4 Pro has been a solid performer for iterating on smaller local models. Much more convenient than spinning up cloud instances for dev.
• max979
  That M4 Pro is probably a beast for quantised models. My M2 Pro handles 34B just barely; what speeds are you seeing?
• xydac
  yes, share performance, numbers if you can, also i wonder if you figured out a way to do a 2way audio with local models, or even explored that. I have a very similar setup but not too happy with the token speed, will try omlx though !!!
• Menu

  mintflow
  Have a macmini m4 32G, not the pro version, previously everytime I tried local LLM is a bit disappointing, and I finally decide to not waste time and perhaps in the future invest a better hardware to server more modern and dense modelI am curious is what is the 80% request served by this setup, I was using it for OpenClaw which run serveral cron jobs that discover stuffs over the wide internet, check my support system's unanswered tickets, browser X and some social media for me to filter the valued ones(though I have to say even with GPT 5.6 sol, the quality is low for the timeline X sent to me)Btw, Tailscale is quite cool and did a good job, I was using it to serve the local LLM and connct the openclaw on a Linux Machine to it.
• tukHelix
  [flagged]
• shell0x
  [dead]