HN

Comments (159)

• Menu

  hypfer
  Old news by now, but you might not be aware that llama-server can do multi-model for a while now,Meaning that you (and by that I mean your AI agent that has read the llama.cpp code) can write an ini file pointing to your models with parameters optimized for the specific model on your specific hardware. (Optimized by you through testing. Not that AI)Then, any api client can just select a model and the system does the right thing.It's great software. It just works.__You just need to ignore the cargo culting commandline options on social media. But you should be listening to the devs.Have you already enabled ngram-mod (or rather just spec-default)? It is practically free.
• Menu

  karimf
  Not sure why it's on the front page now, but I highly recommend using llama.cpp for running AI model locally vs using other inference framework, unless you have a very specific requirement.ggerganov and the team have done a stellar job maintaining the quality while still being fast to implement new models/improvements.
• Menu

  imrehg
  llama.cpp works pretty well for me on the Framework 13 laptop, but the current era of "move fast, break things, rarely fix" (sorry, that's how it feels), bites here quite a bit.Two examples:- https://github.com/ggml-org/llama.cpp/pull/25863 Someone's few lines change broke the native (ROCm) support for the AMD GPU inside Framework (and other integrated systems), and any rollback or proper fix is pending for almost a month. Fortunately there's workaround (switching to Vulkan rather than ROCm devices), but both the way the bug was introduced and the way it is not fixed just doesn't give much confidencen- LM Studio is using llama.cpp internally for GGUF, they ship their own build with their closed source system as "runtimes". Their ROCm runtime does not enable the the AMD GPU inside the Framework, even thought the llama.cpp version would support it. So their runtime keeps telling me that there's no supported AMD GPU -- again, the solution is to use the GPU with the Vulkan devices. Not fixed since Jan at least https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1...I guess overall it's the worst runtime I've seen so far, except for all the other runtimes out there... I'm a fan, though in some cases I don't have enough knowledge, or I don't have access to fix things, and that feels like a bummer...
• Menu

  walrus01
  Anything that suggests curl into bash just plain sketches me out. (edit: I know, this isn't totally rational, it just seems weird to me. We download and trust a lot of software and run code from a bunch of package repositories as a regular activity...).Git clone llama.cpp and build it, it's not hard.https://github.com/ggml-org/llama.cpp/blob/master/docs/build...literally just a few steps for the basics:git clone https://github.com/ggml-org/llama.cppcmake -B buildcmake --build build --config Release
• Menu

  tosh
  I was a bit suspicious of the url but it is also listed on llama.cpp githubhttps://github.com/ggml-org/llama.cpp
• Menu

  pplonski86
  Yesterday I installed llama.cpp to test it with local AI Data Analyst that I'm building. I was also testing other open LLM providers: Ollama, Jan, vLLM, LM Studio. I had older NVIDIA card (RTX 3070) and llama.cpp instalation was smooth, contrary to vLLM which required me to reinstall CUDA drivers because by default it installed the latest one. I'm curious if there is a speed difference between the same open LLM model served with different runners.
• Menu

  prologic
  Is llama.cpp (and thus llama.app) really that much better than Ollama? I've Only ever played with Ollama, so geniously curious to hear other's real-world experiences.
• Menu

  equalsione
  If I said, “I want a setup that is usable for an agentic coding workflow, and it MUST be local”, what’s the smallest/cheapest option right now?It’s _technically_ possible to get agents running on all kinds of setups but there seems to be an (undefined) floor for useful setups.A lot of the stories people have about getting setups running on relatively low end hardware turn out to have huge compromises or run into issues on anything but trivial cases. I’ve found it hard to find a consensus. Or maybe I just don’t like the multiple thousand dollar price tags people are suggesting…
• teleforce
  Check this DLLM D project, a minimal, clean coding agent built directly on llama.cpp without Python, bindings or overhead [1].The blog on the DLLM development [2].[1] DLLM:https://github.com/DannyArends/DLLM[2] Teaching an AI to Know Itself: Building a Local LLM Agent in D:https://blog.dlang.org/2026/06/07/teaching-an-ai-to-know-its...
• car
  This MacOS app used to be called LlamaBarn. Really excellent to see the fast progress being made.Official repo, also has documentation how to configure server parameters:https://github.com/ggml-org/Llama-macOSSmall tip, install llama.cpp with brew before llama.app, which will pick up the existing llama.cpp. That way it's easier to stay up to date with llama.cpp, since llama.app is on a slower release cadence.Also, models installed with the hugging face CLI (hf) are picked up by llama.app automatically. The CLI will keep the model cache updated, e.g. when models get updated.Llama.cpp became part of Huggingface recently.
• TekMol
  I tried curl -LsSf https://llama.app/install.sh | sh and then llama serve -hf unsloth/Qwen3-4B-GGUF:Q4_0 Then I get: W load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden Terminated And the web interface says Server unavailable Maybe it gets killed by the OS because it uses too much RAM?When I try llama serve -hf unsloth/Llama-3.2-1B-Instruct-GGUF:Q4_K_M It seems to work. Nice.
• Menu

  nexawave-ai
  I think I can probably run Gemma 3 12B on my macbook M3 pro with 18GB. The question is, should I do it? This small model is probably not capable of doing a lot or advanced coding or reasoning. What else could it be used for, since it can run locally and privately?
• Menu

  dlcarrier
  I tried to run in on my Arc A770, but all of the binary releases I could find were compiled without OpenVINO support enabled. I tried compiling it myself, but after two days of the compiler running it failed.
• macwhisperer
  pro tip: if u are building ur own harness with python (recommended) , use "llama-cpp-python"...I started with "llama-server" and custom stuff around it, which is great for single model setups.. but for multi-model harness with quick switching, llama-cpp-python is peak
• Menu

  halyconWays
  llama.cpp is like the ffmepg of AI, and one of the reasons I so greatly dislike ollama is that the latter completely obfuscates that they're a rebrand of the former. Georgi Gerganov and team did all the hard work; ollama is langchain-like VC-bait with a HF download wrapper.
• qiine
  Ha the logo is an L made of negative spaces haaa.
• ivorymist_04
  Ivorymist_04
• jrm4
  Hey folks, a bit of a hijack; but I've taken to using the Kobold gui, which I'm liking much more than Ollama -- but are there any major benefits to going straight up llama.cpp? Yes, this is somewhat of a question about laziness.
• sylware
  Any success at transpiling it to C? Using the cfront transpiler improved with coding AI? :)
• jurgenburgen
  There’s now a `llama serve` command? I had to do a double take in case I was reading the `ollama` website.
• anon
  undefined
• Menu

  antonvs
  > No telemetryMust be tough not to be able to monitor your own models!(The odds that that tagline was AI-generated seem high.)
• Menu

  blahblaher
  and? whats the point of this? Doesn't everyone already know about llama.cpp?
• bobbylarson
  [flagged]
• fintuner
  [flagged]
• Serveurperso
  [flagged]
• billybobbildo
  [dead]
• whateveracct
  [flagged]
• anon
  undefined
• mojo-10
  [flagged]
• Menu

  redmoonx
  Llama.cpp team has failed to make their tech easy to install and use for years.Why can’t they figure it out???
• Menu

  helsinkiandrew
  I'm confused, is this from Meta? There's no attribution anywhere. Surely releasing an AI tool called llama breaks their trademark if not
• Menu

  larodi
  This site seems scam for not noting origins of llama.cpp and fails to quickly and clearly communicate it NOT being affiliated with GGML org.
• Menu

  bhouston
  It seems that llama.app is a direct competitor to ollama.comI can understand the desire for the llama.cpp project to want to own the end user relationship, it is true that previous to this they were a tool provider and not really owning the end user experience.