Skip to content
local-ai

llama.cpp

Inference engines Advanced Open source Free

llama.cpp introduced the GGUF format and remains the reference for running quantised models on consumer hardware. Many higher-level tools, Ollama among them, wrap it. Working with it directly gives you control over context length, KV cache handling, offloading, and sampling that convenience wrappers abstract away.

Strengths

  • Runs well across a wide range of hardware, including CPU-only
  • Fine-grained control over quantisation, context, and offloading
  • The GGUF ecosystem and community support are enormous

Weaknesses

  • The learning curve is real, especially the build and flag options
  • You manage models and updates yourself, with less hand-holding
  • Documentation assumes a fair amount of command-line familiarity

At a glance

Licence
MIT
Pricing
Free
Platforms
macOS, Windows, Linux
GPU required
No

Links

Works with

See also

Our coverage

Entry last verified 15 January 2026.