llama.cpp vs Ollama vs vLLM: which engine?
Ollama is built on llama.cpp, and vLLM is a different kind of tool altogether. Here is what you gain by using llama.cpp directly, when vLLM is the right answer, and when it is overkill.
By Leon Mallett
What you’ll learn
How three of the most common inference engines relate, and which to reach for. The short version: Ollama is built on llama.cpp, so choosing between them is about convenience versus control, while vLLM answers a different question entirely, serving a model to many users at once. Our inference engines explained guide is the overview; this one goes deeper on the head-to-head.
The short answer
- One person, one machine, least fuss: Ollama.
- One person who wants full control over how the model runs: llama.cpp directly.
- Many people hitting one model at the same time, on NVIDIA GPUs: vLLM.
None of these is the best; they are built for different jobs. Using vLLM for a single user is as mismatched as using Ollama to serve a busy production API.
At a glance
| Ollama | llama.cpp | vLLM | |
|---|---|---|---|
| Built for | Easy single-user local use | Flexible single-user local use | High-concurrency serving |
| Relationship | Wraps llama.cpp | The underlying engine | Separate project |
| Hardware | Broad: CPU, GPU, Apple Silicon | Broad: CPU, GPU, Apple Silicon | NVIDIA GPUs (data-centre focus) |
| Model format | GGUF (managed for you) | GGUF | Full-precision and quantised weights |
| Control over inference | Sensible defaults, less to tune | Full: context, offload, sampling | Serving-level, tuned for throughput |
| Many concurrent users | Not its job | Not its job | Its whole point |
| Ease | Easiest | Moderate, more to learn | Hardest, server-oriented |
Ollama or llama.cpp?
Ollama is llama.cpp with the sharp edges filed off: it manages downloads, picks a sensible quantisation, and exposes a local API, all behind one command. For most people on their own machine, that is the right trade.
You gain three things by using llama.cpp directly. First, control: you set the context length, how many layers are offloaded to the GPU, and the sampling parameters by hand. Second, choice of quantisation: you pick the exact build rather than accept a default. Third, newest features first, since llama.cpp is upstream of the tools built on it. The cost is that you manage those details yourself. A good rule: start with Ollama, and move to llama.cpp when you hit something its defaults will not let you change.
When is vLLM the right answer?
vLLM is not a bigger Ollama; it is a serving system. Its reason to exist is throughput under concurrency: keeping NVIDIA GPUs busy while many requests arrive at once, which is what you need when a model sits behind an application used by a team or the public. For a single person at a desk it is overkill, and its hardware assumptions (NVIDIA, server-class) make it the wrong starting point for casual local use.
Reach for vLLM when you are serving one model to many concurrent users on NVIDIA hardware and need the GPU used efficiently. Our serving models at scale guide covers that case. For anything smaller, a local tool is simpler and enough.
On performance numbers
You will find many benchmarks comparing these engines. Treat them with care: results swing with the model, the quantisation, the hardware, the context length, and how many requests run at once, so a figure from one setup rarely transfers to another. The safer guide is to match the tool to the job, as above, and measure on your own hardware if speed is critical. We do not publish engine benchmarks we have not run ourselves and dated.
Next steps
- The overview, if you have not read it: inference engines explained.
- Getting more from llama.cpp directly: the llama.cpp catalogue entry.
- Understanding the formats these engines run: understanding quantisation.