Inference engines explained

What you’ll learn

What an inference engine actually is, how the popular ones relate to each other, and how to choose the right one for how you intend to use a model.

The concept

An inference engine is the software that loads a model and runs it to produce output. It handles the maths, manages memory, applies your quantisation, and usually exposes the model through an interface or an API. Several of the names people use interchangeably sit at different layers, which is worth untangling.

llama.cpp is a foundational engine that runs GGUF models efficiently across almost any hardware. Many higher-level tools are built on top of it. Ollama wraps llama.cpp with easy model management and a local API. LM Studio wraps similar capability in a graphical app. So when you use Ollama, you are, in a sense, using llama.cpp with the sharp edges filed off.

A separate family exists for serving models at scale. vLLM and SGLang are built to keep GPUs busy across many concurrent users, rather than to run one model for one person. And MLX is Apple’s framework, often the fastest path on Apple Silicon.

Choosing one

• Easiest start, single user: Ollama. One command to pull and run a model, and an API other tools can use.
• A graphical app, no command line: LM Studio.
• Maximum control over inference: llama.cpp directly, when you want to tune context, offloading, and sampling by hand.
• Best performance on a Mac: try MLX builds alongside the GGUF versions.
• Serving a model to several people: vLLM or SGLang, not a desktop tool.

What can go wrong

• Using a serving engine for a single user. vLLM is excellent at concurrency and overkill for one person at a desk. Start with Ollama.
• Fighting llama.cpp when a wrapper would do. If you do not need the control, the raw engine is more effort than it is worth.
• Assuming one format runs everywhere. GGUF is the common local format, but MLX uses its own, and serving engines often want the original weights. Match the model file to the engine.

Next steps

With an engine chosen, revisit quantisation to get the best quality that fits your hardware, and browse the inference engines in the catalogue for the full picture on each.