Inference engines

A fast inference library for running GPTQ and EXL2 quantised models on consumer NVIDIA GPUs, known for high throughput at low bit-widths.

A self-contained, single-executable inference tool built on llama.cpp, running GGUF models with a built-in web UI plus image, voice, and vision features.

The inference engine that much of the local AI ecosystem is built on. A C/C++ implementation that runs GGUF models efficiently across CPU, CUDA, Metal, Vulkan, and more, with fine control over how inference behaves.

Apple's open-source array and machine-learning framework, built for Apple Silicon's unified memory. With MLX-LM it runs language models efficiently on Macs, and is often the fastest path to good local performance on Apple hardware.

A high-performance serving engine notable for fast structured output and efficient KV-cache sharing. Its RadixAttention makes it particularly strong for prefix-heavy RAG and multi-turn chat, where reused context is the main lever.

NVIDIA's library for extracting peak inference performance from NVIDIA GPUs. It compiles optimised engines for a model and hardware pairing, delivering the best throughput and latency available on supported cards, at the cost of setup complexity.

A high-throughput inference and serving engine built to maximise GPU utilisation and handle many concurrent requests. Its PagedAttention and continuous batching make it the common choice once you are serving a model to more than one user.