GGUF
The file format, introduced by llama.cpp, that packages a quantised model into a single file for local inference across a wide range of hardware.
GGUF is the model file format used by llama.cpp and the
many tools built on it, including Ollama. A GGUF file bundles
a model’s quantised weights and the metadata needed to run it into one file, which
is what you download when you fetch a quantisation like Q4_K_M.
Its wide support is a large part of why quantised models are so easy to run locally: a single GGUF file works across CPU, NVIDIA, AMD, and Apple Silicon through the same engine. The main alternative you will meet on Apple Silicon is MLX, Apple’s own framework, which uses a different format optimised for its hardware.