Quantisation

Quantisation reduces the number of bits used to store each of a model’s weights, for example from 16-bit down to 4-bit. This shrinks the model dramatically, which is what allows large models to run on consumer hardware. The trade-off is a loss of precision that, past a point, degrades output quality.

Common formats are named like Q4_K_M, where the number is the bits per weight. Q4_K_M (4-bit) is a frequent default because it balances size against quality well. Below about 4-bit, quality tends to fall off more sharply, so fitting a larger model by crushing it very low is often a false economy.

The format also depends on where you run it. GGUF quants dominate single-user local use; GPTQ and AWQ are common on GPU serving engines; and some models now ship natively in a low-bit format such as MXFP4. Not all quants of the same size are equal, either: importance-matrix and “dynamic” builds protect the most important weights and hold quality better than a naive quant. Quantisation mainly saves memory, but a smaller model also moves less data per token, so it can improve throughput too. See the guide on understanding quantisation for how to choose one.