Quantisation
Also: quantization
Storing a model's weights at lower numerical precision to shrink its memory footprint, trading a little quality for a much smaller size.
Quantisation reduces the number of bits used to store each of a model’s weights, for example from 16-bit down to 4-bit. This shrinks the model dramatically, which is what allows large models to run on consumer hardware. The trade-off is a loss of precision that, past a point, degrades output quality.
Common formats are named like Q4_K_M, where the number is the bits per weight.
Q4_K_M (4-bit) is a frequent default because it balances size against quality
well. See the guide on
understanding quantisation
for how to choose one.