Throughput
Also: tokens per second, tok/s
How fast a model generates text, usually measured in tokens per second. It depends on the hardware, the model size, the quantisation, and how many requests run at once.
Throughput is the speed at which a model produces output, measured in tokens per second. For a single user, anything above reading speed feels instant; for a served system, total throughput across all users is what matters.
It is shaped by several things at once: memory bandwidth more than raw compute for single-stream generation, the model’s size, the quantisation, and how many requests are batched together. This is why a high-bandwidth card or a mixture-of-experts model can feel faster than its headline specification suggests, and why serving engines work hard to batch requests.