KV cache
Also: key-value cache
The memory a model uses to remember the tokens already in a conversation, so it does not reprocess them at every step. It grows with the context length, and is a major, often overlooked, consumer of VRAM.
As a model generates, it keeps a running store of intermediate values for every token so far, called the key-value cache. This is what lets it continue a long conversation without re-reading everything from scratch each step.
The catch is memory. The KV cache grows with the context length and with the number of requests running at once, and it sits in VRAM on top of the model’s own weights. A model that fits comfortably at short context can run out of memory once conversations get long, which is why “fits” is not the same as “runs well”. Plan for the cache, not just the weights.