Mixture-of-experts
Also: moe
A model design where only a fraction of the parameters are used for each token, so a large model can generate at closer to the speed of a much smaller one.
A mixture-of-experts (MoE) model is split into many sub-networks, called experts, and for each token only a few of them are active. A routing mechanism picks which experts to use. This means a model can have a very large total number of parameters, giving it the knowledge of a big model, while only using a small “active” fraction per token, giving it the speed of a much smaller one.
The name you often see, such as 30B-A3B, spells this out: 30 billion total
parameters, 3 billion active per token. In practice that model generates roughly as
quickly as a dense 3B model, while drawing on the capability of a 30B one.
There is a catch for local use. All the experts must still be loaded into memory, even though only a few are used at a time, so a mixture-of-experts model needs the memory of its full parameter count, not its active count. You get the speed of the smaller number and the memory requirement of the larger one.