A mixture-of-experts model divides some computation among several specialized components called experts. A routing mechanism selects which experts should process each token or input, allowing the model to contain a large total parameter count without activating all of it for every request.
This design can increase model capacity while controlling inference cost, but it introduces tradeoffs. Routing must be learned and balanced, expert specialization may be uneven, and total parameter count can overstate the amount of computation used for one token. Active parameter count provides useful additional context.
Acronyms and aliases
MoE acronymMoE model variant
General terms
Related terms
Frequently asked questions
How does a mixture-of-experts model choose experts?
A learned router scores available experts for each token or input and sends the work to a small selected group. Their outputs are then combined for the next layer of processing.
Why use a mixture-of-experts architecture?
It can give a model more specialized capacity without applying every parameter to every token. This may improve capability while keeping per-request computation lower than a similarly sized dense model.
