A mixture-of-experts model divides some computation among several specialized components called experts. A routing mechanism selects which experts should process each token or input, allowing the model to contain a large total parameter count without activating all of it for every request.
This design can increase model capacity while controlling inference cost, but it introduces tradeoffs. Routing must be learned and balanced, expert specialization may be uneven, and total parameter count can overstate the amount of computation used for one token. Active parameter count provides useful additional context.
ELI5
A mixture-of-experts model is like a large team in which different members are good at different kinds of work. A small routing part looks at each piece of the request and chooses only the experts that seem most useful, instead of asking the whole team to work on everything.
For example, one expert might be especially helpful with code while another is better with ordinary prose. Activating only a few experts can give the model access to many learned abilities without paying the full computing cost for every response, although poor routing can leave some experts overused and others underused.
How does a mixture-of-experts model choose experts?
A learned router scores available experts for each token or input and sends the work to a small selected group. Their outputs are then combined for the next layer of processing.
Why use a mixture-of-experts architecture?
It can give a model more specialized capacity without applying every parameter to every token. This may improve capability while keeping per-request computation lower than a similarly sized dense model.
