Mixture of Experts
A sparse feed-forward layer that routes each token to a small subset of expert layers instead of one dense shared block.
A mixture-of-experts layer, often shortened to MoE, routes each token through a few specialist expert layers instead of one dense shared feed-forward block, trading sparse activation for extra capacity.
At a glance
Released
January 2017
Authors
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, et al.
Optimizes
- Parameter Capacity
- Compute Efficiency
Example models
What It Is
A mixture-of-experts layer, often shortened to MoE, replaces the single dense feed-forward network in a transformer block with many parallel expert layers plus a router. For each token position, a gating network scores every expert and sends the token state to only the top-k experts. The chosen expert outputs are then weighted and merged back into one updated token state.Why It Exists
MoE optimizes for parameter capacity without forcing every token to pay for every expert. Instead of making one dense feed-forward block larger, it keeps activation sparse and lets different tokens use different expert subsets.How It Works
The token state first goes through a router. The router picks a small top-k expert set for that token. Each selected expert runs its own feed-forward sublayer, and their outputs are blended into one result. The next token in the sequence can activate a different expert set even though the block slot in the architecture stays the same.Token state h_tRouter scores expertsTop-k expert selectionExpert 7 hidden layerWeighted sum of chosen expertsUpdated token stateExpert 19 hidden layer...Expert 41 hidden layerMost of the expert pool stays inactive for this tokenToken state h_t to Router scores expertsRouter scores experts to Top-k expert selectionTop-k expert selection to Expert 7 hidden layerTop-k expert selection to Expert 19 hidden layerTop-k expert selection to ...Top-k expert selection to Expert 41 hidden layerExpert 7 hidden layer to Weighted sum of chosen expertsExpert 19 hidden layer to Weighted sum of chosen experts... to Weighted sum of chosen expertsExpert 41 hidden layer to Weighted sum of chosen expertsWeighted sum of chosen experts to Updated token state
Math Or Compute Schema
The dense feed-forward formula below is the baseline that activates one shared path for every token. The mixture-of-experts formula adds routing and weighted expert selection to the same general slot.- Hidden state for token position t.
- Projection into the wide hidden layer.
- Projection back to model width.
- Pointwise activation inside the dense hidden layer.
- Hidden state for token position t.
- Router that scores experts for the current token.
- Operation that keeps only the highest-scoring expert choices.
- Weight assigned to expert e for this token.
- Expert-specific feed-forward layer chosen by the router.
Compared To Nearby Modules
Compared with dense feed-forward variants, MoE changes not just the hidden path shape but the activation pattern itself. The key contrast is dense-for-every-token versus sparse top-k routing.| Comparison dimension | Mixture of experts | Standard feed-forward | SwiGLU |
|---|---|---|---|
| Path shape | Router selects a few expert layers, then merges them | One dense expand -> activate -> project path | Value branch multiplied by a SiLU gate branch |
| Active compute per token | Only top-k experts activate for each token | All hidden units in one shared feed-forward block | All hidden units plus a learned gate |
| Main tradeoff | Adds much more parameter capacity, but routing and balancing add system complexity | Simple dense baseline, but every token pays for the whole block | More expressive dense path, but still no sparse expert capacity |