Models· 3 min
What is Mixture of Experts (MoE)?
MoE models have many specialist sub-networks but activate only a few per token — big brain, small bill.
Key idea
A router picks the top-k 'experts' for each token, so a 400B-parameter model may only use ~40B per forward pass.
Example
Mixtral 8x7B, DeepSeek-V3 and rumored GPT-4 all use MoE to boost capacity without proportional cost.
What to remember
- Better capability-per-FLOP than a dense model
- Harder to serve — experts must be sharded across GPUs
- Training stability requires load-balancing tricks