Mixtral 8x22B
A larger open-weights Mixtral sparse mixture-of-experts model with 141 billion total parameters, about 39 billion active per token, and a 64,000-token context window under Apache 2.0.
Mixtral 8x22B is Mistral AI's larger sparse mixture-of-experts open-weights release. A router activates a subset of expert feed-forward blocks per token, so the model stores 141 billion total parameters while keeping active compute closer to a 39-billion-parameter dense stack.
At a glance
- Family
- mistral
- Source type
- Open weights
- Released
- April 2024
- Authors
- Mistral AI
- Modalities
- Text
- Parameter count
- 141 billion total parameters
- Context length
- 65,536 tokens
What It Is
Mixtral 8x22B is a decoder-only sparse mixture-of-experts language model from Mistral AI, released on 2024-04-17 in the Cheaper, Better, Faster, Stronger announcement. Mistral describes it as a sparse mixture-of-experts model that uses only 39 billion active parameters out of 141 billion total, offering higher capacity than Mixtral 8x7B while keeping sparse activation. The release ships under the Apache 2.0 license as open weights.Inputs And Outputs
Mixtral 8x22B is a text-in, text-out model. Official release materials list English, French, Italian, German, and Spanish support, a 64,000-token context window, and strong mathematics and coding behavior in the base checkpoint. This page keeps those figures because they are the conservative operating points repeated across Mistral's release post and changelog.Architecture
As a decoder-only sparse mixture-of-experts model, Mixtral 8x22B keeps the familiar transformer decoder skeleton but replaces each dense feed-forward block with routed experts at larger per-expert capacity than Mixtral 8x7B. Mistral's release post emphasizes sparse activation patterns that make inference faster than dense models of comparable capability while storing a much larger total parameter budget. The diagram below stays at the decoder level—attention, normalization, and a sparse expert routing block—so it teaches token-to-router-to-selected-experts flow and capacity-versus-activation tradeoffs without turning every internal block name into an extra claim.Output
ProbabilitiesSoftmaxOutput
LinearTransformer architectureAttention sublayer containerSparse expert sublayer containerN×RMSNormGrouped-Query
AttentionRMSNormExpert
RouterEmbeddingRoPEInput
EmbeddingInput
TokensSoftmax to Output
ProbabilitiesOutput
Linear to SoftmaxRMSNorm to Output
LinearExpert
Router to RMSNormRMSNorm to Expert
RouterRMSNorm to RMSNormGrouped-Query
Attention to RMSNormEmbedding to Grouped-Query
AttentionEmbedding to RMSNormInput
Embedding to EmbeddingRoPE to EmbeddingInput
Tokens to Input
Embedding
Important Modules
The registry-backed modules below are the smallest set worth opening first when you want to connect this Mixtral page to the site's module reference. Mixture-of-experts routing is the defining feed-forward behavior here, while grouped-query attention, RMS normalization, SwiGLU, and rotary position encoding anchor the high-level stack shown above.Training
Mistral's release post positions Mixtral 8x22B as a continuation of the open Mixtral family with stronger reasoning, multilingual, and coding behavior than earlier releases. The same announcement also releases an instruction-tuned Mixtral 8x22B Instruct variant, but this page does not restate fine-tuning recipe details beyond what those primary sources publish.Training regimes
No training regimes listed yet.
Linked papers
No linked paper pages listed yet.