Mixtral 8x7B
The first open-weights Mixtral sparse mixture-of-experts model with 46.7 billion total parameters, about 12.9 billion active per token, and a 32,768-token context window under Apache 2.0.
Mixtral 8x7B is Mistral AI's first published sparse mixture-of-experts open-weights release. A router selects two of eight feed-forward experts per token, so the model stores a large total parameter budget while keeping active compute closer to a 12.9-billion-parameter dense stack.
At a glance
- Family
- mistral
- Source type
- Open weights
- Released
- December 2023
- Authors
- Mistral AI
- Source
- Mixtral of experts
- Modalities
- Text
- Parameter count
- 46.7 billion total parameters
- Context length
- 32,768 tokens
What It Is
Mixtral 8x7B is a decoder-only sparse mixture-of-experts language model from Mistral AI, released on 2023-12-11 in the Mixtral of experts announcement. Mistral describes it as a sparse mixture-of-experts network where, at every layer and for every token, a router chooses two of eight expert groups to process the token and combine their output additively. The release ships under the Apache 2.0 license as open weights.Inputs And Outputs
Mixtral 8x7B is a text-in, text-out model. Official release materials list English, French, Italian, German, and Spanish support, a 32,768-token context window, and strong code-generation behavior in the base checkpoint. This page keeps those figures because they are the conservative operating points repeated across Mistral's release post and model card.Architecture
As a decoder-only sparse mixture-of-experts model, Mixtral 8x7B keeps the familiar transformer decoder skeleton but replaces each dense feed-forward block with routed experts. Mistral's release post states that the router selects two experts per token from eight distinct parameter groups, which is why total parameter count and active parameter count diverge. The diagram below stays at the decoder level—attention, normalization, and a sparse expert routing block—so it teaches token-to-router-to-selected-experts flow without turning every internal block name into an extra claim.Output
ProbabilitiesSoftmaxOutput
LinearTransformer architectureAttention sublayer containerSparse expert sublayer containerN×RMSNormGrouped-Query
AttentionRMSNormExpert
RouterEmbeddingRoPEInput
EmbeddingInput
TokensSoftmax to Output
ProbabilitiesOutput
Linear to SoftmaxRMSNorm to Output
LinearExpert
Router to RMSNormRMSNorm to Expert
RouterRMSNorm to RMSNormGrouped-Query
Attention to RMSNormEmbedding to Grouped-Query
AttentionEmbedding to RMSNormInput
Embedding to EmbeddingRoPE to EmbeddingInput
Tokens to Input
Embedding
Important Modules
The registry-backed modules below are the smallest set worth opening first when you want to connect this Mixtral page to the site's module reference. Mixture-of-experts routing is the defining feed-forward behavior here, while grouped-query attention, RMS normalization, SwiGLU, and rotary position encoding anchor the high-level stack shown above.Training
Mistral's release post describes pre-training on data extracted from the open web and training experts and routers simultaneously. The same announcement also releases an instruction-tuned Mixtral 8x7B Instruct variant, but this page does not restate fine-tuning recipe details beyond what those primary sources publish.Training regimes
No training regimes listed yet.
Linked papers
No linked paper pages listed yet.