Llama 3
Meta's open-weight Llama-family decoder-only transformer release, spanning dense and instruction-tuned checkpoints with long-context support.
Llama 3 is a Meta Llama-family, text-focused, decoder-only transformer model family. It reads text tokens and generates text tokens through autoregressive next-token prediction, and the release made open-weight checkpoints at large scale and long context a mainstream reference point.
At a glance
- Family
- llama
- Source type
- Open weights
- Released
- July 2024
- Authors
- Meta AI, Llama Team
- Modalities
- Text
- Parameter count
- 405 billion parameters
- Context length
- 131,072 tokens
- Precision
- bf16, fp16, fp8
What It Is
Llama 3 is Meta's open-weight language model family built around a decoder-only transformer stack. The release spans multiple dense sizes and instruction-tuned variants, with the largest public checkpoint reaching 405 billion parameters and supporting context windows up to 128K tokens.Inputs And Outputs
Llama 3 reads text tokens and produces text tokens. A prompt can be plain text, a chat transcript, or a long document, and the model continues from that context one token at a time through autoregressive generation. Instruction-tuned variants use the same token interface for assistant-style replies, while long-context checkpoints keep earlier tokens available across much longer prompts.Architecture
At a high level, Llama 3 follows the decoder-only transformer pattern: token embeddings enter a repeated stack of causal attention and feed-forward blocks, then a linear head produces next-token probabilities for autoregressive generation. Rotary position embeddings mark token order inside attention, grouped-query attention carries context more efficiently than full multi-head attention, RMSNorm stabilizes each block, and SwiGLU handles the per-token feed-forward step. The 128K context window means the same decoder stack must keep useful attention over a much longer prefix than earlier short-context releases.Output
ProbabilitiesSoftmaxOutput
LinearTransformer architectureAttention sublayer containerFeed-forward sublayer containerN×RMSNormGrouped-Query
AttentionRMSNormSwiGLUEmbeddingRoPEInput
EmbeddingInput
TokensSoftmax to Output
ProbabilitiesOutput
Linear to SoftmaxRMSNorm to Output
LinearSwiGLU to RMSNormRMSNorm to SwiGLURMSNorm to RMSNormGrouped-Query
Attention to RMSNormEmbedding to Grouped-Query
AttentionEmbedding to RMSNormInput
Embedding to EmbeddingRoPE to EmbeddingInput
Tokens to Input
Embedding
Important Modules
The registry-linked modules below are the main moving parts to understand before reading later Llama-family releases. Grouped-query attention and causal attention carry context across the prompt, RoPE encodes position inside attention, RMSNorm keeps the repeated stack stable, and SwiGLU performs the dense feed-forward transformation inside each decoder block.Training
Meta's public sources separate broad text pretraining from later post-training work. Pretraining teaches next-token prediction over large multilingual text mixtures and is the stage that builds the base decoder stack. Post-training and alignment then reshape the same weights into instruction-following assistants without changing the core token interface. The linked training regimes and paper below point to the primary Meta records for each stage.Training regimes
Linked papers
- Llama 3