GPT-3
A large decoder-only language model that made in-context prompting and few-shot text generation a mainstream reference point.
GPT-3 showed that scaling a decoder-only transformer far enough could turn plain text prompting into a practical interface, because the same pretrained model could continue text, follow patterns, and imitate simple tasks without task-specific fine-tuning.
At a glance
- Family
- gpt
- Source type
- Closed
- Released
- May 2020
- Authors
- Tom B. Brown, Benjamin Mann, Nick Ryder, et al.
- Modalities
- Text
- Parameter count
- 175 billion parameters
- Context length
- 2,048 tokens
- Precision
- float16, float32
What It Is
GPT-3 is a closed text-only language model from OpenAI. It is a decoder-only transformer trained to predict the next token in long text streams, and it became the best-known early example of a general-purpose prompting model.Inputs And Outputs
The model reads text tokens and produces text tokens. In practice, a prompt can include instructions, examples, or unfinished prose, and GPT-3 continues from that context one token at a time through autoregressive generation.Architecture
At a high level, GPT-3 keeps the familiar decoder-only transformer stack. Input embeddings are combined with learned positional embeddings, then a repeated decoder block runs masked multi-head attention, add-and-norm, a standard feed-forward network, and another add-and-norm before a linear head and softmax produce next-token probabilities.Output
ProbabilitiesSoftmaxLinearTransformer architectureAttention sublayer containerFeed-forward sublayer containerN×Add & NormMasked
Multi-Head
AttentionAdd & NormFeed
ForwardEmbeddingLearned
Position
EmbeddingsInput
EmbeddingInput
TokensSoftmax to Output
ProbabilitiesLinear to SoftmaxAdd & Norm to LinearFeed
Forward to Add & NormAdd & Norm to Feed
ForwardAdd & Norm to Add & NormMasked
Multi-Head
Attention to Add & NormEmbedding to Masked
Multi-Head
AttentionEmbedding to Add & NormInput
Embedding to EmbeddingLearned
Position
Embeddings to EmbeddingInput
Tokens to Input
Embedding
Important Modules
The modules below are the main moving parts to understand before reading GPT-style descendants. Multi-head attention carries context across the prompt, the standard FFN transforms each position locally, learned positional embeddings mark token order, and layer normalization helps the repeated stack stay trainable.Training
Publicly described GPT-3 training centered on large-scale next-token pretraining over a broad text mixture rather than specialized instruction tuning. The model's importance came less from a brand-new block and more from scale: parameter count, data volume, and compute were pushed far enough that few-shot prompting became much more useful.Training regimes
Linked papers
No linked paper pages listed yet.