GPT-2 Report

A technical report that introduced GPT-2 as a large decoder-only language model trained with broad next-token pretraining and byte-level BPE tokenization.

Language Models are Unsupervised Multitask Learners, usually called the GPT-2 report, became a reference point because it showed that one large decoder-only transformer trained on broad internet text could display useful zero-shot behavior without task-specific fine-tuning for every example.

At a glance

Published

February 2019

Authors

Alec Radford, Jeffrey Wu, Rewon Child, et al.

Why It Matters

The GPT-2 report mattered because it made scale feel concrete to a wide audience. Earlier transformers already existed, but this report showed how a bigger decoder-only model, broad next-token pretraining, and a robust tokenizer could turn one language-model objective into a surprisingly general system. That combination helped set the template later GPT-style language models kept following.

Method Or Architecture

At a high level, the report keeps the familiar transformer architecture in a decoder-only form: each token predicts the next one while attending only to earlier positions. The tokenizer uses byte-level byte pair encoding (BPE), so arbitrary UTF-8 text can still be represented instead of failing on unusual characters. Training stays broad rather than task-specific: the model reads a large internet text mixture and learns by next-token prediction, then the paper studies what kinds of behavior appear when that same objective is scaled up.
The GPT-2 report is easiest to understand as three linked choices: a decoder-only transformer, byte-level BPE tokenization, and broad next-token pretraining at larger scale.

Evidence

The paper's key evidence is not a leaderboard tour. It is the pattern that one large pretrained model can transfer across summarization, translation-style prompting, question answering, and other language tasks without a separate supervised training run for each one. That helped turn zero-shot and prompt-based evaluation into a more serious reader journey for later language-model work.

Limitations

The report does not prove that scale alone solves language understanding, and several behaviors are shown through selective tasks rather than one unified benchmark story. It also predates many later refinements around instruction tuning, alignment, data curation, and compute-optimal scaling, so readers should treat it as a foundational decoder-only reference point, not as the final recipe modern systems use unchanged.

Tags

References

  1. Radford, Alec, et al. "Language Models are Unsupervised Multitask Learners." OpenAI, 2019.

DeepSeek-V4

Previous Page

Latent Diffusion Models

Next Page

On this page

Why It MattersMethod Or ArchitectureEvidenceLimitationsRelated ToTagsReferences