Pretraining

The large base-model training stage where a transformer learns broad language patterns from massive token sequences before later alignment or deployment shaping.

Pretraining is the long first training stage that turns a transformer into a base model: it reads huge amounts of tokenized text, repeatedly guesses the next token, and slowly absorbs broad statistical patterns before any later instruction tuning or preference alignment begins.

At a glance

Released

February 2019

Authors

Alec Radford, Jeffrey Wu, Rewon Child, et al.

Regime type

Training Pretraining

What It Is

Pretraining is the base-model stage of language-model training. A transformer is exposed to very large token sequences from many documents, and its weights are updated so it becomes better at continuing those sequences. The result is not yet a chat assistant or policy-shaped product model. It is a broad statistical learner that has picked up vocabulary, syntax, factual associations, style patterns, and many reusable internal features.

Why It Exists

Modern systems need one large shared stage where the model learns general language structure before narrower behavior shaping begins. Pretraining fills that role. Scale matters here because a larger model, more tokens, and a longer run usually let the system absorb more reusable patterns, though each gain costs more money and time. It gives later stages such as alignment, instruction tuning, or Direct Preference Optimization a capable starting point, because those later stages work best when the base model already knows how text is usually formed and how many patterns in the world tend to co-occur.

How It Works

The training run starts with tokenized corpora such as web text, books, code, or other large text sources. During each update, the model sees a prefix of tokens and predicts the next token. The loss pushes probability mass toward the observed continuation, and gradient updates slowly improve the model across billions or trillions of such prediction events. In plain language, the model learns by being wrong, measuring how wrong it was, and adjusting its parameters so the next-token guess becomes a little better next time.
Pretraining training flow
Request and weight flow
Pretraining turns huge token corpora into a base model by repeating the next-token objective at scale.
\\max_\\theta \\sum_t \\log p_\\theta(x_t \\mid x_{<t})
θ
model weights
t
token position

xt

the observed next token at position t

x<t

all earlier tokens before position t
pθ(·)
the model probability distribution given weights \theta

Compared To Nearby Regimes

Pretraining builds the broad base model. Alignment and other post-training regimes come later and reshape behavior for human use, safety, preference fit, or product constraints. DPO, reinforcement learning from human feedback, and other alignment methods do not replace pretraining; they start from a model that pretraining already made broadly capable. Tokenization pages matter here because the model does not learn from raw characters or ideas directly. It learns from token sequences produced by choices such as byte-level tokenization or BPE.GPT-2 is a useful historical bridge for this stage: it showed that a transformer pretrained at broad internet scale could already perform many language tasks before any later chat-style alignment layer was added. GPT-3 then pushed the same pretraining idea much further in model size, data, and compute.

Limitations And Failure Modes

Pretraining can only absorb the patterns available in its data mixture and only up to the compute budget used to fit those patterns. Data mixture matters because web text, books, code, and other sources teach different habits, knowledge, and failure modes, so the blend shapes what the base model becomes good at. Compute matters because hardware time, memory, and optimization budget decide how far the training run can push the model before the project has to stop. If the corpus is narrow, noisy, or skewed, the base model inherits those limits. If the run is too small or too short, the model leaves capability on the table. And because the objective is usually next-token prediction rather than direct human preference shaping, the pretrained model may still be unhelpful, unsafe, or poorly steered until later post-training stages are added.

Tags

References

  1. Radford, Alec, et al. "Language Models are Unsupervised Multitask Learners." OpenAI, 2019.
  2. Brown, Tom B., et al. "Language Models are Few-Shot Learners." arXiv, 2020.
  3. Kaplan, Jared, et al. "Scaling Laws for Neural Language Models." arXiv, 2020, https://arxiv.org/abs/2001.08361.

Diffusion Training Objective

Previous Page

Direct Preference Optimization

Next Page

On this page

What It IsWhy It ExistsHow It WorksCompared To Nearby RegimesLimitations And Failure ModesRelated ToTagsReferences