Continued training on a pretrained base checkpoint before final post-training, using targeted data or objectives to extend capability without yet shaping final assistant behavior.
Mid-training is continued training that starts from a broadly pretrained base checkpoint and runs before the final post-training stages that shape assistant behavior. Teams use it to extend capability, refresh knowledge, or steer the model toward a domain while the model is still learning from broader corpora rather than final instruction or preference datasets.
At a glance
Released
July 2024
Authors
Meta AI, Llama Team
Regime type
Training
Related modules
No related modules listed yet.
What It Is
Mid-training is the training stage between large-scale pretraining and final post-training. A team loads weights from a pretrained base model and keeps training on additional data or objectives. The run may look like pretraining in mechanics—token prediction, masked language modeling, or similar losses—but it starts from an existing checkpoint instead of random initialization and usually uses a narrower or more targeted data mixture than the first broad pretraining pass.
Why It Exists
Pretraining creates a general base model, but product teams often need more capability before behavior shaping begins. Mid-training fills that gap. A longer or refreshed pretraining-style run can add domain knowledge, improve reasoning patterns, or adapt the model to new languages or code styles without yet switching to instruction demonstrations or preference optimization. It lets teams invest compute in capability growth while keeping the final assistant-alignment stage separate.
How It Works
Training resumes from a pretrained checkpoint. The team chooses a data mixture—general web text, code, synthetic data, domain documents, or a blend—and an objective that matches the goal. Updates continue until the checkpoint reaches the desired capability level or budget limit. The output is an intermediate model that is stronger or more specialized than the original base but has not yet received the final post-training passes that teach instruction following, preference fit, or deployment-specific safety behavior.
Mid-training flow
Pretrained base checkpointContinued training on targeted dataIntermediate checkpointPretrained base checkpoint to Continued training on targeted dataContinued training on targeted data to Intermediate checkpoint
Press enter or space to select a node. You can then use the arrow keys to move the node around. Press delete to remove it and escape to cancel.
Press enter or space to select an edge. You can then press delete to remove it or escape to cancel.
Request and weight flow
Mid-training resumes from a pretrained checkpoint and continues training before final post-training behavior shaping.
Broad pretraining builds the initial base model from large, general corpora using self-supervised objectives such as next-token prediction. Mid-training continues from that pretrained checkpoint instead of starting from random initialization. The mechanics may look similar, but the data mixture is usually narrower or more targeted and the goal is to extend or refresh capability on top of an existing base rather than create that base from scratch.
Post-training is the later stage where teams shape behavior for instruction following, preference fit, safety, or deployment style. Mid-training comes before that final assistant-shaping work. A mid-training run may improve capability, but it is not the stage that teaches the model to act like a polished product assistant.
Instruction tuning and supervised fine-tuning, often shortened to SFT, learn from explicit prompt-and-answer demonstrations or other supervised targets. Each example shows the model what to output for a given input. Mid-training may still use broad corpora, synthetic mixtures, or domain documents with pretraining-style objectives rather than curated demonstration pairs, because its main job is continued capability learning before final behavior shaping.
Domain adaptation is a narrower goal: steer a model toward one field, language, or product surface. That focus can be one reason to run continued training, but domain adaptation names the target outcome, not the whole mid-training category. A mid-training run may pursue broader goals such as refreshed general knowledge, longer context habits, or mixed capability upgrades without being limited to one domain.
Distillation transfers behavior from a teacher model to a student through teacher-generated supervision. Mid-training does not require a teacher. Teams can continue training with ordinary corpora, synthetic data, or targeted mixtures using standard objectives, without copying outputs from a separate stronger model.
Limitations And Failure Modes
Mid-training inherits the limits of its starting checkpoint and the new data mixture. A narrow or noisy continued-training corpus can skew capability without fixing underlying base-model gaps. Because the stage often reuses pretraining-style objectives, it does not by itself produce reliable instruction following or preference-aligned behavior; those outcomes still depend on later post-training.