On-Policy Distillation

A DeepSeek-V4 post-training routine where the student is refined on trajectories produced under its current policy rather than only on frozen offline traces.

Offline distillation can drift away from the states a student model actually visits. On-policy distillation closes that gap by training on trajectories produced under the current model policy, so the supervision stays closer to the behavior you are trying to shape.

At a glance

Released

January 2024

Authors

Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, et al.

Regime type

Distillation

Related modules

What It Is

This is a distillation routine used in the DeepSeek-V4 report. The important idea is that data is refreshed from the model's own current behavior rather than treated as one fixed offline replay set.

Why It Exists

It aims to improve policy consistency and reduce the mismatch between training supervision and the states the student actually enters at inference time.

How It Works

A current policy samples trajectories, those trajectories are scored or filtered, and the student is updated on the accepted behavior. The loop then repeats with the new policy.
On-Policy Distillation training flow
Request and weight flow
On-policy distillation keeps refreshing supervision from the student's current behavior.
textteachersignalfreshnessapproxfracDeltatextpolicy−matchedbehaviorDeltatextrolloutcost+Deltatextteacherevaluationcost\\text{teacher signal freshness} \\approx \\frac{\\Delta \\text{policy-matched behavior}}{\\Delta \\text{rollout cost} + \\Delta \\text{teacher evaluation cost}}

Compared To Nearby Regimes

Unlike plain offline distillation, the training set is not fully frozen. Unlike full RL, the emphasis is still on teacher-guided or filtered behavior rather than only scalar reward optimization.

Limitations And Failure Modes

If the current policy explores badly, the loop can reinforce narrow behavior. The regime also depends on good filtering or teacher signals.

Tags

References

  1. Agarwal, Rishabh, et al. "On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes." The Twelfth International Conference on Learning Representations, 2024, https://openreview.net/forum?id=3zKtaqxLhW.
  2. DeepSeek-AI. "DeepSeek-V4 Technical Report." 2026.

Distillation

Previous Page

Dropout

Next Page

On this page

What It IsWhy It ExistsHow It WorksCompared To Nearby RegimesLimitations And Failure ModesRelated ToTagsReferences