Distillation
A teacher-student training regime where a smaller model learns from supervision produced by a larger or stronger teacher model.
Distillation is a teacher-student training regime. A stronger teacher model produces supervision, and a smaller student model learns to match it so teams can deploy a cheaper model that keeps much of the teacher's useful behavior.
At a glance
Released
January 2024
Authors
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, et al.
Regime type
Distillation
Related modules
No related modules listed yet.
What It Is
Distillation is a teacher-student training regime. A larger or stronger teacher model produces supervision, and a smaller student model learns to match that supervision. The teacher signal can appear as soft targets that spread probability across many plausible tokens, generated traces the teacher writes for the student to imitate, or labeled examples where the teacher marks the outputs the student should copy.Why It Exists
Teams distill when they need a model they can run cheaply at scale. The student usually costs less per request, responds with lower latency than the full teacher, and fits deployment constraints such as smaller GPUs, on-device serving, or tighter serving budgets. Distillation trades extra training time and teacher compute for a model that keeps much of the teacher's useful behavior after deployment.How It Works
The teacher model produces supervision such as soft targets, generated traces, or labeled examples. Those training examples or traces become the data the student learns from, the student weights are updated to match the teacher signal, and the trained student is the deployment-oriented model teams serve instead of the full teacher.Distillation training flow
Teacher modelTraining examples or tracesStudent updateDeployment-oriented studentTeacher model to Training examples or tracesTraining examples or traces to Student updateStudent update to Deployment-oriented student
Request and weight flow