Knowledge distillation transfers behavior from a teacher model to a student model. The student learns from probabilities, generated examples or other signals produced by the teacher, often alongside ordinary training data, so it can approximate useful capabilities with a different size or architecture.
Distillation can make models cheaper or faster, but it also creates commercial and governance concerns. A provider may restrict model access when outputs could help improve a competing system. The resulting student is not an exact copy because its quality depends on the prompts, examples, objectives and training process used.
Acronyms and aliases
KD acronymmodel distillation synonym
Related terms
Frequently asked questions
Why is knowledge distillation used?
It can transfer useful behavior into a smaller, faster or more specialized model. This may reduce inference cost and make deployment practical on hardware that cannot run the teacher model.
Does knowledge distillation copy a model exactly?
No. The student learns from selected teacher signals and training examples, so it usually reproduces only part of the teacher's behavior and can have different strengths, weaknesses and errors.
