Model distillation transfers behavior from a teacher model to a student model. The student can learn from teacher-generated labels, probability distributions, reasoning traces where permitted, synthetic examples, or other structured outputs instead of relying only on the original training data.
Distillation can reduce serving cost or create a model specialized for a narrower task, but it raises technical, contractual, and provenance questions. Access terms may restrict using one provider's outputs to train a competing model, and a student can inherit teacher errors or lose capabilities during compression.
ELI5
Model distillation is a way to teach one AI model using examples or guidance produced by another model. The goal is often to make a smaller, cheaper student copy useful parts of a stronger teacher's behavior.
For example, a teacher might label many difficult questions and a student model trains on those labeled examples. The student may become good at that task, but it can also copy mistakes, miss subtle skills, and violate access terms if the teacher's outputs were not permitted for training.
