Knowledge distillation transfers behavior from a teacher model to a student model using outputs, probability distributions, features, generated examples, or feedback. The student may become cheaper or faster while retaining part of the teacher's capability.
Authorized distillation is a standard model-development technique, but providers may prohibit using their outputs to train competing systems. Safeguards can look for automated extraction patterns, excessive sampling, or attempts to reproduce protected model behavior.
ELI5
Knowledge distillation trains a smaller or different AI model to imitate useful behavior from a more capable teacher. The student learns from teacher probabilities, generated examples or other signals in addition to ordinary data.
For example, a large model can create labeled answers that help train a faster local model. The student is not an exact copy, and its quality depends on which prompts, examples and objectives were used during the transfer.
Why use knowledge distillation?
It can reduce model size, latency, hardware needs, energy use, or serving cost while retaining useful behavior from a stronger teacher.
Why can model providers restrict distillation?
Large-scale output collection can be used to reproduce proprietary behavior or train competing systems, so provider terms may limit or prohibit it.
