THE DIRECT ANSWER

Distillation transfers behaviour from a teacher system into a student model through a training process. It can be used to create a different size or task fit. The student must still be evaluated independently because it may not preserve every capability or limitation of the teacher.

The teaching relationship

A teacher model can supply outputs or other signals for training a student. This may help the student perform a defined task with fewer resources. It does not transfer every capability of the larger model, and the quality of the teaching material matters.

Inspect the examples

Teacher mistakes can become training examples too. Check representative samples and keep evaluation data separate. A support classifier should see ambiguous and out-of-scope requests, not only clean examples with obvious answers. Decide what the student should do when evidence is insufficient.

Evaluate the result directly

Compare the student with a simple baseline on task success, latency and resource use. Look specifically for cases the teacher handled but the student missed. Document training-data permissions and version the dataset. A distilled model can be useful for a narrow workflow without being suitable for general conversation. The useful outcome is dependable behaviour under real operating conditions, not a smaller system that merely imitates the teacher’s confident writing style.

A worked example

For a fictional categorisation task, a teacher might help produce training examples under an appropriate, permitted process. The student is then tested on separate human-reviewed cases. Include errors and ambiguous examples in the review instead of assuming the teacher's output is always correct. A student that imitates flawed labels can reproduce the same weakness efficiently.

COMMON QUESTIONS

Questions readers ask

Does a distilled model inherit all of the teacher's abilities?

No. Distillation targets particular behaviour through selected data and training. Evaluate what the student actually does on relevant and held-out examples.

SOURCES & NOTES

Follow the evidence.

AI-assisted explanation with linked technical references. Examples are illustrative learning exercises, not reported customer deployments or experimental results.

  1. arxiv.org — research paper

Public sources support the factual background. Analysis and illustrative exercises are identified in the text. How we assess AI · Editorial standards

Spot a factual error or an outdated detail?