---
language: en
title: "Knowledge distillation"
description: "Knowledge distillation trains a student model to imitate a teacher. Learn how to assess cost, speed, and quality loss by task."
canonical: "https://tropk.ai/en/resources/glossary/knowledge-distillation"
last_modified: "2026-10-12T16:13:23.315Z"
---

# Knowledge distillation

Reviewed on 2026-10-06.

## What it means

Knowledge distillation is a technique in which a student model learns to reproduce predictions or outputs from a teacher model. The teacher can also be a set of models. By transferring this behavior to a smaller student, the aim is to make it easier to use with fewer resources. How closely the outputs match depends on the training and the task.

## Why it matters for marketing

For a marketing team running classification or generation at high volume, distillation makes it possible to weigh operating costs against quality. The decision requires identifying which capabilities need to be preserved. A smaller model that works well for a simple routine may still be unsuitable for exceptions; reducing its size does not demonstrate general equivalence to the teacher.

## Example in practice

Imagine a real estate portal that uses a large model to identify listings for sale or rent. A student trained on its outputs maintains classification for typical listings, but misclassifies texts offering both options. The team compares processing time and errors for this subset. Before replacing the teacher in the routine, it decides to send ambiguous listings for review.

## How to apply or verify

Identify the teacher, student, task, and data used to transfer the behavior. On a separate evaluation set, compare the student with reviewed answers and with the teacher. Measure quality by category, time, and resource consumption under the same workload. Document acceptable losses; agreement between models can include shared errors and is not sufficient on its own.

## Important distinctions

Distillation is not simply removing parameters or reducing the numerical precision of a ready-made model: it involves learning guided by a teacher. It also differs from [fine-tuning](/en/resources/glossary/fine-tuning) described solely as adapting to task examples, although training may combine objectives. Research results in image or speech recognition do not guarantee the same preservation of capabilities in every [LLM](/en/resources/glossary/llm).

## Sources and references

- [Distilling the Knowledge in a Neural Network](<https://arxiv.org/abs/1503.02531>) — Hinton, Vinyals e Dean / arXiv
- [Machine Learning Glossary: distillation](<https://developers.google.com/machine-learning/glossary#distillation>) — Google for Developers

## Continue a leitura

- [Fine-tuning](</en/resources/glossary/fine-tuning>)
- [Inference](</en/resources/glossary/inference>)
- [LLM](</en/resources/glossary/llm>)

## About the author

[Vinicius Pacheco](</en/team/vinicius-pacheco>)

Responsible for Tropk’s technical content, focusing on crawling, indexing, structured data, and agent access to content.

---

[All pages in Markdown](<https://tropk.ai/sitemap.md>)