Model distillation is a training strategy where a large, powerful model (the 'teacher') transfers its knowledge to a smaller, more efficient model (the 'student'). The goal is to create a compact model that performs nearly as well as its massive counterpart, but with a fraction of the computational cost and latency.

Imagine you have a highly capable model like Llama-3-70B, with its 70 billion parameters, and you want to deploy a similar capability on a device with limited resources. Instead of training a smaller model from scratch, you can use the 70B teacher to guide the training of a student model, like Llama-3-8B. The teacher, having already learned complex patterns from vast datasets, provides a richer, more nuanced training signal than the original dataset alone ever could.

Traditional supervised learning uses 'hard labels'. If you're training an image classifier, a picture of a cat is labeled 'cat'. This is the single, ground-truth answer. The model's goal is to output a probability of 1.0 for 'cat' and 0.0 for everything else.

Distillation takes a different approach by using 'soft labels'. Instead of the ground-truth label, the student model is trained on the teacher model's output probabilities. A powerful teacher might see an image of a Siberian Husky and output probabilities like: 90% Husky, 6% German Shepherd, 3% wolf, and 1% fox. This distribution is the soft label.

This nuanced output reveals the teacher's internal understanding of the data. It shows why it thinks the image is a Husky by highlighting its similarity to related classes. This rich, relational information is often called dark knowledge. It's the subtle reasoning that isn't present in the simple 'Husky' label. The student learns not just to identify a Husky, but to see the world through the teacher's more experienced lens.

To effectively transfer dark knowledge, we need to control the 'softness' of the teacher's probability distribution. A teacher model is often very confident, assigning a probability near 1.0 to its top choice. This leaves little information in the other probabilities for the student to learn from.

We solve this by introducing a 'temperature' parameter, $T$, into the softmax function. The softmax function converts the model's raw output scores (logits) into probabilities. The standard softmax looks like this:

By adding temperature, we divide each logit by $T$ before applying the exponential function:

The goal of using temperature is to maximize the amount of information passed from teacher to student. We can measure this using the concept of information entropy, which quantifies the uncertainty or randomness in a probability distribution.

A distribution with high entropy is more uniform (e.g., [0.25, 0.25, 0.25, 0.25]), while one with low entropy is 'peaked' or certain (e.g., [0.97, 0.01, 0.01, 0.01]).

By increasing the temperature $T$, we increase the entropy of the teacher's soft labels. This higher-entropy distribution contains more information for the student to learn from. The student's training objective is then to minimize the difference between its own output distribution (also calculated with the same temperature $T$) and the teacher's softened distribution. This ensures the student doesn't just mimic the teacher's top prediction, but also its entire reasoning process.

The final step involves combining this distillation loss with a standard supervised learning loss, using the ground-truth hard labels. This blend ensures the student learns both the teacher's nuanced reasoning and stays grounded in the actual correct answers. This dual-objective approach is key to creating a small but powerful model.