Here we will explore what the word distillation covers, why the three things it covers need different setups, and how I picked a recipe for a run on my own machine.
The short version: one family needs a teacher whose logits you can read, one works fine against a closed API, and one needs the teacher in the loop while the student generates. They get written up under the same word, and the family you pick decides which teacher you can use and what a result would prove.
What R1’s small models were built from
The most cited distillation result in open weights is DeepSeek’s R1 distill series, and it is the clearest example of the ambiguity.
The model card for DeepSeek-R1-Distill-Qwen-1.5B says the distill models are “fine-tuned based on open-source models, using samples generated by DeepSeek-R1”, with 800,000 curated samples. No logits. The student trains on text the teacher wrote.
The same card lists the base model as Qwen2.5-Math-1.5B. The 1.5B that posts strong maths scores after distillation started from a checkpoint that had already been tuned on maths.
Both facts are on the card and neither is hidden. But the popular reading, that a weak small model was made strong by absorbing a large one, is not what the card describes. The student was already pointed at the task, and the transfer was supervised fine-tuning on generated text.
The three families
Soft-target distillation is the original, from Hinton et al.. The student matches the teacher’s full output distribution, softened by a temperature. The signal is the shape of the distribution: what the teacher nearly said carries information that the single chosen token throws away. This needs the teacher’s logits, so the teacher has to be open weights running on your hardware.
Sequence-level distillation is the Alpaca shape. The teacher generates text, the student fine-tunes on it. A closed API teacher is fine, because all you consume is the output. This is the family R1’s distills belong to, and it is the same operation as synthetic-data fine-tuning under a different name.
On-policy distillation (MiniLLM, GKD) has the student generate and the teacher score the student’s own text. It exists to fix exposure bias: a student trained only on teacher text never learns to recover from its own mistakes, because it never sees them during training.
| Model | Teacher | Student | Family |
|---|---|---|---|
| Alpaca | GPT-3.5 | LLaMA-7B | sequence-level, 52K |
| Vicuna | ChatGPT | LLaMA | sequence-level, 70K |
| Zephyr | GPT-4 feedback | Mistral-7B | dSFT + dDPO |
| R1 distills | R1 671B | Qwen, Llama | sequence-level, 800K |
Why the family decides the teacher
Sequence-level is the popular family because it is the cheap one. Any API model can be your teacher. That is also why it is the least interesting to test: the result you get is about the quality of a synthetic dataset, and the distillation framing adds nothing you could not have said about the data.
Soft-target is the family where the teacher has to be local. A closed API returns tokens, sometimes a handful of top logprobs, never the distribution over the full vocabulary. If you want to run the Hinton objective, the teacher runs on your machine.
That constraint is what makes it a good run to publish. A 32B teacher and a 1.5B student on one laptop is a setup a reader can reproduce.
Choosing a task the student is bad at
A distillation result only means something if the student starts bad. If the baseline is already strong there is no lift to measure.
Maths is the obvious choice and the wrong one. A 1.5B model failing arithmetic is plausibly failing on capacity: digit tokenisation and multi-step compute are architecture problems. Distillation moves knowledge, behaviour and output format between models. It does not add capacity. Picking a task where failure might be capacity means a null result tells you nothing about the method.
Text-to-SQL on Spider avoids that. Generating a SQL query is not compute-heavy, so a small model has the capacity for it. It has an objective metric in execution accuracy, meaning the query runs and returns the right rows, so grading does not depend on a judge model. And there is no published Spider number for Qwen2.5-1.5B that I could find. Reported baselines cluster at 3B and above. Measuring the small baseline is part of the experiment rather than a lookup.
Reverse KL
Forward KL, the direction in the original paper, is mean-seeking. The student is penalised for putting low probability anywhere the teacher put any, so it spreads mass across the teacher’s whole distribution including its tail. For a classifier that is fine. For a generative model, that tail mass is what surfaces as fluent-sounding wrong output.
Reverse KL is mode-seeking. The student concentrates on the teacher’s major modes and gives up on covering everything. MiniLLM is built on that swap.
That is a claim from the papers, not from my machine. Running both directions on the same student and comparing is the part of this I want to see for myself.
The recipe
Teacher is Qwen2.5-Coder-32B-Instruct at 4-bit under MLX, around 17 GiB of weights. Student is Qwen2.5-1.5B, chosen general rather than task-tuned so the baseline is honest, which is exactly the property the R1-Math base gives up. Objective is soft-target KD with reverse KL and temperature-squared scaling.
mlx-lm has no built-in KD trainer, so the loop is custom. The teacher runs offline over the Spider training set first and its top-k logits get cached to disk, because a 151k vocabulary at full width does not fit alongside a student in training.
I have the teacher and student decode numbers on this machine already, from measuring decode throughput against model size. The training loop is next.