How Big Models Teach Small Models, and Why It Looks Like School

TL;DR. Knowledge distillation is a large model teaching a small one, and the mechanics map almost one for one onto how people learn. Partial credit teaches more than a bare answer key. Sitting next to a senior teaches more than reading their report. A tutor who watches you work beats a textbook of last year’s solutions, which is exactly why on-policy distillation is where 2026 went. The limits map too: you rarely outgrow your teacher, a beginner cannot learn directly from the world expert, and students inherit habits nobody meant to teach.

The question I keep getting at summer dinners

It is August, the calendar is empty, and the conversations get slower and better. Twice this month someone outside the field asked me the same thing in different words: how does a model small enough to run on a laptop end up nearly as good as the giant one behind an API?

The short answer is that somebody taught it. ByteByteGo published a clean explainer on how big models teach small models on 5 August 2026, walking through distillation, soft labels, the three main methods, the limits and the emerging automation. It is worth your ten minutes. What follows is the version I actually say out loud, because knowledge distillation works almost exactly like school, right down to the failure modes.

One thing to get straight first, and the explainer opens on it. Distillation is not compression. Quantization and pruning start from a model you already have and shrink its footprint, either by storing its numbers at lower precision or by removing the parts that barely contribute. Distillation is a different operation. It “produces a genuinely separate model, with its own parameters and often a different design, whose goal during training is to behave like the teacher”. Different exercise, different budget, different line in the plan.

A bare answer is a bad lesson

Ordinary supervised training gives the model a hard label: “cat”. One word, no context. A teacher model gives a soft label instead, a full set of probabilities across the options. ByteByteGo uses cat at 0.70, dog at 0.25 and fox at 0.05. That distribution says the picture was a bit dog-like and not at all fox-like, which the word “cat” never told you. Researchers call the extra signal dark knowledge, “the structure hidden in a model’s confidence that a bare label leaves out”.

You already know the human version. There is the teacher who hands your paper back with a red cross, and the teacher who says “you were close, that answer was tempting because of X, and the third option was never in play”. The second one moves you forward three times faster on the same exercise. Nothing about the exercise changed. The signal did.

The original 2015 paper by Hinton, Vinyals and Dean added a knob called temperature for exactly this. Raise it and the probabilities spread out, exposing more of the fine structure for the student to learn. In classroom terms: how much of their own hesitation the teacher lets you see. Someone who only ever says “the answer is 4” teaches less than someone who says “it is 4, and I nearly wrote 6, here is why I did not”.

Three ways to teach

ByteByteGo splits the field into three methods. They line up neatly with three things that happen in any classroom, and in any engineering team.

1. Mark against the answer sheet. Output distillation, the original 2015 form. The student matches the teacher’s final outputs, soft labels included, and nothing else. Direct, cheap, and it only transfers what shows up at the end.

2. Sit next to the senior. Feature distillation. The student matches the teacher’s internal representations, meaning the intermediate values a model computes while processing an input, before it settles on a final answer. Google’s EmbeddingGemma is built this way, a 308M model distilled from Gemini Embedding by matching embeddings rather than outputs. This is the pairing week. You learn which log the senior opens first, what they check before touching anything, which detail made them change direction. The written report at the end never contains that, which is why reading reports has never made anyone senior.

3. Work the problem set. Synthetic data distillation. The teacher generates a dataset of examples and the student is fine-tuned on it like any ordinary training data. Stanford’s Alpaca did this in 2023, and the explainer notes it is now the most common form in practice for one blunt reason: it needs only the teacher’s text output. No weights, no internals, no cooperation beyond an API key. It works on closed models.

The third dominates for the same reason worked exercises dominate in school. It scales, it needs nothing special from the teacher, and the teacher does not have to be in the room.

The tutor beats the textbook, and 2026 noticed

There is a fourth distinction the explainer does not draw, and it is the one that matters most this year. Sergio Paniego’s survey of distillation in 2026 on Hugging Face (July 2026) sorts the current model families by whether the teaching is off-policy or on-policy.

• Off-policy is the textbook. The teacher writes out solutions in advance, the student studies them. Gemma 3 and 4 distill from a large instruction-tuned teacher. DeepSeek-R1-Distill pushed reasoning traces into Qwen and Llama students by plain supervised fine-tuning on teacher-generated text.
• On-policy is the tutor. The student attempts the problem itself and the teacher grades the student’s own attempt, token by token. DeepSeek-V4 trains separate domain experts for math, code and agentic work, then has the student generate rollouts against those specialists. MiMo-V2-Flash does multi-teacher on-policy distillation. Nemotron 3 Ultra reportedly uses more than ten specialized teachers with domain-specific pipelines.

Anyone who has been taught anything knows why the second works better. A textbook of worked solutions shows you what a correct path looks like. A tutor watching you work corrects your mistake, the one you were actually about to make, at the moment you make it. The correction lands on your errors instead of on a stranger’s.

Hold on to one phrase from that comparison: letting the student explore the whole space and rewarding it when it stumbles onto the right answer. It is not a figure of speech. It is a description of an optimizer. I pulled that thread into a companion post, why leading costs about ten times more than following, which is where the numbers on that comparison live.

It works, and it works narrowly

DeepSeek shipped a distilled family from 1.5B to 70B parameters in early 2025, and the headline everybody repeats is that a 7B student scored higher than a 32B model on a competition mathematics benchmark, from plain fine-tuning on the larger model’s outputs. That result is real.

The qualifier gets dropped in the retelling. ByteByteGo is careful about it: those wins tend to appear on narrow, well-defined tasks such as mathematics and code, while distilled models stay weaker on broad general knowledge. It is the student who was drilled hard for one exam. Outstanding on that paper, unchanged everywhere else. Distillation is coaching for a specific test, and the literature is honest about that if you read past the headline.

If you want the depth behind the summary, the reference work is “A Survey on Knowledge Distillation of Large Language Models” (Xu et al., 2024). It organizes the field around algorithm, skill and verticalization, and covers domain-specific distillation in law, healthcare, finance and science. That survey is the map. The ByteByteGo piece is the tour.

I took apart a concrete example last month: what actually works when you distill frontier reasoning into a local model, including the economics and the parts the launch posts skip.

Four limits any teacher would recognize

You rarely outgrow your teacher. A student trained on a teacher’s output tends to stay at or below the teacher’s level on the kind of data it saw. Worse, in ByteByteGo’s words: “When the teacher produces a wrong answer, the student learns that wrong answer along with the right ones.” The teacher’s quality sets the bar. Every engineer has inherited one bad habit from a senior they admired and carried it for a decade. Pick your teacher on quality, not on price or availability.

Do not put a first-year in a doctoral seminar. A larger, stronger teacher does not automatically produce a better student. When the gap is very wide, transfer degrades, because the student has too little capacity to absorb everything a much larger model expresses. The literature’s fix is the one schools have used forever: insert an intermediate teacher assistant (Mirzadeh et al., 2019). Not the world expert straight to the beginner. The person one level up, who still remembers what was confusing.

Method beats raw brain. In one study cited by the explainer, a 32B student outperformed a 70B student on the same task, because the smaller one was built on a stronger base architecture. Size alone is a weak predictor of how well distillation will go. Same story in a team. The junior with a working method beats the smarter junior without one over any horizon that matters. Choosing the student model is not a “how many parameters can we afford” question.

Students pick up what nobody taught on purpose. This is the one that should make you uncomfortable. In work published in Nature (Cloud et al., first posted as “Subliminal Learning” in July 2025), a teacher model was given a quirk, a disproportionate fondness for owls, then used to generate training data consisting only of number sequences. Students trained on those numbers came out fond of owls. The data had been filtered to strip any visible trace of the trait. It transferred regardless. The effect showed up again with math reasoning traces and with code, and only when teacher and student shared a base model. ByteByteGo’s reading is the right one: distillation can carry across more than the task being taught, and filtering the visible data is sometimes too coarse to stop it.

Which is precisely what every manager has watched happen with people. New engineers absorb the team’s tone, its shortcuts, its blind spots and its unwritten “we do not touch that service”, none of which appears in the onboarding doc. Distillation reproduces that faithfully, including the parts you would have filtered out if you could see them. You cannot see them. That is the finding.

The teacher is starting to run the whole class

The newest turn is the one ByteByteGo closes on, and the primary source is worth reading directly. In “Agentic Knowledge Distillation” (ElZemity, Sylvester, Arief and De Lemos, February 2026) the teacher runs the entire loop unsupervised. In the authors’ words, “the teacher LLM autonomously generates synthetic data and iteratively refines a smaller on-device student model until performance plateaus”. It writes the syllabus, sets the exercises, marks the work, generates its own held-out exam, and decides when to stop.

The task is SMS phishing detection, chosen because labelled threat data goes stale fast. They tried four teachers, Claude Opus 4.5, GPT 5.2 Codex, Gemini 3 Pro and DeepSeek V3.2, against two tiny students, Qwen2.5-0.5B and SmolLM2-135M. The best configuration reached 94.31% accuracy and 96.25% recall, against 50% to 80% for a Direct Preference Optimization baseline. A 135M-parameter model is small enough to run on a phone.

Then read their caveat, because it is the same one running through this entire post: outcomes depend strongly on which teacher LLM is used. Automate the whole pipeline and the one decision you cannot automate away is which teacher you hire. Schools worked that out a long time ago.

What changes when you have to plan the infrastructure

The analogy is fun. The invoice is not. Three things I look at when a team tells me they are going to distill.

• The bill moves, it does not disappear. Distillation converts a recurring inference cost that scales with usage into a one-time training cost plus hardware you own. Good trade at high volume on a fixed task. Bad trade at low volume, and the one-time cost includes every generated sample you throw away, which is most of them if your gate is strict.
• The teacher is a supply dependency. Your pipeline now depends on a model somebody else controls, at a price they set, under terms that may forbid using the outputs to train a competitor. The agentic paper makes the point empirically: swap the teacher and the result moves. Access changes too. Plan for it the way you plan for any single-source vendor.
• Without a verification gate you are copying confident mistakes. If you cannot check each generated example automatically, tests passing, schema validating, ground truth on hand, then you are not distilling knowledge. You are distilling the teacher’s errors at scale, faster than you could have made them yourself. The gate is the method. The teacher names on the model card are packaging.

Then there is the sizing question, which is my day job: capacity planning and platform sizing. A distilled 0.5B to 14B model changes the hardware conversation completely. You stop asking for a GPU fleet and start asking how much VRAM per inference node, how much KV cache at your real context length, and what happens to that number when the agent loop gets chatty. On-policy distillation moves the question again, because the student generates rollouts during training and that is a very different job to size than a straight fine-tune. Different purchase, different rack, often a different vendor entirely. Worth working out before the quote arrives, not after you have signed it.

Where I would use it, and where I would not

Use it when the task is narrow, verifiable and high volume, and a capable teacher exists. That covers a lot of the boring, expensive work already running in production: classification, extraction against a schema, code checked by tests, routing decisions, and exactly the kind of detection job the agentic paper picked. Those are the places where paying frontier prices per call is waste, and where a small owned model quietly wins on cost, latency and data control.

Skip it when the goal is broad, open-ended capability. ByteByteGo lands in the same place, and it is the right conclusion. That is coaching for an exam that does not exist. You will spend the training budget and end up with something worse than the API you were trying to replace. And skip it for the work your product actually competes on, where quality and speed decide the outcome and the frontier premium is buying the win rather than the tokens. I made that argument with numbers in the cost case for model fusion.

The teaching analogy holds all the way to the end, including the uncomfortable part. A good teacher makes a student who is fast, confident and shaped by someone else’s judgment. That is a bargain worth making on the jobs where the syllabus is clear. It is a poor one everywhere else.

One question this post does not answer: why the arrangement is so lopsided in the first place, and why being the model everyone else copies costs an order of magnitude more than being one of the copies. That one is in the companion piece, with the GPU-hour numbers and a bike race.

Sources

• ByteByteGo, “How big models teach small models”, 5 August 2026. The explainer that prompted this post.
• Sergio Paniego, “Distillation in 2026 (so far): which frontier models use it and how”, Hugging Face, July 2026. Off-policy against on-policy, family by family.
• Xu et al., “A Survey on Knowledge Distillation of Large Language Models”, arXiv 2402.13116, 2024. The reference survey.
• DeepSeek-AI, “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning”, arXiv 2501.12948, 2025. The distilled 1.5B to 70B family.
• ElZemity et al., “Agentic Knowledge Distillation: Autonomous Training of Small Language Models for SMS Threat Detection”, arXiv 2602.10869, February 2026. The fully automated teaching loop.
• Cloud et al., “Language models transmit behavioural traits through hidden signals in data”, Nature. The owls.
• Hinton, Vinyals and Dean, “Distilling the Knowledge in a Neural Network”, 2015. Where soft labels and temperature come from.
• Mirzadeh et al., “Improved Knowledge Distillation via Teacher Assistant”, 2019. The capacity gap fix.

If you are weighing this up

If your team is deciding between a frontier API bill and a distilled model you host yourself, or sizing the hardware that would run the result, an outside read before you commit is usually cheaper than the bill. Book a 15-minute intro call, or take a focused hour on the decision with an Expert Call.