TL;DR. A frontier lab has no model above it to imitate, so it optimizes an absolute target and has to explore. A challenger optimizes a distance to the leader, which is a dense differentiable signal available on every token, so it can exploit. That is the whole cost asymmetry, and it is measurable: Qwen3 reached a better score through on-policy distillation for 1,800 GPU hours where reinforcement learning needed 17,920. The same shape shows up in a bike race, where riders in the bunch face a tenth of the drag the leader is paying. This is a hypothesis about why, not a measurement, and I am explicit about which half of it I can stand behind.
Somebody else has already done the hard part
In a companion post I went through how big models teach small ones: soft labels instead of a bare answer key, feature matching instead of a written report, synthetic problem sets that scale, and the awkward fact that a student inherits its teacher’s ceiling along with its teacher’s skill. That piece is about how the teaching works.
This one is about why the arrangement exists at all, and why being the model everyone copies costs so much more than being one of the copies. It is shorter, more speculative, and it comes out of a different part of my background.
So let me be plain about which half of it I can stand behind. I have never trained a frontier model and I am not going to pretend otherwise. What I do have is a PhD in numerical optimization, where the balance between exploration and exploitation was the object of study rather than a metaphor, and twenty years since of making optimizers converge on real hardware. Take the optimization half of what follows as something I know, and the mapping onto LLM training as a hypothesis worth arguing with.
The frontier model is the student with no teacher
Start with the asymmetry the school analogy circles without ever naming. The frontier model has nothing above it to imitate. Everybody else has someone to copy. That one fact changes the objective function, and the objective function is what sets the bill.
- A frontier lab optimizes against an absolute target: be more capable than anything that currently exists. There is no reference output to match, so progress gets measured with benchmarks. Sparse, noisy, delayed, partly gameable, and available only at the end of a long rollout.
- A challenger can optimize against a relative target: get as close as possible to what the leader produces. That objective is a distance. It is defined on every token, it is differentiable, and it is cheap to evaluate.
In optimization vocabulary that is the exploration and exploitation split, and the two regimes have famously different costs. NSGA-II searches globally: population-based, derivative-free, no starting point, thousands of evaluations before it converges, because it is mapping a landscape nobody has seen. L-BFGS-B does the opposite: start near the answer, follow the gradient, converge in tens of iterations. Neither is better. They solve different problems, and the thing that separates them is whether you have a gradient and whether you have a decent starting point.
Those two are not random picks. They were my daily instruments for years, and the lesson that stuck is that the gap between them is not an implementation detail. It is a property of what you knew before you started.
A teacher model hands the challenger both things. The starting point is the teacher’s output distribution. The gradient is the divergence from it.
The exploration tax has a price tag
This is measurable, and it has been measured. Kevin Lu’s write-up at Thinking Machines Lab (October 2025) pulls the comparison out of the Qwen3 technical report:
| Post-training method | AIME’24 | GPU hours |
|---|---|---|
| Off-policy distillation | 55.0% | n/a |
| plus reinforcement learning | 67.6% | 17,920 |
| plus on-policy distillation | 74.4% | 1,800 |
A better score for roughly a tenth of the compute. Their explanation is information-theoretic and worth quoting directly: reinforcement learning “only teaches O(1) bits per episode”, while distillation “teaches O(N) bits per episode, where N is the number of tokens”.
That is the exploration tax stated precisely. RL against an outcome reward gets one bit at the end of a long trajectory: did that work or not. With one bit, you have to search. Distillation gets a correction on every token, so it can descend. One bit per episode against N bits per episode is the gap between NSGA-II and L-BFGS-B written in a different notation.
Two places my own version needed fixing
The exploration does not happen in the weight optimizer. Leaders and challengers both run gradient descent on the inner loop, Adam or a cousin. Nobody is running a genetic algorithm over a trillion parameters. The exploration lives one level up, in the outer loop: which architecture, which data mixture, which post-training recipe, which RL environment, which reward. That outer loop is a derivative-free search over a small number of extremely expensive evaluations, which is precisely the regime where population methods and Bayesian optimization earn their keep, and precisely the regime that costs the most per unit of progress. Inside RL post-training, the exploration is over trajectories under a sparse reward. So the honest claim is not that leaders use global optimizers and challengers use local ones. It is that leaders pay for the search over which recipe and which objective work at all, and challengers inherit the answer.
Knowing the target value is worth nearly as much as knowing the gradient. A frontier lab does not know what score is achievable. A challenger does, because the leader published it. Any optimizer behaves differently when the value of the optimum is known: you know when to stop, you know when a run is going badly, and you can kill it early instead of paying it out. On a training budget that is not a rounding error. It is also a large part of why the open models chasing the frontier stay a fairly constant distance behind it. Epoch AI puts that distance at roughly nine months on GPQA Diamond, for models small enough to run on a single consumer GPU. That gap is cheap to close and impossible to jump, because closing a known distance and extending one are not the same problem.
The peloton
Which brings me to the bike race at the top of this page, where the physics is measured rather than felt. Blocken and colleagues put a 121-rider peloton through CFD simulation with wind tunnel validation and found that riders in the mid rear of the bunch see their drag fall to 5% to 10% of that of an isolated rider. Opening the road costs ten to twenty times what following it costs.
That the ratio lands close to the GPU-hour ratio is a coincidence and I am not building anything on it. What transfers is the shape. The cost of being in front is not slightly higher. It is an order of magnitude higher, and it is paid by whoever is in the wind.
The rest of what a cyclist knows about that arrangement transfers too, including the parts nobody enjoys.
- You do not win from the wheel. Sooner or later the follower has to come out into the wind and pay full price like everyone else. A challenger optimizing a distance to the leader converges to the leader and stops there. That is the same ceiling the companion post reaches from the teaching side, arriving here from a completely different direction.
- Drafting works only while the leader is visible. Hide the reasoning traces, forbid training on outputs by contract, stop shipping open weights, and the slipstream closes. The follower’s cost advantage is not a law of nature. It is a function of what the leader currently allows you to see.
- The bunch is the safe place, and it is where nobody contests the finish. If your product has to be at the front, sitting nine months back in the bunch is not a saving. It is a decision to race for second.
What this changes if you are the one signing
The useful part of this framing is that it tells you what frontier pricing actually buys. It is not a margin on tokens. It is the price of somebody else having done the exploration, on an objective nobody could evaluate cheaply, in a landscape nobody had mapped. Sometimes that is exactly what you want to pay for. Often it is not, and the bunch is right there.
Three questions decide which case you are in.
- Is this work on your innovation frontier, or is it plumbing? For what your product competes on, those nine months matter and the premium is buying the win rather than the tokens. For classification, extraction against a schema and routing, nine months is irrelevant and you are paying a leader’s price for a follower’s task. Most organizations run both at once, and the expensive mistake is one policy for both.
- Can you evaluate your own outputs cheaply? The challenger’s whole advantage comes from having a dense, cheap objective. If you cannot score results automatically, tests passing, schema validating, ground truth on hand, then you do not have that advantage. Your distillation project quietly turns back into an exploration problem, funded by you instead of by a lab.
- What happens when the slipstream closes? Your cost advantage depends on what the leader currently lets you see, and on terms that may forbid using their outputs to train a competitor. Price the scenario where that changes, the way you would price any single-source vendor.
Then there is the hardware, which is my day job: sizing the platform that will run it. A model you own has to run on iron you specify: VRAM per inference node, KV cache at your real context length, and what that number does once an agent loop gets chatty. On-policy distillation moves the question again, because the student generates rollouts during training and that is a very different job to size than a straight fine-tune. Different purchase, different rack, often a different vendor. Worth working out before the quote arrives, not after you have signed it.
I have put numbers on the two neighbouring versions of this decision elsewhere: what a distillation project actually costs, and why a panel of cheap models can beat one expensive one.
Sources
- Kevin Lu, “On-Policy Distillation”, Thinking Machines Lab, October 2025. The Qwen3 GPU-hour comparison and the O(1) against O(N) bits framing.
- Blocken et al., “Aerodynamic drag in cycling pelotons: new insights by CFD simulation and wind tunnel testing”, Journal of Wind Engineering and Industrial Aerodynamics 179 (2018), 319 to 337. Where the 5% to 10% figure comes from.
- Epoch AI, the gap between frontier models and models that run on a consumer GPU. Where the nine months comes from.
- Sergio Paniego, “Distillation in 2026 (so far): which frontier models use it and how”, Hugging Face, July 2026. Which families teach off-policy and which teach on-policy.
If you are weighing this up
If your team is deciding between a frontier API bill and a model you train and host yourself, or sizing the hardware that would run the result, an outside read before you commit is usually cheaper than the bill. Book a 15-minute intro call, or take a focused hour on the decision with an Expert Call.
0 Comments