A field guide to performance and cost engineering: claim-checking with the scaling and queueing laws, measurement discipline, where GPU and CPU budgets leak, and how to turn a benchmark into a hardware decision.
A frontier lab optimizes an absolute target and has to explore. A challenger optimizes a distance to the leader and can exploit. That is the whole cost asymmetry, and it shows up in GPU hours and in a bike race.
I used whisper.cpp on an RTX 5070 to transcribe about 100 hours of podcasts in roughly three hours, then turned the Markdown output into a searchable prompt knowledge base.
Most AI infrastructure teams spend 35-60% more on GPU compute than they need to. The cause isn’t cloud pricing. It is architecture, and it is fixable.
GPUs are no longer the bottleneck. Data movement is. The bottleneck in AI infrastructure has moved from compute to data movement. The claim was articulated in an online talk I attended recently, and it matches what I have seen for years across very different systems: raw compute stops being the Read more