The Invisible Trillion · The Architecture That Works

What Compute Used Well Looks Like: Self-Driving

After all of that, the hopeful part — because the architecture that works isn’t a thought experiment. It’s already on the road.

Driving for a living is, for most of the people who do it, brutal. Long-haul truckers spend weeks at a stretch on the road, eating badly, sleeping in the cab, away from their families, soaking up the cardiovascular damage of fifteen-hour days in a seat, with crash-fatality rates among the worst of any job in the country and a life expectancy to match. It has been a real path into the middle class, especially for immigrants and for Black drivers, and the human cost of it has been enormous and mostly uncounted by the economy that depends on the freight.

Self-driving is trying to end that job, and after a couple of decades of grinding it has reached the point where, in the cities where it runs, the autonomous cars crash less often than the humans. The compute behind that has a very particular shape. The training is a legitimate hyperscale workload — huge, centralized, done by the people who need it. The inference lives on the edge, on the car, because a vehicle deciding whether to brake can’t wait on a round trip to a data center; the latency and the reliability requirements force the computing to be local. Heavy centralized training, distributed edge inference. The physics of the problem left the engineers no choice, so they built the right thing.

That’s the model. If the rest of the AI buildout were shaped like this, most of the inflation in this chapter would simply not exist. The training compute would still get built, in the volumes real training demand justifies, by the labs that need it. The inference would move to the edge — onto phones and laptops and enterprise hardware — for the large majority of workloads. And the hyperscale inference buildout currently rising in residential neighborhoods would be maybe a third of its size, or less.

The detail I find most telling is that some of the same operators who got self-driving’s architecture right are getting general AI’s architecture wrong — Tesla under Musk on the first, xAI under Musk on the second. That tells you the architecture isn’t a question of capability. It’s a question of incentive. When the physics forces the right answer, the operators build it. When the physics doesn’t force it, they build whatever maximizes their own returns, even when a different design would be better for everyone else — which isn’t a moral failing peculiar to any one person, just what happens in a market that doesn’t price externalities.

General AI’s architecture is wrong mostly because the people who’d benefit from the right one — everyone paying for inference, the communities hosting needless data centers, the planet absorbing the extra power — aren’t the people who design it. Self-driving worked because the physics wouldn’t let it not. The lesson for the rest of the buildout is that the physics has to be reasserted some other way: by regulation, by consumer demand for edge-default designs, by enterprises voting with their workloads.

And the good news is that the convergence has, slowly, begun — the edge migration, the enterprise awakening, the steadily improving small models are all early signs of it. Self-driving has shown one path. The rest of the industry will, most likely, find some version of it. To the degree the current buildout runs ahead of where that path leads — which is what the inflation hypothesis predicts, and what I think the evidence favors, while admitting it isn’t certain — the question is how much of the excess gets built before the path goes mainstream, and how much damage lands on the host communities along the way.

What Comes Next

So that’s the case: what an honestly-sized buildout would look like, why idle capacity and a frustrating user experience can be true at the same time, how the inflation plays out on the ground at Colossus, what kind of financial machine is paying for it, and what compute used well looks like when somebody is forced to build it right.

What I haven’t touched is the question any reader is now holding: who’s actually paying for all this, and what happens to them if the inflation resolves the way I’ve suggested it might — and, alongside it, what about national security, what about China, what about the world this buildout is supposedly defending. That’s the next chapter, The Bill.

All 19 chapters of The Invisible Trillion · Library · All courses