Railroads were never really about trains. The trains were the visible part, the thing you could photograph. The actual event was the decision to lay track toward places that did not yet have a reason to exist, on the bet that reasons would show up once the track did. Capacity first, demand later.
That is roughly the shape of the announcement from AWS and NVIDIA, who say they will deliver 2 million additional GPUs along with next-generation infrastructure aimed at agentic and physical AI. The number is the headline. The framing is the more interesting artifact.
Read the Category, Not the Count
Two million is a figure most of us cannot hold in our heads with any precision. It is large. Fine. What I find more informative as a researcher is that the announcement names its intended workloads explicitly, and those workloads are agentic AI and physical AI. Neither of those is a synonym for “training bigger models.”
For most of the past several years, the implicit story behind large GPU commitments was pretraining. Bigger runs, more tokens, longer schedules. That workload has a distinctive profile: enormous, tightly coupled, synchronous, and relatively predictable once you have committed to a run. You know roughly what you need and for how long.
Agentic workloads do not behave that way, and anyone who has tried to run a nontrivial agent system in production already knows it. The compute profile is bursty and awkward. A single agent trajectory might involve dozens of model calls interleaved with tool invocations, retrievals, code execution, and waiting on external systems that have no interest in your GPU utilization curve. Multiply that across concurrent sessions and you get demand that spikes unpredictably, holds state across long horizons, and punishes you for every millisecond of latency in the loop because the latency compounds with each step.
Why Inference Is the Harder Engineering Problem
Training is expensive but forgiving in one specific way: nobody is waiting on the other end of it. Agentic inference is the opposite. The user, or another agent, is waiting. And unlike a single chat completion, an agent’s cost is not one forward pass. It is a tree of them, with retries, reflection steps, and tool calls that each add round trips.
This changes what infrastructure has to be good at:
- Memory bandwidth over raw FLOPs. Autoregressive decoding at long context lengths is memory-bound. Growing KV caches across long agent trajectories make this worse, not better.
- Interconnect for serving, not just training. Once models are too large to sit on one device, every token in a multi-step agent loop pays a communication tax.
- Scheduling under uncertainty. A fleet sized for average agent load will fail at peak. A fleet sized for peak sits idle. Neither is acceptable at this scale, which makes the orchestration layer as consequential as the silicon.
- Statefulness. Agents are long-lived processes with memory. Infrastructure designed around stateless request-response is a poor fit for something that needs to remember what it tried twenty minutes ago.
The word “next-generation infrastructure” in the announcement is doing quiet work here. Two million GPUs bolted onto an architecture designed for a different traffic pattern would be an expensive way to learn a lesson.
The Physical AI Half
Pairing agentic AI with physical AI in the same sentence is not a marketing flourish, and I think it is the most technically coherent part of the framing. Robotics and embodied systems need three distinct compute regimes: simulation at enormous scale to generate training data that reality cannot supply fast enough, training on that synthetic data, and then low-latency inference on or near the device.
The first regime is the one that actually explains large cloud capacity. Simulated environments for embodied learning are voracious in a way that text corpora are not. You are manufacturing experience rather than scraping it. That is a compute-for-data trade, and it only pencils out when compute is abundant enough to be spent carelessly.
What I Am Watching For
Capacity announcements are, structurally, statements of belief about the future rather than descriptions of the present. Track laid toward towns that may or may not grow. The belief being expressed is that agents will consume compute at a scale that dwarfs what conversational AI consumes today, and that embodied systems will follow.
That belief is plausible. It is also not yet demonstrated at the level of reliability that would justify it. Agent systems still fail in ways that are hard to characterize, harder to debug, and expensive to retry. Every failed trajectory is compute spent on nothing.
Which is why I read a commitment of this size less as a prediction that agents will work, and more as a bet that the remaining problems are engineering problems rather than conceptual ones. That is a real distinction, and it is where the interesting research still lives. Infrastructure of this scale removes compute as an excuse. What is left is architecture.
🕒 Published: