The Missing Layer in AI Infrastructure
Why inference needs a control plane above models, hardware, clouds, and providers
Part 2 out of 3
Previously in the series: Inference is becoming an ownership problem—not because every enterprise should buy GPUs, but because someone has to own the decisions that determine cost, quality, latency, and placement.
Enterprises will need to manage a distributed inference fleet: models, runtimes, accelerators, regions, and providers, owned and rented. Each workload has its own service-level objective (SLO), and the control plane’s job is to make those SLOs explicit and optimize against them continuously. That takes seeing the whole system, which no single tool does today. In an IDC survey sponsored by Google, 54 percent of organizations ran multiple AI frameworks and hardware platforms, and 92 percent of those said the mix hurt efficiency.23
Not all tokens are equal, and the market already prices the difference. Real-time tokens, for chat, voice, and customer-facing agents, need a fast first token and steady streaming. Batch tokens, for evaluations, data enrichment, and document processing, can wait: OpenAI, Anthropic, and Google all sell batch processing at half price, and OpenAI adds a slower tier at batch rates and a priority tier at a premium.24 Background agents tolerate latency but carry long contexts, so prompt caching drives much of their cost. Steady real-time tokens make the case for owned or reserved capacity; tokens that can wait belong on batch APIs, spot capacity, or an owned fleet’s idle hours. Sorting and placing them is the control plane’s first job.
The missing piece is a vendor-neutral control plane above the orchestrators, not another one beside them. It needs three things: cross-stack tracing that ties every request’s cost and quality to model, configuration, hardware, and region; a closed loop that acts on what it sees; and validation on the customer’s own fleet and traffic. Which lever it pulls first depends on what the company owns. A company that rents APIs can change which model and service tier each task uses, which is where the Ito and Replit savings came from. A company that runs its own fleet can also tune batching, caching, parallelism, and placement, validating each change before production with telemetry no gateway sees. A real control plane uses both, and knows which one each workload needs.
Inference performance is decided at several layers, each optimized today by a different party for a different objective.
Managed inference providers such as Fireworks, Baseten, and Together sit outside the numbered stack. Each is a vertically integrated alternative to layers 1 through 4, sold as a price per token or a dedicated endpoint; the enterprise sees the price, not the levers. To a control plane, each is one more place a workload can run.
Three trends make the stack more heterogeneous. Runtimes are being built for mixed silicon. Gimlet Labs runs a single AI workload across different kinds of hardware at once. In June, Qualcomm agreed to buy Modular, maker of the hardware-agnostic MAX engine, for about $3.9 billion in stock: a chip vendor paying up for the layer that makes chips interchangeable.25 The silicon is diversifying. TPUs and Trainium compete with GPUs on throughput and cost; Cerebras and SambaNova sell speed per user, which suits real-time tokens; Etched and Positron are building chips for inference. In December 2025, NVIDIA licensed Groq’s technology and hired its founder.26 And the fabric that connects chips and nodes is its own layer, and its own investment thesis. NVIDIA, Broadcom, Arista, and HPE Juniper compete there with newer entrants such as DriveNets, Nexthop AI, and Cornelis, some building for the compute that does not want to live inside NVLink.27 Matching each class of token to the right silicon is a placement decision no vendor in the stack will make neutrally.
Most of the waste a control plane recovers is invisible from any single dashboard. The patterns repeat.
- The utilization lie. The GPU reads 90 percent busy while the bottleneck is memory bandwidth or a misconfigured engine, so the team buys GPUs that do not help.
- Prefill and decode fighting for the same GPUs. Long prompts starve token generation; splitting the two phases fixes one traffic mix and hurts another.
- A cache tuned for last month’s prompts. KV and prefix caches sized for traffic that has since changed, thrashing quietly.
- Batching set for the wrong SLO. Batches sized for throughput on a workload that needs a fast first token, or the reverse.
- Faults that look like capacity problems. One throttled GPU or flaky link paces a whole tensor-parallel group, and communication stalls read as low utilization.
- Autoscaling on the wrong signal. Scaling on request count when load is in tokens, so capacity arrives late and lingers.
- A model change that moved the numbers. A provider release, a fine-tune, or a lower-precision format that shifted token counts, latency, or accuracy, with no benchmark that would have caught it.
- Placement that ignores the rack. Replicas placed on nodes with no power or thermal headroom, so effective capacity falls short of nominal.
Each of these shows up first as a cost or quality anomaly, and the root cause is rarely in the layer where the anomaly appears. That is why observing is not enough, and why every fix has to be validated: the change that resolves one of these can cause another.
A control plane can see the waste. It can connect a cost or quality anomaly to the model, engine, hardware, network, region, and workload behind it. The harder question is who can act across that entire stack without being tied to one vendor’s product.
That is where the market structure becomes decisive.