Inference Is Becoming an Ownership Problem

Part 1 out of 3: Why enterprises will move from renting model access to controlling their own inference, and the layer that is still missing

By Ori Barzilay, Partner at Team8, October 2026 

In brief. This year, inference is set to outspend training on AI cloud infrastructure, and it is the part of the stack enterprises control least. The answer is not to own every GPU. It is to own the decisions: which model runs, where, in what configuration, against what service objective. The layer that makes those decisions does not yet exist in mature form, and the discipline that needs it, inference engineering, is just forming.

In July, Ito published what happened when it moved its largest AI workload, a test-running agent that consumes roughly 100 billion tokens a week, from GPT-5.3-Codex to MiniMax M3. Median LLM cost per pipeline run fell 55 percent. But median step time rose from 23 to 29 minutes, the failure rate nearly doubled to 4.5 percent, and the cheaper model needed 23 percent more turns and 2.5 times the tokens. Savings ranged from 81 percent on some repositories to 13 percent on others.¹ Ito kept the cheaper model; for its workload, the tradeoff was worth it. For a customer-facing agent, it might not have been.

One model swap, five outcomes.

Savings varied widely by repository.

The second result came from a lab.

Researchers at Microsoft and the University of Illinois held a cluster's model fixed and kept reconfiguring the cluster as load shifted: instance count, how the model was split across GPUs, and GPU clock speed.

Cost to the customer fell 61 percent and energy use 53 percent, while latency targets held.2

Figure 1. One model swap, five outcomes. Savings varied widely by repository.

The second result came from a lab. Researchers at Microsoft and the University of Illinois held a cluster's model fixed and kept reconfiguring the cluster as load shifted: instance count, how the model was split across GPUs, and GPU clock speed. Cost to the customer fell 61 percent and energy use 53 percent, while latency targets held.²

Figure 2. What continuous reconfiguration recovered, with the same model on the same hardware.

Two results, two levers: which model runs, and how the hardware runs it. For many enterprises, getting a return on AI now takes continuous decisions about both, made against a target for cost, latency, and quality. The first phase of enterprise AI hid those decisions behind an endpoint, which suited experimentation. In production, inference shapes both product quality and operating cost, and a change to one can sharply move the other. Few organizations have a system for making them yet.

Our view is that enterprises will move from renting model access to owning their inference economics: the decisions that set what their AI costs, how it performs, and where it runs. Where it pays, they will own the fleet underneath too. The layer that makes this possible is the most important whitespace in AI infrastructure today.

The control plane and the data plane

Software-defined networking split networks into a control plane that decides where traffic goes and a data plane that moves it. Inference is heading the same way. The data plane is wherever tokens are produced: your own GPUs, a neocloud's, a lab's API. The control plane decides which model runs, on which hardware, in which region and configuration, against what quality bar.

Owning those decisions matters even when the tokens are rented: which model, which provider, which service tier. Owning the data plane is a narrower call, made workload by workload. For high-volume, high-stakes, repetitive workloads, the answer is increasingly a baseline fleet you control, plus rented capacity for spikes. Three forces are making that the right choice for a growing share of workloads: cost, control, and sovereignty.

1. Cost

Gartner expects inference to overtake training this year in spending on AI-optimized cloud infrastructure, $23.3 billion against $19 billion, and to reach 59 percent of that spending in 2027.³ For AI-native companies, inference efficiency flows straight to gross margin.

Falling token prices make the pressure worse, because cheaper inference invites more of it. Google now processes more than 3.2 quadrillion tokens a month, seven times as many as a year earlier.⁴ On OpenRouter, reasoning models went from a negligible share of tokens to more than half during 2025, and the average prompt grew fourfold.⁵ Total spend rises even as unit prices fall, so the savings come from deciding which tokens to spend and how the hardware spends them.

The price list does not predict the bill. Ito's cheaper model consumed 2.5 times the tokens. Nous Research found the same pattern more broadly: on comparable tasks, open-weight reasoning models used 1.5 to 4 times as many tokens as closed ones, sometimes enough to cost more per query.⁶ It cuts the other way just as sharply. In a peer-reviewed study on a vulnerability-detection benchmark, a 4-billion-parameter open-weight model matched the best frontier model's accuracy at one sixty-sixth of the cost per task.⁷ What matters is cost per task, and only the workload itself can measure it.

2. Control

A company that rents its inference also rents the controls over it. It can pin a model version only until the provider retires it. OpenAI promises six months' notice for generally available models and less for variants and previews. In July it shut down more than a dozen models, including GPT-5-Codex, and it scheduled fine-tuned GPT-4 and GPT-3.5 Turbo models that customers trained themselves for shutdown on October 23.⁸ Quality can change underneath the customer, too: in September 2025, Anthropic disclosed that three infrastructure bugs had intermittently degraded Claude's responses for several weeks, hitting 16 percent of one model's requests at the worst hour.⁹ Consistent low latency costs extra, through tiers such as OpenAI's priority processing.⁸ In a survey by Broadcom, which sells private cloud through VMware, performance was cited as often as cost predictability as a reason to bring workloads back from public cloud.¹⁰

3. Sovereignty

Gartner expects spending on sovereign cloud infrastructure to reach $80 billion in 2026, up 35.6 percent, as organizations outside the U.S. and China respond to rising geopolitical tension.¹¹ Security and compliance was the top reason enterprises gave Broadcom for moving workloads off public cloud, cited by 51 percent.¹⁰ Banks are already running models on their own infrastructure. BNP Paribas serves open-source and Mistral models from GPUs in its own data centers, which it says keeps deployment "within the Group's infrastructure, in compliance with data privacy requirements." HSBC has signed a multi-year deal to self-host Mistral's models on its internal systems.¹² For a bank, an insurer, or a government agency, whether renting inference is allowed often comes before whether owning it is cheaper. And inference run in-house still needs a control plane on top.

What ownership puts on the table

Ownership pays off through the two levers from the opening examples.

The first is how the hardware runs a model. The best serving configuration changes as models, hardware, and traffic evolve, and there are too many options to search by hand. The Vidur project at Microsoft Research and Georgia Tech estimated that exhaustively testing configurations for one 70-billion-parameter model would take about 42,000 GPU-hours.¹³ NVIDIA's own configuration tool, which claims up to 1.7 times the throughput of manual tuning, is built for initial setup.¹⁴ What happens after setup is the gap: traffic shifts, and each new model can invalidate the benchmarks, routing policies, and capacity plans built for the last one. Reconfiguring continuously as load shifted is where the second result's 61 percent came from.² Closing it takes a control loop: observe production, test alternatives, validate results, and adapt, without shipping any change that has not been proven safe.

Mixture-of-experts models cut both ways. MiniMax M3, the model Ito switched to, holds about 428 billion parameters but activates only about 23 billion per token.¹⁵ All the experts must sit in memory while only a few compute, so serving it well means spreading them across many GPUs, and for the largest models across nodes. That enlarges the configuration space, which favors a control loop. It also pushes the largest models toward rack-scale serving, which favors providers with scale for any workload that does not justify its own fleet.

The second lever is which model runs, and until recently there was little to choose from. Epoch AI puts the best open-weight models about four months behind the closed frontier this year. On cyber tasks, the UK AI Security Institute measures four to seven months, down from six to ten through most of 2025.¹⁶ Usage has followed: OpenRouter, which routes some 25 trillion tokens a week, found open-weight models at roughly a third of its traffic by late 2025.⁵

Figure 3. The capability gap narrowed, and open-weight models now carry a large share of real traffic.

The economics follow. Pinterest's CEO says open-source models can reach similar performance at less than 10 percent of the cost of leading proprietary ones.¹⁷ Fireworks, one of the largest managed inference providers, says more than 95 percent of the tokens it serves come from models specialized on customers' own data.¹⁸ Replit routes each task to the model best suited to it and, in its own testing, got the same output quality at 65 percent lower cost than the previous version of its Max Mode.¹⁹

Most inference is still rented. That is the point.

The obvious objection is that none of this describes the average enterprise, and the data agree. IDC finds that 84 percent of spending on AI infrastructure goes to cloud and shared environments.²⁰ Ramp's spending data shows about 44 percent of U.S. businesses paying Anthropic, and about 40 percent paying OpenAI, for subscriptions or tokens.²¹ Accenture says about one organization in four runs open-weight and proprietary models in equal measure.²² Most inference is rented, and much of it should stay that way. The objection is right about today. It misses where the pressure is building.

The real split is by workload. Teams rent for capability-first work: the newest model, the hardest problem, the pilot. They run their own fleets for high-volume, stable work where the model is good enough but the bill, or the service level they owe their customers, is not. That category is a minority of workloads and a majority of the pain, and it grows with every pilot that reaches production. Broadcom's survey of 1,800 IT leaders at large enterprises found 56 percent running or planning production inference on private cloud, and public-cloud use for those workloads down from 56 to 41 percent in a year.¹⁰ Rented capacity comes with someone else's optimizer. Owned capacity comes with open-source engines and no one to keep them tuned.

Figure 4. Where large enterprises run production inference, and why. Survey by Broadcom, which sells private cloud.

The market has made the case for a hybrid model: rent for experimentation and bursts; own the workloads where cost, latency, quality, or sovereignty justify control. But a hybrid fleet creates a new problem. Someone has to decide, continuously, what runs where—and prove that each change is safe.

That is the missing layer.