Rethinking Cloud CPU Capacity for Stateful, Bursty AI Agents: Cost, Security and Governance Trade-offs

Open a cost-and-capacity review for an agent platform and the accelerator line item usually dominates the room while the CPU plan quietly falls apart. Agent sessions park context for days, consume almost nothing while parked, then fan out into rapid bursts of tool calls, website access and code execution — a load curve that sizing spreadsheets, virtual-machine boundaries and egress policies were never built to absorb. Mark Lohmeyer, vice president and general manager for AI and computing infrastructure at Google Cloud, framed that shift plainly in a conversation with EE Times: the needs of agentic AI are reshaping data center CPUs and the broader infrastructure wrapped around them.

1. Chat and Agents Are Two Different Capacity Problems

The cleanest way to reset a planning debate is to separate the two interaction shapes. Lohmeyer draws that line explicitly: “Chat is relatively straightforward—there’s a question and an answer, and it relies heavily on inference across GPUs or TPUs,” he said. “With agents, the nature of those interactions is changing quite a bit.”

A conversational request is short-lived and mostly accelerator-bound. An agent deployment takes a user’s high-level intent, divides it into subtasks, and passes those subtasks to a group of agents that may call other APIs, visit websites, and write and run code — work that falls to CPUs. Sessions can last much longer than a chat round trip, involve multiple human users, and require context and state to be maintained across days or weeks.

The utilization curve is where infrastructure teams get caught out. “Agents are often bursty, and often idle,” Lohmeyer said. “They don’t do anything for [say] a day or two until you come back to them, or maybe until some API responds… It’s unique from older CPU workloads, in that it needs to be stateful over a long period of time, but is often idle during that period of time, and then when it’s not idle, it’s highly bursty.”

Managing that pattern with both performance and security intact is, in his assessment, a major industry challenge, and it lands on top of a demand curve that is already steep. “Demand is already very high and is growing rapidly,” he said. “So, how do we provide the infrastructure that can support that growth and do it cost-effectively?”

2. Split the Load: New Agent Compute versus Existing Tool Access

For architecture work, Lohmeyer separates the compute needed to run the agent itself — a new workload requiring new infrastructure — from tasks such as website and API access, which already existed but previously interfaced with humans. “Agentic workloads will have a meaningful impact on both of those elements,” he said.

The existing half is the one teams under-forecast. Those parts of the workload will grow dramatically because agents are far more demanding than humans and can be brought online in hundreds or thousands quickly, while human user counts cannot rise as fast. Every internal API and web property that once absorbed human interaction rates now absorbs machine interaction rates.

Placement follows from that pressure. “Ideally, you’d like to have those agents and their context, the applications and data, relatively close to each other from a latency perspective, for performance reasons,” Lohmeyer said. Locality matters, but enterprises will continue to operate in a distributed world, with most still using multiple clouds and private clouds for now — a reality that keeps high-performance networks important inside data centers and between data centers.

“Dynamic, stateful agents will need to have a new type of computing infrastructure that supports them,” he said. “They’re accessing all the existing apps and data, so the demand there is growing, and all this needs to be connected together through the network. Google is investing in all of these elements.”

3. How the CPU-to-Accelerator Mix Is Moving

For teams building a multi-year capacity model, the optimal CPU-to-GPU compute ratio for agentic AI is still evolving with the workload, Lohmeyer said. What is not in question is direction: the share of CPU compute required is definitely increasing as the shift moves from training foundation models to agentic inference.

“We expect both [CPU and TPU/GPU] workloads to continue to grow rapidly, but the proportion of CPUs is going from a relatively small portion to a meaningful portion, and we think that will continue to evolve as agents expand into new use cases and new environments,” he said.

Google’s own processor portfolio is part of that hardware story.

Google’s Arm-based Axion CPU

4. Workload Signature: Conversational AI Side by Side with Stateful Agents

Planning dimension	Conversational (chat) workload	Stateful, bursty agent workload
Interaction shape	A question and an answer	High-level intent divided into subtasks, passed to a group of agents
Dominant compute	Inference across GPUs or TPUs	API calls, website visits, and code written and run on CPUs
Session lifetime and state	Request-scoped; no long-lived context to hold	Context and state maintained across days or weeks
Utilization curve	Active while the user waits for the reply	Often idle, sometimes for a day or two, then highly bursty
Scaling driver	Human user count, which cannot rise as fast	Agents brought online in hundreds or thousands quickly
Latency placement	Serving path to accelerator resources	Agents, their context, the applications and data relatively close to each other
Multi-site reality	One request/response path per turn	Distributed world: multiple clouds and private clouds, connected through the network
Isolation primitive	Service or virtual-machine boundary	MicroVM boundary around each agent on a shared server
Primary cost lever	Accelerator utilization	CPU hardware utilization and the resulting cost per agent

5. The Stack-Level Answer: Co-design, Hibernate, Reload, Isolate

The answer to agentic inference will keep depending on co-design across the stack, from CPUs and storage to memory and networking, Lohmeyer said. “Google Cloud has always had a philosophy of workload-optimized infrastructure,” he said. “This is more important than ever in the agentic era.”

Because these workloads are bursty and often leave the CPU idle, agents can effectively be “hibernated”: their state is moved to storage so the CPU can serve other agents, then reloaded quickly when the session is needed again. “This requires a new type of orchestration in software, and in this environment, having really high-performance storage that’s attached to that server, so I can restart it quickly, is important,” he said.

Isolation is the second lever. Google Cloud uses microVMs, described as a “slimmed down” version of a virtual machine, to create a secure boundary between agents packed onto the same server.

“Security and governance for agents is absolutely critical, and Google is investing a lot in every layer of the stack,” he said. At the lowest layer, virtualization is key. Google plans to support multiple ways to sandbox agents, with the microVM particularly important: because a microVM has a streamlined feature set, performance improves, and its smaller attack surface helps security compared with traditional virtual machines. The company is also investing in agent gateways that can provide egress controls at the network level, along with other higher-level controls.

Looking ahead, running thousands of microVMs for agents will affect core density, storage, how storage is attached, and the orchestration software used to manage it all, Lohmeyer said. Higher hardware utilization is the expected benefit, and that should lower the cost per agent.

“This will allow our customers to scale their business, but do so with a good business model, good profitability, but without having to sacrifice performance as they are scaling,” he said. “These capabilities will be very important going forward.”

6. Traditional VM versus MicroVM for Agent Fleets

Evaluation axis	Traditional VM	MicroVM
Form factor	Full-featured virtual machine	A “slimmed down” version of a virtual machine
Density on the same hardware	100 instances as the reference point	1,000 instances as the reference point
Feature set	Broad	Streamlined
Performance implication	Baseline	Performance improves because the feature set is streamlined
Security implication	Larger attack surface	Smaller attack surface helps security
Boundary for packed agents	Coarser unit of co-tenancy	Secure boundary between agents packed onto the same server
Role in sandboxing strategy	One of the options enterprises already run	Particularly important among the multiple ways to sandbox agents
Downstream design pressure	Existing VM capacity models	Core density, storage, how storage is attached, and the orchestration software that manages it all
Expected business outcome	Utilization as it stands today	Higher hardware utilization and a lower cost per agent

7. Common Misconceptions in Agent Capacity Planning

• “Agents are just chat with more turns.” Chat relies heavily on inference across GPUs or TPUs; an agent run decomposes intent into subtasks, calls APIs, visits websites and executes code, which is CPU work with a different failure profile.
• “CPU is a footnote in an AI build-out.” The share of CPU compute is explicitly moving from a relatively small portion toward a meaningful portion as the center of gravity shifts from training foundation models to agentic inference.
• “Headcount is the right scaling signal.” Human user counts cannot rise as fast as agents, which can be brought online in hundreds or thousands quickly; forecasting off seat counts under-forecasts the load.
• “Locality means consolidating onto one cloud.” Close placement matters for latency, but most enterprises still operate multiple clouds and private clouds for now, so inter- and intra-data-center networking stays on the critical path.
• “One VM per agent is a safe enough boundary.” The traditional virtual machine is the comparison point for a reason: the streamlined microVM exists to shrink the attack surface and raise density for packed fleets.
• “Cost-effectiveness is purely a hardware question.” Restart speed depends on a new type of orchestration in software plus high-performance storage attached to the server, not on CPU selection alone.

8. Preparing Your Cloud: A Working Checklist

1. Classify every agent capability as either new agent compute or existing tool access, and forecast those two halves separately.
2. Model sessions by state lifetime — hours, days, weeks — rather than by request count, and record how long each fleet stays idle.
3. Measure the idle-to-burst ratio per fleet; it is the input to CPU sizing, storage placement and orchestration choices.
4. Design an explicit hibernate path: move session state to storage, release the CPU to serve other agents, and define the reload target.
5. Attach high-performance storage to the server that owns the session, because restart speed gates the whole density argument.
6. Choose the isolation primitive per co-tenancy tier, and evaluate microVM boundaries for agents packed on shared hardware.
7. Plan network-level egress controls, including agent gateways, alongside higher-level governance controls.
8. Re-forecast the internal APIs and web properties agents will call, since they now receive machine-rate traffic instead of human-rate traffic.
9. Co-design across CPUs, storage, memory and networking instead of optimizing a single layer in isolation.
10. Track cost per agent as the business metric that links utilization work back to scalability and profitability.

9. FAQ

Is agentic AI a GPU problem or a CPU problem?

Both curves grow. The shift from training foundation models to agentic inference increases the share of CPU compute required, while accelerator workloads keep expanding as well; the CPU-to-GPU ratio is still evolving with the workload.

What actually moves when an agent is hibernated?

The session state. It is written to storage so the CPU can serve other agents, then reloaded quickly when the session resumes — which is why attached high-performance storage and a new type of orchestration in software are the binding constraints.

How dense can a server get with agent workloads?

Google Cloud’s reference point is density at the isolation layer: rather than 100 traditional VMs, the same hardware might run 1,000 microVMs, each forming a secure boundary between agents packed onto the same server.

Is the microVM the only sandbox option?

No. Multiple ways to sandbox agents are planned, with the microVM particularly important because its streamlined feature set improves performance and its smaller attack surface helps security compared with traditional virtual machines.

Does low idle activity mean agents are cheap to run?

Only if the platform can reclaim the idle capacity. Idle sessions become value when state is parked in fast storage and the CPU serves other agents; otherwise long-lived state simply reserves compute that nothing is using.

What is the governance layer for agents?

It spans the stack: virtualization at the lowest layer, sandbox boundaries around each agent, agent gateways that can provide egress controls at the network level, and other higher-level controls.

10. Action Recommendations: Sequence over Wishlist

1. Instrument first. Capture idle duration, burst amplitude and state lifetime for one representative agent fleet before touching a capacity model.
2. Prove hibernate and resume on that fleet, and publish the reload time as a hard service objective.
3. Decide the boundary, then the controls: pick the sandbox primitive for packed agents, then layer egress controls and higher-level governance on top.
4. Re-forecast the existing-app half of the load at machine rates, and validate latency placement for agents, context, applications and data.
5. Convert the utilization gains into a unit economic, so the CPU conversation ends as a cost-per-agent discussion rather than a core-count discussion.

The practical takeaway for cloud architects is that the CPU tier has stopped being a commodity line item and become its own design surface, with a workload signature that demands state handling, fast reload and tighter isolation at the same time. Treat the idle-to-burst curve as the input to orchestration and storage decisions, treat the sandbox as a governance choice rather than a runtime accident, and let cost per agent be the metric that ties the architecture back to the business case. Teams that sequence those three moves first will be the ones that can grow an agent estate without growing waste along with it.