The CPU Comeback—And How Talent Drives Performance

The CPU Comeback—And How Talent Drives Performance

For the last few years, the direction of AI infrastructure felt obvious: if you wanted better performance, you added accelerators.  

GPUs became the center of gravity in data centers. Entire roadmaps from hardware and software to hiring were built on the assumption that AI is fundamentally a parallel computing problem.  

For a specific phase of AI, that assumption was right. But it was also too narrow, because it was built around training.  

But what’s happening in AI now is very different from the first phase, and it’s making a CPU comeback inevitable.

The Workload Has Changed: How Inference vs Training Fuel the CPU Comeback  

Let’s zoom out to look at the broader pattern behind what we’re seeing. According to the view proposed by SemiVision Research: 

“The first wave was NVIDIA GPUs. The second wave was data center storage and optical modules. The third wave will no longer belong solely to accelerators. It is beginning to shift back toward the central processor, the CPU.” 

What’s happening at the workload level is a direct reflection of that third wave. As AI systems move from development (training) to production (token delivery and inference), the nature of the workload is shifting substantially.  

Inference is quickly becoming the dominant form of compute. By most estimates, inference workloads are on track to represent roughly two-thirds of total AI compute within the next few years, a sharp increase from about one-third just a few years ago.  

At the same time, inference is where costs live. Some estimates suggest it accounts for 80–90% of total lifecycle compute cost for deployed AI systems. 

This shift is also visible in how workloads are structured at the request level. With a traditional inference call — a user submits a prompt, and the model returns a response. That’s relatively tractable.  

What’s increasingly common is something else: multi-step agentic workflows where models call tools, retrieve data, reason across intermediate outputs, and loop back. These sequences can involve dozens of discrete operations per user interaction. 

SemiVision writes: 

“As AI evolves from one-shot prompting into agentic workflows, tool calling, multi-model chaining, task decomposition, memory management, cross-node collaboration and real-time response, the nature of the workload becomes increasingly fragmented, dynamic, and dependent on orchestration and control.” 

Researchers at MIT and others have noted that agentic workloads introduce irregular memory access patterns and unpredictable branching that are fundamentally mismatched with how GPU compute is provisioned. 

In lay terms: companies that built their AI inference infrastructure around training-era assumptions are now running workloads it was never designed for. Training is episodic; you run it, you finish it, you move on.  

But inference is continuous. It runs every time a user interacts with a system—millions or billions of times per day.  

From Linear Execution to Decision-Making Systems: The GPU Model Under Strain  

As you’ve read, today’s systems are increasingly more complicated. 

Modern AI applications, especially those described as “agentic,” are built around sequences of decisions, not just computations. As noted in a SemiEngineering interview with Arm’s Jeff Defillippi, these workloads introduce branching logic, memory access, and multi-step coordination, rather than a single linear execution path.  

This turns inference into something closer to a workflow than just a single computation, and workflows behave differently. They branch, stall, depend on memory, and require coordination between multiple components.  

That’s a control problem, not just a scaling problem.  

None of this diminishes the importance of GPUs. They remain unmatched for executing large volumes of parallel operations, and that’s not changing. But they were never designed to manage systems. They are optimized for throughput, not coordination.  

When workloads become irregular, the limiting factor is how effectively you can manage what happens next, much more than just adding raw compute power.  

This is already visible in system behavior. In large-scale AI environments, inefficiencies often come from the surrounding system: data movement, synchronization, and workload scheduling overhead:  “Token cost is not determined only by how chips compute, but also by how data flows.” 

In fact, multiple analyses have shown that data movement can consume as much or more energy than the computation itself.  

The GPU utilization data makes this concrete. As noted in our previous post, Beyond the Processor: The Connectivity Bottleneck, Astera Labs’ COO Sanjay Gajendra has pointed out that in modern AI infrastructure, GPUs are idle roughly 50% of the time—not because the compute isn’t powerful enough, but because the system around them can’t keep up.  

This is the clearest signal that the bottleneck has moved. Utilization below 50% on hardware that costs tens of thousands of dollars per unit is a coordination and data movement problem, not a compute problem.  

Adding more accelerators in that environment doesn’t increase output. In some cases, it increases coordination overhead faster than it adds useful throughput. The system becomes harder to manage, not easier.   

This is the dynamic that changes the economics of AI infrastructure, and it’s why the limiting factor is no longer what any single chip can do. Adding more compute doesn’t necessarily solve the problem, and it can make it harder to manage.  

The industry is taking note. In a recent SBT C-Suite interview, Moshe Tanach, CEO of NeuReality explains how agentic AI stresses systems:

“I think agentic AI is definitely something that is stressing what we do… agentic AI and all the agents using LLMs 10 times, 50 times, 100 times more than before for every job, every request kind of redefined how fast we should work… What we saw is that when you’re trying to keep the GPU busy, the head node is being stressed—the head node being the CPU that manages it.” 

In fact, NeuReality’s technology is working to manage these exact challenges. 

The CPU’s Role Becomes Visible Again  

This is where the CPU re-enters the picture: not as a replacement for accelerators, but as the layer that makes them usable. 

If we think in terms of system economics: 

“If the future AI factory is really about end-to-end token production efficiency rather than isolated peak accelerator performance, then the CPU’s role begins to shift from ‘background controller’ to ‘system coordination hub.'” 

And as EE Times notes, even in the most advanced AI clusters, CPUs remain essential for orchestrating workloads and managing the system layers that accelerators depend on. They:  

• Run the operating system  
• Manage networking and I/O  
• Schedule workloads across devices  
• Coordinate memory access  
• Handle control flow and branching logic  

This isn’t a new role for the CPU. What’s new is the scale at which they’re operating, and the degree to which system performance depends on how well CPUs executes these functions.  

In large clusters where dozens or hundreds of accelerators have to coordinate across memory tiers, networking fabrics, and workload queues, the CPU’s scheduling and workload orchestration decisions have a compounding effect. A poorly managed workload queue doesn’t just affect one accelerator; it can cascade across an entire cluster. 

The implication is straightforward according to SemiAnalysis and others: “In the future, the most valuable capability may not be who has the most compute, but who can prevent expensive compute from going idle.” 

Intel has been explicit about this in its public roadmap communications, framing next-generation server CPUs not as compute competitors to GPUs but as orchestration engines—the layer responsible for making heterogeneous systems coherent. AMD’s data center roadmap reflects the same logic. The positioning has shifted from “CPU vs accelerator” to “CPU as the control plane that makes accelerators usable.”  

That’s a meaningful reframe, and it’s one the talent market has been slow to absorb.

The Structural Shift from Compute to System Design  

What’s changing now is how explicitly this is all being addressed in architecture. The industry is moving toward heterogeneous systems, optimizing different components for different roles.  

The performance gains aren’t coming from breakthroughs in transistor density, but from better coordination between components that already exist. That’s a systems engineering problem, not a silicon problem.  

It’s also why the competitive advantage in the next phase of AI infrastructure won’t be determined by who has access to the best chips. Access to leading hardware is increasingly commoditized. The differentiator is whether your team can make the system around those chips perform.

It would be easy to interpret this as a cyclical change: CPUs fading out and then coming back. But that misses what’s actually happening.

The CPU comeback is driven by system complexity, not improvements in performance. The importance of coordination increases as AI applications evolve toward multi-step reasoning, real-time interaction, integration with external tools and data, and distributed execution across nodes.  

Yet coordination does not scale the same way compute does. You can add more accelerators to increase throughput, but you can’t add more accelerators to simplify decision-making.  

That layer has to exist somewhere, and right now, it lives on the CPU.  

The Gap Lies in Talent, Not Hardware 

Our recent posts trace a consistent pattern. We talked about AI EDA tools compressing chip design timelines and argued that the real constraint wasn’t tooling, but engineers who could operate at the intersection of AI and hardware design. We’ve also talked about connectivity, not compute, becoming the binding constraint in AI infrastructure—and the resulting talent gap of system-native engineers who understand data movement across entire clusters, not just within a chip. And we also talked about the rise of the inference systems architect, and why it’s so challenging to find them.  

This post describes the same pattern in the system orchestration layer.  

In each case, the tools and architectures have evolved. The talent pipelines haven’t kept pace. The gap isn’t getting smaller; it’s moving up the stack.  

The issue isn’t so much a shortage of AI talent broadly as it is a structural mismatch between what the hiring market has been optimized to find and what data center architectures now require.  

According to the World Economic Forum’s Future of Jobs Report 2025, employers in technology sectors consistently rank systems integration and cross-functional technical reasoning among the hardest competencies to find. 

This becomes most visible in terms of how teams today are built. The past several years of hiring have focused heavily on specialization. We’ve seen companies searching for experts in accelerator design, GPU kernel optimization, and model performance tuning.  

Those roles are still critical, but they don’t solve the emerging problem: how the system behaves as a whole.  

What’s increasingly needed are systems integration engineers and inference systems architects who can operate across layers with deep understanding of systems:  

• How CPUs and accelerators interact  
• How workloads are scheduled and coordinated  
• How memory and data movement affect performance  
• How to optimize systems end-to-end  

This isn’t a traditional profile. The cross-layer engineer sits somewhere between hardware, software, and distributed systems—and is in short supply.  

What Comes Next: SBT’s Take  

As you’ve read, CPUs fading out and then coming back isn’t just a cyclical change. That would miss the structural shift that’s happening: The real change is that the market is moving from a single-chip narrative toward a full-system narrative. 

While a simple idea defined the last phase of AI infrastructure—the team with the most compute power wins—the next phase is more complicated. The team with the talent that best understands the AI infrastructure system will win. 

The architecture has already shifted. Systems are now heterogeneous by design. CPUs are coordinating workloads at a scale and complexity that didn’t exist three years ago. And increasingly-complex agentic AI workloads are making this coordination problem harder, not easier.  

We see this happening with every shift: infrastructure moves, then the org charts follow. 

The companies that figure this out early will build teams that can operate across the full system stack, from silicon to scheduler, from memory hierarchy to workload coordination.   

Those that don’t will continue to hire for the last era’s bottleneck.  

The next era will be defined by who can coordinate compute and who has the people to make AI system coordination work. 

Are you still stuck in a recruiting mindset that worked for the “compute arms race”? Let’s talk about hiring talent who can help your team see the bigger picture and build for what’s next.