Breaking The Connectivity Bottleneck: Beyond the Processor

Breaking The Connectivity Bottleneck: Beyond the Processor

Why AI‘s next scaling wall isn’t compute—it’s how efficiently data flows

The AI “arms race” has been framed around one metric: compute. But today a new challenge has emerged: the connectivity bottleneck.

Training frontier AI models now requires tens of thousands of GPUs operating in tightly coupled clusters, so the industry continues building faster processors and bigger, more power-hungry AI datacenter architectures.  

But these systems depend on constant, high-speed data exchange between compute and memory, and they can no longer keep up. Compute isn’t the problem anymore; the connectivity bottleneck is.  

In a conversation with SBT, Vivek Raghunathan, founder of Xscape Photonics, puts it bluntly: the biggest problem is how fast data can move through the system. 

“The hardware bottleneck has only grown. What that really means is you need much bigger memory so the system can fetch the data, come up with solutions, store them, and go get more data.”

AI Turned Compute into a Data Movement Problem 

For decades, system architecture has revolved around the simple assumption that the processor is the center of gravity. Memory feeds it, interconnect supports it, and everything else is secondary. 

That model worked until AI entered the picture. 

Today’s workloads are distributed across thousands of accelerators that are constantly moving data between memory tiers. These workloads depend on synchronization across systems. 

In fact, in the most advanced systems, moving data can consume more energy than compute, flipping a long-standing assumption in system design. 

When data can’t move fast enough, something counterintuitive happens: your most expensive hardware sits idle. Companies are investing billions in CapEx to acquire and deploy hardware to run AI workloads, so waiting isn’t just inefficient; it’s costly. 

Astera Labs’ COO Sanjay Gajendra notes that in modern AI infrastructure interconnect, “About 50% of the time the GPUs or accelerators are actually sitting idle waiting for data or memory. Our products enable those bottlenecks to be reduced so that the GPU can be utilized at the maximum level.”

To solve this problem, he explains, the industry has to change: “Connectivity fundamentally is going to be different than what it was done in the last 20, 25 years… you have to reinvent the nervous system that involves multiple processors talking to each other, each one of them capable of producing and consuming a lot of data.” 

The Hidden Constraint: Why Interconnect is Breaking First 

System architectures for AI don’t scale just by making compute chips bigger. They scale by adding more chips, nodes, and racks to form clusters. The approach is called heterogenous computing, and it’s been gaining momentum. 

But scaling compute is easier than scaling communication across different areas of the system. 

Traditional interconnects – like PCIe – were designed for general-purpose workloads, not for constant, high-throughput, low-latency synchronization across massive AI clusters. Bandwidth demand is scaling faster than compute, and that’s something traditional architectures were never designed for. 

Now, connectivity is hitting a wall, and bandwidth limitations, latency penalties, and memory access are the bricks. Even with advances like CXL memory pooling and composability, the central issue remains: you can scale compute linearly, but you can’t scale data movement as easily. 

When that imbalance is present, the entire system is network-bound and is only as fast as its slowest data pathway. NVIDIA’s Jensen Huang has explicitly discussed this issue, stating that the challenge is no longer compute but efficiently moving data through the system. 

The Shift from Processor-Centric to Connectivity-First Architecture 

This is where the connectivity bottleneck starts to change architecture itself. 

Systems used to be designed around the processor, but leading companies today are designing systems around data flow. You can already see this shift in how next-generation systems are being built. 

Raghunathan explains, “It’s all about making the entire data center behave as one large GPU and that requires being able to communicate across GPU at the same pace and the same bandwidth as a communication within the GPU in the same package.” 

Full-stack inference players are optimizing accelerator-to-accelerator communication, not just compute throughput. And photonics companies are rethinking the physical layer entirely, because electrical interconnect simply can’t keep up. 

The market is reacting quickly to this shift. The adoption of advanced communication technologies like optical switching and optical interconnects – which together can reduce data processing latency by as much as 40% compared to electronic systems – is driving a shift toward data centers designed for terabit-level transmission. 

Some analysts project photonics market growth of USD 372.3B at a CAGR of 6.6% between 2026 and 2030 with the optical interconnect AI segment growing at a CAGR of 14.6% between 2025 and 2030, reaching a value of USD 41.36 B by 2030.  

This is a clear signal that new approaches to next-generation connectivity are gaining traction in a market desperate for solutions. 

What Breaks: Why Traditional Engineering Models Don’t Work Anymore 

This shift isn’t just technical. It breaks how organizations are built. There are a few reasons for this: 

1. Siloed Teams Can’t Optimize System Performance

You can’t separate compute, memory, and networking when performance depends on how all three interact, yet most teams are still structured in a traditional way: designing individual blocks or tiles as if they’re disparate parts.  

2. Tools Don’t Solve System-Level Problems

You can simulate components and optimize blocks. But system-level data flow, latency across nodes, and real-world workload behavior don’t live inside a single tool. They require architectural judgment. 

3. Legacy Hiring Models Fall Behind

Most companies are still hiring CPU architects, RTL designers, and block-level specialists. But the hardest problems have leveled up. 

Similar to how AI supercharged EDA tools and compressed timelines, revealing gaps in talent strategy, AI’s impact on connectivity is exposing a new gap: there aren’t enough engineers with a deep understanding of systems. 

The New Talent: The Rise of the System-Native Engineer 

A new profile is emerging quietly but critically: the system-native engineer. These engineers understand: 

• Data movement in AI systems, not just computation 
• Systems, not just chips 
• Throughput across clusters, not just performance per core 
• PCIe and CXL 
• Memory hierarchies and disaggregation 
• Network fabrics and system-level connectivity bottlenecks 

But more importantly, they understand how these all fit together. These system-native engineers are becoming the highest-leverage hires in AI infrastructure talent.

What Companies are Getting Wrong 

Companies may see this shift happening, but they often misdiagnose it, treating connectivity as a component decision, a standards upgrade, or a late-stage optimization. It’s really not any of these things. 

But the AI connectivity bottleneck is now a first-principles architectural constraint. 

There are a few common mistakes we see being made in this area. For example, technical leaders might over-index on compute while underinvesting in interconnect. They might assume that adopting CXL will fix the problem. Or they might just hire narrowly instead of building system-level capability. 

Either way, the result is teams that can build powerful chips but not high-performing systems. 

Where this is Going 

This shift is still in its early innings, but directionally, what needs to change is clear. 

Over the short term, we’re going to see CXL and other interconnectivity standards accelerate, with increased participation and adoption across the entire industry. Memory pooling will become more common. We’ll start to see optical interconnect moving closer to the package, and co-packaged optics will continue to gain traction. 

Eventually, fully disaggregated, composable infrastructure will be the norm, and the network will become the dominant factor in performance. At that point, the distinction between compute and connectivity will all but disappear. The real distinction for talent will be who understands how to create and build designs that shatter the connectivity bottleneck.

SBT Take 

AI didn’t just increase demand for compute. It exposed system bottlenecks faster than most organizations have been able to adapt. 

The constraint is now data movement—and the people who understand how to design for it. 

The companies that win won’t have faster chips. They’ll have teams that understand how systems move data at scale. As ever, the real connectivity bottleneck isn’t compute OR bandwidth. It’s talent. 

Are you building around the processor or around the system? 

If your architecture has changed to meet the challenges the connectivity bottleneck presents, but your hiring hasn’t, you’re already behind. 

Let’s talk.