The three infrastructure layers every AI system needs - compute, network, storage

Welcome back to the OptimusEdge. A $70,000 GPU can sit at 5% utilization for a reason that has nothing to do with the GPU. Compute gets all the marketing budget. Network and storage are what actually decide whether you get to use what you bought.
___________________________________________________________
The Edge Upload: Today’s Insights

Storage re-enters the AI conversation: Google announced its Managed Lustre platform now delivers up to 10 TB/s of throughput to GPU and TPU training environments over RDMA a sign storage vendors are finally treating themselves as part of the critical path, not an afterthought.

The switch ASIC race continues: Broadcom's Tomahawk 6 chip now drives 64 ports of 1.6 Tb/s Ethernet on a single ASIC, built specifically to keep external cluster fabric from throttling next-generation NVLink speeds.

Rubin's bandwidth jump: NVIDIA's upcoming Rubin platform doubles GPU-to-GPU bandwidth again to 3.6 TB/s, roughly 2x Blackwell's 1.8 TB/s, continuing a trend where interconnect speed has to keep pace with compute or the extra compute goes to waste.

This is the layer that gets the headlines the GPU itself. But "compute" isn't just the chip; it's how fast that chip can feed itself.

An H100 has roughly 2.9 TB/s of local memory bandwidth (HBM), the speed at which the GPU can pull its own model weights out of memory. Blackwell's B300 pushes that to 288 GB of HBM3e per GPU, and NVIDIA's upcoming Rubin platform (shipping H2 2026) jumps again to HBM4 at 13 TB/s per GPU.

That number matters because a GPU that can't feed its own compute cores fast enough sits idle waiting on its own memory before a single byte ever leaves the chip.

Here's the part most non-engineers never learn: bandwidth drops by roughly an order of magnitude at every hop away from the GPU die. The hierarchy runs HBM → NVLink → PCIe → InfiniBand → Ethernet, each tier meaningfully slower than the one before it.

NVLink NVIDIA's proprietary intra-node interconnect hits 900 GB/s on H100/H200 and 1,800 GB/s on Blackwell's NVLink 5, but it only connects GPUs within a server or rack. A full GB200 NVL72 rack aggregates that up to 130 TB/s of system-wide bandwidth across 576 GPUs.

The moment data has to leave that rack, it drops onto InfiniBand or Ethernet today's high-end clusters run 800 Gb/s per GPU via Quantum-2 InfiniBand or 800G Ethernet, which sounds fast until you realize it's roughly 18x slower than the NVLink it's connected to.

This is the concrete reason "GPU networking" is its own engineering discipline: the whole job is making sure that hop doesn't become the bottleneck that strands everything upstream of it.

This is the layer that gets skipped in almost every infrastructure conversation, and it's often the one actually responsible for low utilization.

A single H100 can consume training data at 3–4 GB/s; scale that to a 512-GPU cluster and you need sustained storage throughput of 400–600 GB/s just to keep pace during data loading.

Fall short, and GPUs idle between batches waiting for data a pattern with a name: GPU starvation, which WEKA identifies as the primary reason enterprise GPU utilization can fall as low as 5% the exact number we covered from Cast AI's report in issue #1, from a completely different root cause.

Checkpointing makes this worse in bursts: large models can generate 350 GB of write traffic every few hours to preserve training state, flipping the I/O pattern from steady reads to a sudden massive write.

The fix gaining traction in 2026 is GPUDirect Storage (GDS), which routes data straight from NVMe to GPU memory instead of through the CPU cutting a 671B-parameter model's load time from roughly 18 minutes down to about 2.

That's today's briefing. If your infra team keeps arguing about GPU count and nobody's checked storage throughput, forward this to them. Past issues are in the archive. See you tomorrow.

NVIDIA DCGM Exporter: The same tool from an earlier issue engineer skill section, worth mentioning again here it's the standard way to see whether your compute layer is actually the bottleneck, or whether it's sitting idle waiting on the other two layers.

WEKA AI Storage Architecture Guide: A practical breakdown of GPU starvation, synchronous vs. asynchronous checkpointing, and how to size storage throughput against your actual GPU count the storage-side equivalent of a GPU spec sheet.

⚙️ Compute Layer: NVIDIA Data Center GPU Specs: A Complete Comparison Guide - HBM bandwidth and FLOPS across Hopper, Blackwell, and the Rubin roadmap

🔌 Network Layer: GPU Interconnects and Rack-Scale Topology - the full NVLink/NVSwitch/NVL72 bandwidth hierarchy explained

💾 Storage Layer: AI Workloads Storage Architecture: Training & Inference throughput math for training clusters at scale

📊 GPU Starvation Root Cause: AI Training: GPU Performance Depends on Storage the direct link between storage I/O and the 5% utilization problem

That’s it for today !☀

Edge AI is levelling up—are you? Until next time, stay curious, stay building, and don’t let your machines take over. 🤖😆

Your Edge AI Explorer,
Sharat Sami (Let’s connect on LinkedIn)