Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode

The Physical AI Inference Gap: Why Faster GPUs Don't Always Mean Faster LLM Decoding

When running AI on robots, autonomous vehicles, or edge copilots, we usually assume faster memory translates directly to faster inference. In these "physical AI" workloads, a single agent or camera feed typically processes one token at a time in a batch-1 setting. There is no opportunity to hide latency through throughput-oriented batching (processing multiple requests simultaneously).

Current wisdom suggests this workload is strictly memory-bandwidth-bound. Each decode step must stream model weights and the active KV cache (the memory containing previous token keys and values) from High Bandwidth Memory (HBM). Consequently, the industry assumes moving up the hardware ladder—from an L4 to an H100—should yield proportional latency gains. This paper finds that assumption is incomplete. On high-end silicon, the bottleneck shifts from the memory bus to the CPU-side kernel launch overhead. This creates an efficiency gap that makes expensive hardware significantly less effective for single-stream tasks.

The Problem

The status quo relies on the "roofline model." This model posits that if arithmetic intensity (the ratio of compute to memory traffic) is low, performance is capped by peak HBM bandwidth. For batch-1 autoregressive decode, this is technically true but practically misleading. While the GPU waits on memory, it also spends a disproportionate amount of time waiting for the CPU to trigger the next task.

As the paper demonstrates, as peak HBM bandwidth rises, the fraction of that bandwidth actually utilized during decode drops .

On an L4 GPU, the system achieves roughly 81% of its theoretical memory floor. However, on an H100, that utilization collapses to just 27%. This happens because the time required to launch a sequence of small kernels from the CPU does not scale down with faster memory. On fast GPUs, this "launch tax" becomes a dominant part of the total step time. It effectively masks the benefits of superior silicon.

How It Works

The authors isolate this mechanism by analyzing the anatomy of a single decode step. A typical 7–8B GQA (Grouped-Query Attention) transformer layer consists of roughly ten distinct kernel launches .

These include projections, RMSNorm (a normalization layer), and the attention mechanism.

The mismatch arises from the relationship between launch overhead and compute time:

1. The Launch Tax: Every time the CPU issues a command to the GPU, there is a non-trivial overhead (roughly 30 $\mu$s per kernel).
2. The Scaling Divergence: On slower hardware like the L4, the GPU compute time per kernel is relatively long (~200 $\mu$s). This makes the 30 $\mu$s launch overhead a minor fraction of the total cycle.
3. The H100 Collapse: On high-performance hardware like the H100, the compute time per kernel shrinks (~10 $\mu$s). The launch overhead remains constant. This causes the launch gaps to dominate the execution timeline .

To prove this, the authors used CUDA Graphs. This technology captures a sequence of kernel launches once and replays them as a single unit. This bypasses repetitive CPU-side dispatch. By comparing "eager" execution (standard per-kernel launches) against "graphed" execution, they isolated the cost of the launch tax. On the H100, the intervention provided a 1.259$\times$ speedup .

On the L4, the improvement was negligible (1.028$\times$). This confirms the L4 is limited by memory bandwidth, not launch overhead.

Numbers

The most striking takeaway for a hardware planner is the inversion of the cost-per-token ladder. An H100 has roughly 11$\times$ the peak HBM bandwidth of an L4. Yet, it does not provide 11$\times$ the performance for batch-1 workloads.

When applying optimal software levers, the gap narrows significantly. The paper finds that an L4 running GPTQ+ExLlamaV2 (an integer-4 quantized runtime) achieves a step time of 17.36 ms. Meanwhile, an H100 using CUDA Graphs hits 11.78 ms . Given published cloud pricing ($3.50/hr for H100 vs. $0.30/hr for L4), the L4 actually delivers the same workload at roughly 6$\times$ less cost per million tokens .

Furthermore, the study shows that for the L4, the choice of quantization runtime is more impactful than the bit-width itself. Standard paths like AutoAWQ+Marlin only yielded a 1.38$\times$ speedup over bf16. Conversely, the Ada-optimized ExLlamaV2 kernels achieved a 3.59$\times$ speedup . This proves that memory savings only manifest if the runtime efficiently streams compressed weights through the specific hardware architecture.

What's Missing

While the measurement sweep is rigorous, certain areas remain unverified:

• Architectural Generalization: The study focuses exclusively on GQA (Grouped-Query Attention) models with a head dimension of 128. It is unknown if the launch-tax dominance persists in architectures using MQA (Multi-Query Attention) or MLA (Multi-head Latent Attention).
• Larger Model Scales: The research targets 7–8B class models. Larger models involve more weights per layer. This might increase arithmetic intensity enough to shift the bottleneck back toward the memory bus.
• Advanced Decoding Techniques: The benchmarks are restricted to simple autoregressive decoding. Techniques like speculative decoding (using a smaller model to predict tokens) could change the launch-to-compute ratio.

Should You Prototype This

Yes, specifically for edge and robotics deployments.

If you are building a real-time agentic system on a single stream, do not assume an H100 will automatically solve your latency requirements. The "Physical AI" gap is real. Instead, focus your engineering efforts on the software stack. Implement CUDA Graphs to reclaim the launch tax on high-end hardware. Prioritize Ada-optimized quantized kernels (like ExLlamaV2) to maximize the utility of cheaper, bandwidth-constrained silicon like the L4. The data suggests a well-tuned, cheaper GPU is often more cost-effective than an oversized, underutilized flagship.

Figures from the paper