Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode

Abstract: Physical AI systems, including robots, autonomous vehicles, embodied agents and edge copilots, often run a different inference workload from cloud LLM serving: single-stream, batch-1 autoregressive decode, where one robot, camera feed or user session waits on the next token. This workload is usually described as memory-bandwidth-bound. Each decode step streams model weights and the active KV cache, so latency should scale with peak HBM bandwidth. We show that this account is true but incomplete. We measure batch-1 decode for three 7 to 8B-class GQA transformers across four NVIDIA GPUs: H100 SXM5, A100-80GB SXM4, L40S and L4. We evaluate context lengths from 2048 to 16384, producing 44 valid cells under a controlled bf16 SDPA setup. The achieved fraction of peak HBM bandwidth falls as peak bandwidth rises. On the headline Qwen-2.5-7B ctx=2048 cell, an L4 reaches roughly 81 percent of its analytic memory floor, while an H100 reaches only 27 percent. Physical-AI decode is memory-dominated, but faster memory does not translate into proportional latency gains. We test the missing term with a CUDA Graphs A/B experiment. On H100 at ctx=2048, CUDA Graphs improves decode latency by 1.259x across N=10 fresh sessions, with a 95 percent bootstrap confidence interval of 1.253 to 1.267. On L4, the same intervention gives only 1.028x. This isolates a launch-side overhead that becomes visible on fast GPUs but remains mostly hidden on slower, bandwidth-bound GPUs. The deployment implication is that memory savings matter only when the runtime realises them. On L4, bf16 decode sits close to the memory floor, but common quantised paths do not recover the expected 4x weight-traffic reduction: bnb-nf4 reaches 59.36 ms/step and AutoAWQ+Marlin reaches 45.24 ms/step from a 62.32 ms bf16 baseline. GPTQ+ExLlamaV2, with Ada-tuned int4 kernels, reaches 17.36 ms/step.

• The paper demonstrates that batch-1 decode latency is driven more by kernel launch overhead than by raw HBM bandwidth, as evidenced by R_floor measurements.
• The paper leverages CUDA Graphs and per-layer profiling to reveal that host-side kernel launch costs account for over 20% of wall time on H100 compared to L4.
• The paper shows that optimized quantization kernels like ExLlamaV2 can invert the deployment cost ladder by achieving similar latency at significantly reduced operating costs.

Memory-Bound but Not Bandwidth-Limited: Detailed Analysis of the LLM Inference Gap

Introduction and Problem Setting

This paper presents a rigorous experimental analysis of single-stream, batch-1, autoregressive decode for 7–8B-parameter class Grouped-Query Attention (GQA) transformers executed on diverse NVIDIA architectures: H100 SXM5, A100-80GB SXM4, L40S, and L4. Unlike throughput-oriented cloud LLM serving, such workloads, typified by robotics policy heads and embedded or edge copilot inference, are latency-sensitive with strictly sequential token production. The standard assumption in the literature is a bandwidth-bound regime whereby decode step time is set by “roofline” physical memory bandwidth, motivating extensive research focus on memory traffic reduction via quantization and KV cache optimizations.

The central claim tested is that this bandwidth-bound model is incomplete. While HBM bandwidth constrains the minimum achievable step time, in practice, a substantial “physical AI inference gap” emerges, especially on high-end accelerators. The observed throughput shortfall is attributed not solely to raw bandwidth, but to kernel launch and host overheads which do not scale with HBM BW. The authors systematically quantify, validate, and dissect this gap, with deployment and system implications for both kernel implementation and hardware procurement.

Empirical Protocol and Measurement Matrix

The experimental design covers 44 valid (architecture, GPU, context) cells, encompassing three contemporary GQA LLMs—Qwen-2.5-7B, Mistral-7B-v0.3, and Llama-3.1-8B—with context lengths 2k, 4k, 8k, and 16k. Measurements are performed with controlled software/driver stacks (PyTorch sdpa, bf16 weights, Modal cloud instances), strict warmup, and medians computed over 30 decode steps per cell. The achieved time per decode tokens ($t_{\textrm{obs}}$) is compared directly to the “analytic floor” $t_{\textrm{floor}} = (W+K)/B_\textrm{peak}$, where $W$ is static weight size, $K$ is context-dependent KV bytes, and $B_\textrm{peak}$ is HBM bandwidth.

A key metric is $R_\textrm{floor} = t_\textrm{floor}/t_\textrm{obs}$, i.e., the fraction of analytic peak bandwidth realized in practice for the full decoding step.

The pattern across devices is unequivocal: L4 (300 GB/s) achieves $R_\textrm{floor} \approx 0.8$, whereas H100 (3.35 TB/s) realizes only $R_\textrm{floor} \approx 0.25$–$0.30$ at short context, indicating sharply sublinear scaling of step time with HBM bandwidth.

Figure 1: The inverted deployment ladder for single-stream 7–8B decode; step time for L4 and H100 versus expected HBM fractions.

Mechanistic Disentanglement: CUDA Graphs and Launch-Bound Regimes

To determine whether per-kernel launch overhead—not HBM—becomes the limiting factor on faster devices, the study applies CUDA Graphs as an A/B test. CUDA Graphs eliminate Python/C++/host launches by capturing and replaying a kernel DAG. On H100 at ctx=2048, CUDA Graphs yields a statistically robust 1.259× speedup ($N=10$, 95% CI $t_{\textrm{floor}} = (W+K)/B_\textrm{peak}$0). By contrast, the identical intervention on L4 gives only 1.028×, confirming near-ideal bandwidth-coupling on L4 but substantial launch tax on H100.

Figure 3: CUDA Graphs A/B speedup across context and batch, highlighting diminishing effect on higher batch sizes.

This launch-bound behavior is further corroborated by per-kernel profiling: on H100, host-side launch accounts for $t_{\textrm{floor}} = (W+K)/B_\textrm{peak}$1 of wall time, whereas compute fully dominates on L4. The batch dependency is explicit; moving to batch=4 reduces the launch fraction on H100, shrinking the benefit from CUDA Graphs.

Figure 5: Eager versus graphed step times on H100 showing noise floor and tight speedup clustering.

Kernel and Backend Analysis: The Limits of Backend “Upgrades”

The paper systematically benchmarks a range of attention kernel backends (PyTorch default SDPA, FlashAttention-2, FlashInfer, FA-3, etc.) on H100 at ctx=2048. Contrary to prevailing practice, default SDPA is found to outperform all explicit alternatives, including those optimized for prefill throughput or multi-stream inference. Explicit cuDNN attention is not usable at this single-token GQA shape.

Figure 7: Per-layer attention latency on H100 for multiple backends. Default SDPA outperforms FlashAttention and FlashInfer at batch-1 decode with GQA configuration.

Although kernel choice is non-binding for H100 in this regime, on L4 the situation inverts: the quantization kernel implementation dominates. Switching from bf16 to GPTQ+ExLlamaV2 reduces L4’s step time from 62.32 ms to 17.36 ms (3.6×), whereas prevailing nf4 and AutoAWQ+Marlin implementations underdeliver due to dequant overhead or suboptimal GEMMs. ExLlamaV2, with Ada SM89-tailored int4 GEMM, recovers most of the expected bandwidth saving.

Figure 2: L4 quantization step times for Qwen-2.5-7B at ctx=2048. Only properly tuned kernels (ExLlamaV2) achieve near floor performance.

Deployment Cost and Ladder Inversion

A practical outcome is the inversion of the deployment cost ladder: an L4 with ExLlamaV2 int4 kernels provides similar per-token latency to an H100 (17.36 ms vs. 11.78 ms) at nearly 12× lower per-hour cost. For single-stream, batch-1 decode—the dominant form for physical AI, robotics policy heads, and edge copilot stacks—purchasing additional HBM bandwidth provides limited ROI. The cost-per-token-served is determined by the software stack and quantization implementation, not solely by silicon bandwidth.

Figure 4: Cost-per-million-tokens-served at batch-1 decode across Modal SKUs and kernels. Ladder direction inversion relative to raw bandwidth.

Theoretical and Practical Implications

Theoretical: The work underlines the inadequacy of the conventional roofline/bandwidth-bound inference model for batch-1 autoregressive decode on high-BW modern accelerators. This shifts the focus from physical hardware scaling to overhead amortization and kernel graphing techniques in low-batch, latency-bound deployments. Architectural and systems research must now address CPU/GPU launch costs, dispatcher and allocator overhead, and runtime-fused operations.

Practical: For teams deploying physical AI at scale, kernel choice and runtime tuning dominate over hardware upgrades once HBM exceeds moderate levels. Pipelined quantization kernels, dispatch graph replay, and single-call operator fusion are critical to avoiding “false hardware gaps.” For hardware vendors, the distinction between bandwidth and practical deployable throughput is salient for accelerator roadmap planning, especially as new memory technologies (HBM4, LPDDR6) shift the balance.

Limitations

The scope is explicitly restricted to 7–8B GQA transformers at bf16/int4, batch-1 decode, and only NVIDIA H100/A100/L40S/L4 on Modal cloud instances. Broader generalization for higher-parameter models, MHA/MQA, alternative attention mechanisms, higher batches, non-NVIDIA architectures, or speculative decoding is not claimed.

Conclusion

The study demonstrates that in batch-1 autoregressive decode for 7–8B-class GQA transformers, fast accelerators such as H100 are no longer HBM-bandwidth-limited, but rather constrained by non-scalable kernel launch costs. Substantial per-token latency reductions are unlocked not by hardware upgrades, but by launch elimination (CUDA Graphs) and backend kernel optimization. The deployment cost ladder is thus inverted for these regimes, with practical implications for robotics, edge AI, and copilot deployments. Future AI systems research should prioritize the complete runtime path—launch, kernel fusion, quantization efficiency, and host-GPU co-design—as much as raw accelerator bandwidth.