Measure whether the cache-pressure threshold varies across H100 variants

Determine whether the working-set-to-KV-capacity ratio at which prefix-cache degradation begins differs between the 94\,GB H100 NVL and smaller-HBM Hopper variants such as the 80\,GB H100 PCIe and SXM.

Background

The study characterizes prefix reuse on a single 94\,GB NVIDIA H100 NVL and finds that eviction-related degradation is governed primarily by the ratio of the prefix working set to available KV-cache capacity. Capacity pressure is reached experimentally through both a naturally smaller KV pool on a 32B model and an artificially capped KV pool on a 7B model.

Because only the 94\,GB H100 NVL was measured, the reported pressure boundary may not transfer directly to Hopper accelerators with less HBM. Establishing whether the onset ratio itself changes across H100 variants would clarify the portability of the benchmark’s cache-capacity guidance.

References

All measurements use a single 94\,GB H100~NVL. The pressure regime is reachable here once the working-set-to-capacity ratio passes parity, but we have not measured whether the ratio at which degradation begins differs on smaller-HBM Hopper variants (e.g.\ H100~PCIe/SXM at 80\,GB).

— PrefixBench-H100: Characterizing Prefix Reuse and Time-to-First-Token in H100 LLM Serving  (2609.19657 - Shewale et al., 17 Sep 2026) in Section 7, Limitation (ii)