Empirically validate the prompt-length break-even threshold

Measure directly whether the fixed-cost model’s predicted approximately 17k-token threshold accurately separates prompts for which remote KV loading improves median latency from prompts for which recomputation is faster on the GLM-5.3 FP8 deployment using B300 GPUs.

Background

The cost model predicts that the fixed per-hit cost makes remote loading slower than recomputation for prompts shorter than approximately 17k tokens. However, the latency experiments concentrate on prompts between 14.7k and 17.9k tokens, and the fixed cost is fitted from a single point. The paper therefore leaves direct empirical validation of the predicted prompt-length threshold unresolved.

References

One prompt-length bucket. Latency results are for 14.7k--17.9k-token prompts. $T_0$ is fitted on one point, and the prompt-length threshold it implies is a prediction that we have not yet measured directly.

— When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill  (2610.21500 - Ibrahim, 15 Oct 2026) in Section 6, Limitations, bullet “One prompt-length bucket”