Batch-and-context-aware selection of sparse speculation

Determine how to track both active batch size and context length to decide when sparse speculation is useful within a long-context reinforcement-learning rollout.

Background

The paper’s profiling shows that the opportunity for sparse-attention acceleration depends jointly on aggregate live KV volume, active batch size, and per-request context length. Although aggregate KV volume is a useful first-order indicator of the attention bottleneck, a shrinking active batch can reduce the benefit of sparse speculation even when responses remain long. The paper therefore leaves unresolved how a runtime should use both batch and context information to decide when to deploy sparse speculation.

References

These profiles motivate tracking both batch and context when deciding where sparse speculation is useful within an RL rollout, which is left for future work.

— S$^3$R: Fast and Faithful RL Rollouts via Self-Sparse Speculative Decoding  (2610.13202 - He et al., 9 Oct 2026) in Appendix F.2.1, “Larger live KV volumes make attention the bottleneck”