Explain the batch-position restriction in original H2O

Establish the rationale and consequences of selecting the same H2O cache slots for every batch position rather than allowing cache-slot decisions to depend on the batch position.

Background

The original H2O implementation aggregates H2O scores across batch positions and consequently selects identical cache slots for each batch position. The paper removes this restriction in its own implementation, but states that the motivation for the original design remains unresolved.

References

First, their code selects the same cache slots for each batch position $b$, using the score $\phit_{\text{h2o-orig}(h, j) = \sum_b \phit_{\text{h2o}(b, h, j)$. The rationale for this restriction is unclear, we implement H2O without it as well.

— Learning how to Forget: Fine-tuning for Long-Context Sparse Attention  (2608.19920 - Seeger et al., 20 Aug 2026) in Section 3.1, “Variants of Heavy-Hitter Oracle”