Mechanism Behind Low-Rank Query Calibration (Q-Cal) Gains

Ascertain the underlying mechanism by which the low-rank query calibration (Q-Cal) module in Hierarchical Landmark Sparse (HiLS) Attention improves extrapolation and in-domain performance, and determine whether the added low-rank query residual indeed decouples token-level attention from the chunk-level mass surrogate.

Background

HiLS-Attention augments queries with a low-rank calibration (Q-Cal) when scoring chunk summaries. Ablations show that removing Q-Cal severely hurts extrapolation and retrieval performance, while oversizing its rank can also degrade extrapolation, indicating a nontrivial effect.

The authors conjecture that Q-Cal works by decoupling token-level attention from chunk-level mass estimation, but note that the underlying mechanism remains not fully understood and call for a more principled investigation.

References

We conjecture that AQ helps decouple token-level attention from the chunk-level mass surrogate, as they operate at different information level. Although this phenomenon is consistently observed in our experiments, the underlying mechanism is not yet fully understood, and we leave a more principled investigation to future work.

— Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling  (2607.02980 - Hu et al., 3 Jul 2026) in Section 5.3 (Ablation Study)