Origin of HiLS-Attention’s Length-Extrapolation Ability

Determine whether the strong length-extrapolation ability of Hierarchical Landmark Sparse (HiLS) Attention arises from improved in-domain retrieval capability during inference and training.

Background

The paper proposes HiLS-Attention, a hierarchical sparse attention mechanism that learns chunk selection end-to-end and shows strong extrapolation to contexts far beyond training length. In small-scale studies, HiLS-Attention outperforms full attention on variable-tracking tasks and achieves near-perfect in-domain retrieval, which the authors connect to the benefits of chunk-level compression.

The authors explicitly conjecture that the mechanism underlying HiLS-Attention’s extrapolation is its improved in-domain retrieval capability, but this connection has not been formally established, motivating verification of this causal link.