Effectiveness of HiLS-Attention at Larger Training Context Lengths Without Context Parallelism

Establish the effectiveness of Hierarchical Landmark Sparse (HiLS) Attention when training with larger context lengths, given that the current implementation lacks context parallelism support and its performance in such regimes remains to be fully validated.

Background

A key aim of HiLS-Attention is enabling ultra-long or even infinite-context training with bounded compute via native sparse retrieval. However, the current implementation does not support context parallelism, which is often necessary for training at very large context lengths.

The authors explicitly note that the method’s effectiveness at larger training lengths remains to be fully validated, identifying an open empirical and systems-level question about scalability and performance under such settings.