Analyze the role of initial padding tokens as attention sinks

Investigate whether initial padding tokens in the generalized `smart_lastrec` cache policy function as attention sinks and thereby explain why protecting a variable range of tokens does not improve performance over protecting only an initial prefix.

Background

The generalized smart_lastrec policy protects a range of tokens beginning after initial padding, whereas the prefix variant protects an initial prefix. The generalized policy unexpectedly provides no improvement in the reported experiments. The authors explicitly state that this requires further analysis and suggest, without resolving the issue, that some initial padding tokens may serve as attention sinks.

References

Surprisingly, in our experiments, the general variant did not improve over the prefix variant,\footnote{This needs further analysis. Maybe some initial padding tokens are used as so-called "attention sinks"~\citep{Xiao:24}.}

— Learning how to Forget: Fine-tuning for Long-Context Sparse Attention  (2608.19920 - Seeger et al., 20 Aug 2026) in Appendix, Section “Policy smart_lastrec (slr),” footnote