Head-Selective Training-Time Protection of Attention Sinks

Construct and evaluate a head-selective training-time intervention that protects attention-sink updates during vision-language training, for example by freezing layers below the first sink layer and projecting sink-head query and key updates away from the sink direction, and determine whether it preserves format-sensitive text capability.

Background

The experiments show that post-pretraining insertion of per-head QKQK-RMSNorm does not reproduce the protection associated with native normalization, and that post-vision-language weight merging does not recover the lost capability under the tested settings. These negative results leave training-time, head-level protection as the paper’s principal unresolved intervention direction. The proposed mechanism suggests freezing lower layers and constraining the query/key updates of sink heads, but the effectiveness of this strategy has not been established.

References

The post-VL merging methods we test do not recover the lost capability, while post-pretraining $QK-RMSNorm$ injection fails to reproduce the native protective regime, leaving head-selective training-time intervention as an open direction (Section~\ref{sec:controls}). The bound of Section~\ref{sec:method} suggests one concrete candidate: freeze the layers below the first sink layer and project the sink-head $W_q, W_k$ updates away from the sink direction, which should slow how fast VL training closes the $G_{\mathrm{base} - B_{\mathrm{gap}$ margin. A direct causal intervention would separately test sufficiency, for instance disabling per-head $QK-RMSNorm$ during VL training of Qwen3 to see whether IFEval collapses. We leave both to future work.

— Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis  (2609.00746 - Choi et al., 1 Sep 2026) in Limitations section; Section 5, Controls and Discussion

Two aspects of the diagnosis remain open. SAGE is sink-aware at the level of depth rather than of individual tokens: the PIS diagnostics identify where in the decoder steering is useful, and the additive bias then reallocates the attention budget through Eq.~eq:odds_ratio. This design needs no per-token sink classifier, but whether explicitly suppressing sink tokens would add to the depth-level effect is untested, as is whether the same normalized depth transfers across model families without re-running the diagnostic.

— SAGE: Sink-Aware Guided Emphasis for Visual Grounding in Vision-Language Decoders  (2610.11469 - Song et al., 8 Oct 2026) in Section*{Limitations}, paragraph beginning “Two aspects of the diagnosis remain open.”