Head-Selective Training-Time Protection of Attention Sinks

Construct and evaluate a head-selective training-time intervention that protects attention-sink updates during vision-language training, for example by freezing layers below the first sink layer and projecting sink-head query and key updates away from the sink direction, and determine whether it preserves format-sensitive text capability.

Background

The experiments show that post-pretraining insertion of per-head QKQK-RMSNorm does not reproduce the protection associated with native normalization, and that post-vision-language weight merging does not recover the lost capability under the tested settings. These negative results leave training-time, head-level protection as the paper’s principal unresolved intervention direction. The proposed mechanism suggests freezing lower layers and constraining the query/key updates of sink heads, but the effectiveness of this strategy has not been established.

References

The post-VL merging methods we test do not recover the lost capability, while post-pretraining $QK-RMSNorm$ injection fails to reproduce the native protective regime, leaving head-selective training-time intervention as an open direction (Section~\ref{sec:controls}). The bound of Section~\ref{sec:method} suggests one concrete candidate: freeze the layers below the first sink layer and project the sink-head $W_q, W_k$ updates away from the sink direction, which should slow how fast VL training closes the $G_{\mathrm{base} - B_{\mathrm{gap}$ margin. A direct causal intervention would separately test sufficiency, for instance disabling per-head $QK-RMSNorm$ during VL training of Qwen3 to see whether IFEval collapses. We leave both to future work.

— Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis  (2609.00746 - Choi et al., 1 Sep 2026) in Limitations section; Section 5, Controls and Discussion