Determine whether compensatory retrieval preserves next-item accuracy

Determine whether the increase in targeted attention observed in unablated heads after ablating small subsets of Qwen retrieval heads helps preserve next-item accuracy during continued enumeration.

Background

The head-ablation experiments suggest that targeted retrieval heads contribute to next-item generation and final counting. In a qualitative inspection, however, ablating small subsets of Qwen heads increased targeted attention in heads that were not ablated in later layers, a pattern compatible with compensatory retrieval or backup-head behavior.

The functional consequence of this increased attention was not tested, leaving unresolved whether it actually mitigates the effect of ablation and preserves accurate retrieval of the next record.

References

In a qualitative inspection, ablating small subsets of Qwen heads increased targeted attention in unablated heads in later layers. This pattern is compatible with compensatory retrieval, resembling the backup-head behavior and self-repair reported in prior work~\citep{wang2022interpretability,mcgrath2023hydra}. Whether this increase helps preserve next-item accuracy remains untested.

— Targeted Retrieval, Compact Representations: How CoT Reasoning Improves Long-Context Counting  (2609.38958 - Shan et al., 30 Sep 2026) in Appendix, Section "Further details for the Thinking mode experiment," subsection "Causal tests of the retrieve--update--read pathway," paragraph "Retrieve: head ablation and head-count dependence"