Retrieval-Transition Heads in Multilingual Transformers

• Retrieval-Transition Heads (RTH) are specialized attention heads in multilingual transformers that link target output tokens to source-language tokens, enabling cross-lingual transitions.
• They are distinct from monolingual retrieval heads, concentrating in mid-layers to bridge latent, language-agnostic reasoning with specific language output.
• Ablation studies reveal that masking RTHs significantly degrades multilingual chain-of-thought performance and output coherence, highlighting their critical role.

Searching arXiv for the most relevant papers on Retrieval-Transition Heads and closely related retrieval-head work. Searching for "Retrieval-Transition Heads multilingual retrieval heads (Patel et al., 25 Feb 2026)" and related retrieval-head papers. Retrieval-Transition Heads (RTH) are attention heads in multilingual transformers that attend from target-language output tokens back to source-language “needle” tokens that express the same content, thereby governing the transition from latent or language-agnostic reasoning to target-language output. The term is introduced explicitly in multilingual analysis, where RTHs are distinguished from monolingual retrieval heads and shown to be especially important for cross-lingual Chain-of-Thought reasoning (Patel et al., 25 Feb 2026). Closely related work that uses the term retrieval heads describes a sparse set of attention heads that perform copy-paste-style retrieval from long contexts and whose ablation induces retrieval failure or hallucination; this broader literature suggests that RTHs are a specialized, cross-lingual refinement of a more general retrieval-head family (Wu et al., 2024).

1. Origins and conceptual scope

The immediate antecedent of RTHs is the retrieval-head framework of Wu et al., which defines a small subset of attention heads as the mechanistic substrate of long-context factuality. In that framework, retrieval heads are universal, sparse, intrinsic, dynamically activated, and causal: all the explored models with long-context capability have a set of retrieval heads; only a small portion, less than $5\%$, of the attention heads are retrieval; the same set persists through context extension, alignment, and MoE upcycling; some are always active while others are context-sensitive; and completely pruning retrieval heads leads to failure in retrieving relevant information and results in hallucination, while pruning random non-retrieval heads does not affect the model’s retrieval ability (Wu et al., 2024).

RTHs narrow this picture to multilingual generation. In that setting, a retrieval head in the older sense is a monolingual copy-paste head that links answer tokens to same-language needle tokens, whereas an RTH links target-language output tokens to source-language tokens that encode the same content. The multilingual paper frames this as a bridge from a language-agnostic or latent reasoning space to specific target-language realizations, and reports that RTHs are largely disjoint from retrieval heads, concentrated in different layers, and more vital for multilingual CoT reasoning (Patel et al., 25 Feb 2026).

A further conceptual precursor comes from the study of stacked attention heads on synthetic retrieval chains. There, the retrieval problem requires following a chain $x_0 \to x_1 \to \dots \to x_D$, and the analysis proves that a transformer needs depth growing logarithmically with the input size. The learned solution uses an induction-like head first, followed by later heads that retrieve successive chain elements. This suggests that “transition” in RTH is not necessarily implemented by a single head; it can emerge as a layered circuit in which earlier heads set up the pointer structure and later heads execute successive retrieval steps (Musat, 2024).

2. Operational definitions and scoring rules

In the monolingual retrieval-head literature, the standard detector is a Needle-in-a-Haystack copy-paste metric. A head $h$ is scored by the fraction of answer tokens it effectively copies from the inserted needle span, formalized as

$$\text{retrieval\_score}_h = \frac{|g_h \cap k|}{|k|},$$

where $k$ is the set of answer tokens and $g_h$ is the set of tokens that the head copy-pastes. In DeCoRe, following Wu et al., a head is considered a retrieval head if $\text{retrieval\_score}_h \ge 0.1$, and the top $N$ ranked heads form the retrieval-head set used for masking and contrastive decoding (Gema et al., 2024).

The multilingual RTH paper introduces an explicitly cross-lingual score. Let $a_h^t$ be the attention distribution of head $h$ at decoding step $x_0 \to x_1 \to \dots \to x_D$0, let $x_0 \to x_1 \to \dots \to x_D$1 be the current target-language token, and let $x_0 \to x_1 \to \dots \to x_D$2 denote the aligned source-language tokens supplied by an external aligner. The Retrieval Transition Score is

$x_0 \to x_1 \to \dots \to x_D$3

This makes RTH identification explicitly dependent on cross-lingual alignment rather than exact same-language copying (Patel et al., 25 Feb 2026).

Subsequent work broadens the detector family beyond literal copy-paste. QRHEAD replaces the token-copy criterion with query-focused passage attention, defining heads whose query-conditioned attention mass concentrates on gold relevant documents in realistic retrieval tasks; those heads have strong downstream utility for long-context reasoning and re-ranking (Zhang et al., 11 Jun 2025). LOCOS goes further and scores heads by the projection of their OV-circuit outputs onto the answer-token unembedding direction, contrasting needle and off-needle source positions in a single forward pass. That detector is write-aware rather than attention-only, and is designed for non-literal retrieval in which the answer is synthesized from the meaning of a context span rather than literally copied from it (Gema et al., 1 Jul 2026). Taken together, these detectors imply that RTH-like behavior can be indexed either by where a head reads, by what it writes, or by both.

3. Layer organization, multilingual structure, and transition behavior

The multilingual RTH study reports that both retrieval heads and RTHs are sparse, with only $x_0 \to x_1 \to \dots \to x_D$4–$x_0 \to x_1 \to \dots \to x_D$5 of heads having $x_0 \to x_1 \to \dots \to x_D$6 for RTHs, and that the two sets occupy different parts of the network. In Qwen-2.5 7B, language-specific retrieval heads cluster in final layers, whereas RTHs concentrate in middle layers, roughly layers $x_0 \to x_1 \to \dots \to x_D$7–$x_0 \to x_1 \to \dots \to x_D$8. The overlap between the two sets is small in the most influential heads: for Qwen-2.5 7B, the intersection between top-$x_0 \to x_1 \to \dots \to x_D$9 RH and RTH heads is 2 for English, with similarly low overlap for German, Chinese, and Swahili (Patel et al., 25 Feb 2026).

This organization is consistent with a broader pipeline picture. In the “Atlas of In-Context Learning,” in-context heads split into task heads and retrieval heads, while parametric heads form a separate set. Task heads specialize in instruction or question comprehension and peak in middle layers; retrieval heads specialize in retrieving or copying answer tokens from context and peak in later layers; parametric heads encode entities’ relational knowledge and dominate different layer regions. A plausible implication is that RTHs occupy the interface between task-head-like mid-layer processing and later retrieval or realization heads, rather than exhausting the entire transition process by themselves (Kahardipraja et al., 21 May 2025).

Late-layer concentration also appears in non-literal retrieval analysis. LOCOS finds that high-scoring non-literal retrieval heads concentrate in late layers in Qwen3 and Gemma-3-27B, mostly in the upper quarter or third of layers, although Gemma-3-12B and OLMo-3.1-32B are more distributed. The paper interprets these heads as jointly performing retrieval and part of the answer-transition step through their OV circuits. This suggests that the transition from retrieved content to answer logits may itself be stratified: some middle-layer heads govern cross-lingual or task-level transition, while later heads write answer-aligned directions directly into the residual stream (Gema et al., 1 Jul 2026).

Language sharing sharpens the multilingual picture. Retrieval heads are partly shared across languages and partly language-exclusive, with family-specific patterns: Qwen-2.5 7B has a high fraction of shared retrieval heads across English, German, Chinese, and Swahili, whereas Phi-3.5 3B has many English-specific heads. RTHs also show substantial overlap across target languages, supporting the claim that they are a shared bridge from latent reasoning to multiple output languages rather than a set of isolated translation gadgets (Patel et al., 25 Feb 2026).

4. Causal evidence from masking, ablation, and RoPE interventions

Causal intervention studies establish that retrieval heads and RTHs are not merely correlates. In the original retrieval-head work, masking the top retrieval heads causes Needle-in-a-Haystack performance to drop sharply, while masking random non-retrieval heads has far smaller effect; once roughly 50 retrieval heads are masked, needle retrieval falls below $h$0 for all tested models, and the resulting errors include incomplete retrieval, hallucination, and wrong extraction (Wu et al., 2024).

The multilingual RTH paper shows that RTH ablations are more destructive than RH ablations on multilingual benchmarks. For Qwen-2.5 7B, masking 25 RTHs produces average drops of about $h$1 on MMLU-ProX and about $h$2 on MGSM, versus about $h$3 and about $h$4 when masking 25 RHs. For Llama-3.1 8B, the strongest result is on MGSM, where masking 25 RTHs yields an average drop of about $h$5, compared with about $h$6 for RH masking. The paper further distinguishes failure types: RH masking mainly causes failure to retrieve, whereas RTH masking much more often causes loss of coherence, with repetitive, ungrammatical, or structurally broken outputs (Patel et al., 25 Feb 2026).

Write-aware ablations strengthen the case for a broader retrieval-transition family. LOCOS shows that on Qwen3-8B, mean-ablating the top 50 LOCOS heads collapses NoLiMa ROUGE-L from $h$7 to $h$8, while the strongest attention-based baseline still retains $h$9. The same ablation drops MuSiQue from $$\text{retrieval\_score}_h = \frac{|g_h \cap k|}{|k|},$$0 to $$\text{retrieval\_score}_h = \frac{|g_h \cap k|}{|k|},$$1 and BABI-Long from $$\text{retrieval\_score}_h = \frac{|g_h \cap k|}{|k|},$$2 to $$\text{retrieval\_score}_h = \frac{|g_h \cap k|}{|k|},$$3, while a random-heads control stays within $$\text{retrieval\_score}_h = \frac{|g_h \cap k|}{|k|},$$4 of baseline. Parametric recall and arithmetic reasoning remain at baseline under the same ablation, indicating retrieval-specific rather than generic importance (Gema et al., 1 Jul 2026).

RoPE analysis adds a structural intervention at the sub-head level. Across open-weight 7–8B models, masking the detected retrieval heads in OLMo-2 collapses recall from $$\text{retrieval\_score}_h = \frac{|g_h \cap k|}{|k|},$$5 to $$\text{retrieval\_score}_h = \frac{|g_h \cap k|}{|k|},$$6, while masking matched random heads has no effect; Qwen shows the same qualitative result. More finely, zeroing the lowest-frequency RoPE dimensions of retrieval heads degrades recall dose-dependently, from $$\text{retrieval\_score}_h = \frac{|g_h \cap k|}{|k|},$$7 to $$\text{retrieval\_score}_h = \frac{|g_h \cap k|}{|k|},$$8 when 32 of 128 dimensions are zeroed, versus $$\text{retrieval\_score}_h = \frac{|g_h \cap k|}{|k|},$$9 for random dimensions. The paper concludes that the causal variable is RoPE frequency, not norm-utility, and that low-frequency RoPE dimensions are the load-bearing substrate for long-range retrieval in these heads (Bayram, 19 Jun 2026).

5. Training and decoding interventions built around retrieval-head mechanisms

Once retrieval heads are treated as manipulable mechanisms, several methods use them as control points for faithfulness, reasoning, and long-context performance. DeCoRe constructs a retrieval-impaired model by hard-zeroing the outputs of the top retrieval heads and then contrastively decodes between the base and masked distributions. On Llama3-8B, DeCoRe entropy raises XSum factKB from $k$0 to $k$1 and MemoTrap macro/micro from $k$2 to $k$3; on Llama3-70B it improves NQ-Swap EM from $k$4 to $k$5. The method is training-free and operationalizes a transition between retrieval-enabled and retrieval-impaired regimes at decoding time (Gema et al., 2024).

RHIO uses retrieval-head masking to generate realistic unfaithful outputs for long-form question answering, then trains the model with faithful and unfaithful control tokens and applies contrastive decoding between those modes. On GroundBench, RHIO reaches an average FaithScore of $k$6 for Llama-2-7B and $k$7 for Llama-2-13B; GPT-4o is reported at $k$8. The paper argues that masking retrieval heads produces fabricated, incomplete, and inconsistent hallucinations with proportions similar to intrinsic model errors, so the induced negatives are mechanistically matched to the model’s own failure modes (Huang et al., 23 Jan 2025).

RetMask turns the same intuition into preference optimization. It generates chosen outputs from the full model and rejected outputs from a retrieval-head-ablated variant, then performs DPO. On Llama-3.1-8B-Instruct, RetMask improves HELMET at $k$9K from $g_h$0 to $g_h$1, with the largest task-wise gains on generation with citation and passage re-ranking: Cite rises from $g_h$2 to $g_h$3, and Re-rank from $g_h$4 to $g_h$5. The paper further reports that the masked retrieval heads themselves show the largest post-training retrieval-score gains, supporting a mechanistic reading of the improvement (Ma et al., 16 Jan 2026).

MuDAF directly optimizes selected retrieval heads by a contrastive loss over question queries and passage-level keys, combined with the causal language-modeling objective. On the six reported LongBench and ZeroSCROLLS tasks, Llama-3.1-8B-MuDAF reaches an average of $g_h$6, compared with $g_h$7 for Vanilla-SFT and $g_h$8 for the off-the-shelf base model. The largest gains occur on multi-document, multi-hop tasks, consistent with the view that a small subset of heads controls access to relevant context under severe distractor load (Liu et al., 19 Feb 2025).

6. Systems implications, efficiency uses, and unresolved questions

A major systems consequence of retrieval-head sparsity is that long-context inference need not treat all heads equally. DuoAttention partitions heads into retrieval heads, which keep full attention and full KV cache, and streaming heads, which use sink-plus-window attention. On MHA models it reports memory reduction up to $g_h$9, latency reduction up to $\text{retrieval\_score}_h \ge 0.1$0, and pre-fill speedup up to $\text{retrieval\_score}_h \ge 0.1$1; on GQA models the corresponding maxima are $\text{retrieval\_score}_h \ge 0.1$2, $\text{retrieval\_score}_h \ge 0.1$3, and $\text{retrieval\_score}_h \ge 0.1$4. The method uses synthetic passkey retrieval and a gate $\text{retrieval\_score}_h \ge 0.1$5 to identify which heads require full attention (Xiao et al., 2024).

RazorAttention reaches the same conclusion through a different detector and cache policy. It preserves full KV only for retrieval heads and discards remote tokens in non-retrieval heads, adding a compensation token for the dropped segment. With the default setting of top $\text{retrieval\_score}_h \ge 0.1$6 induction heads plus top $\text{retrieval\_score}_h \ge 0.1$7 echo heads as retrieval heads, it reports KV-cache reduction by over $\text{retrieval\_score}_h \ge 0.1$8 without noticeable impacts on performance, and emphasizes compatibility with FlashAttention (Tang et al., 2024). ZigzagAttention then imposes an exclusive layer structure in which each layer is entirely retrieval or entirely streaming, avoiding the latency of mixed-head kernels; at $\text{retrieval\_score}_h \ge 0.1$9 KV budget it reports up to $N$0 acceleration in $N$1k context length with only modest LongBench degradation (Liu et al., 17 Aug 2025).

Hybrid architectures use retrieval-critical heads even more aggressively. Retrieval-aware distillation for Transformer–SSM hybrids preserves only the heads identified as essential on a synthetic retrieval task and distills the rest into recurrent heads. Preserving just $N$2 of attention heads recovers over $N$3 of teacher performance on retrieval-heavy tasks, and the resulting hybrids are $N$4–$N$5 more memory-efficient than comparable hybrids. This frames retrieval heads, and by extension RTH-like structures, as sparse global-access channels that can coexist with much cheaper local or recurrent backbones (Bick et al., 11 Feb 2026).

Several unresolved questions remain. Synthetic context-extension experiments show that retrieval heads learned on synthetic data have high overlap with retrieval heads learned on real data, and that there is a strong correlation between the recall of heads learned and downstream performance; however, attention knockout and activation patching show that retrieval heads are necessary but not totally sufficient (Zhao et al., 2024). Detector choice is also unsettled: query-focused attention scoring identifies heads with strong downstream retrieval utility on real tasks (Zhang et al., 11 Jun 2025), while LOCOS shows that literal-copy detectors miss non-literal retrieval heads by construction (Gema et al., 1 Jul 2026). The multilingual RTH paper adds further caveats: its RTS relies on a pivot-language approximation of concept space, depends on an external aligner, and focuses only on attention heads rather than the interaction of attention, MLPs, and residual-stream features (Patel et al., 25 Feb 2026). A plausible implication is that future RTH research will need joint head-and-neuron analyses, dynamic rather than static head roles, and detectors that integrate attention placement with OV-circuit write effects.