Attention Dilution Effect in Transformers

• Attention Dilution Effect is a phenomenon where positionally privileged tokens consume disproportionate attention, starving semantically relevant inputs.
• It appears across domains like long-context fine-tuning and autonomous driving, with studies highlighting how positional bias and attention sinks degrade performance.
• Researchers can counteract dilution by leveraging bilevel optimization and sparse, risk-aware attention strategies to improve model focus and safety-critical outcomes.

Attention dilution effect denotes a family of mechanisms in which a finite aggregation budget is consumed by positionally privileged, misleading, or low-value inputs, leaving insufficient influence for semantically relevant or safety-critical signals. In current arXiv usage, the term is not standardized. In long-context supervised fine-tuning, it is explicitly defined as the “starvation of content tokens in the attention distribution” caused by positional bias and attention sinks (Pei et al., 11 May 2026). In end-to-end autonomous driving, a closely related formulation is “global attention dilution,” where dense softmax attention over many agents suppresses truly dangerous ones (Ding et al., 7 Apr 2026). Adjacent literature studies allied phenomena under different names, including “semantic dilution” in multi-head self-attention for video prediction (Nguyen et al., 28 Jan 2025) and “The First Drop of Ink” effect in long-context reasoning, where a small fraction of hard distractors disproportionately captures attention (Gao et al., 11 May 2026).

1. Terminology and scope

The exact phrase attention dilution effect appears most directly in the long-context fine-tuning literature, where it names a training-time failure mode rather than a purely inference-time retrieval error (Pei et al., 11 May 2026). A second explicit variant, global attention dilution, appears in autonomous driving and refers to softmax attention spreading across many agents so that collision-critical agents are insufficiently emphasized (Ding et al., 7 Apr 2026). Other papers describe cognate mechanisms without using the same phrase.

Usage in the literature	Domain	Operational meaning
Attention dilution	Long-context SFT	Starvation of content tokens due to positional bias and attention sinks
Global attention dilution	Autonomous driving	Broad all-agent attention, with important conflict agents receiving too little weight
Semantic dilution	Next-frame prediction	Embedding partition across heads weakens holistic semantic content
“First Drop of Ink”	Long-context QA	A small fraction of hard distractors captures disproportionate attention

This distribution of terminology matters. The literature does not present a single invariant definition spanning all domains. Rather, it presents a recurring structural pattern: limited attention or fusion mass is allocated according to positional privilege, denominator competition, or representation splitting instead of task relevance. This suggests that “attention dilution effect” is best treated as a cross-domain family resemblance, anchored most strongly in long-context transformer training but extending to related architectures and decision systems.

2. Core mechanism: competition inside normalized aggregation

The basic transformer formulation used in the long-context papers is standard scaled dot-product attention: $$q_i = W_Q h_i,\qquad k_j = W_K h_j,\qquad v_j = W_V h_j,$$

$$z_{i,j} = \frac{q_i^\top k_j}{\sqrt{d_k}},\qquad \alpha_{i,j} = \frac{\exp(z_{i,j})}{\sum_{\ell=1}^{T}\exp(z_{i,\ell})},\qquad o_i = \sum_{j=1}^{T}\alpha_{i,j} v_j.$$

Under causal masking, the sum is restricted to $j \le i$ (Pei et al., 11 May 2026).

Within this normalization, dilution arises when large portions of probability mass are attracted to candidates that are easy to attend to for reasons other than semantic usefulness. In long-context SFT, the privileged tokens are early or boundary positions; in autonomous driving, they are numerous low-risk agents in dense scenes; in long-context QA, they are semantically related but answer-absent passages whose logits are close to the gold passage (Ding et al., 7 Apr 2026). The effect is thus not merely “attention becomes diffuse.” It is a specific failure of competitive allocation under softmax.

The strongest formalization of this competition appears in the hard-distractor analysis of long-context QA. For gold passage attention at hard-distractor proportion $p$, the paper derives

$$\alpha_{i,\mathcal{J}^*}(p) = \frac{1}{1 + (1-p)\cdot a + p\cdot b + c},$$

with $a := T_d \cdot e^{-\Delta_e}$, $b := T_d \cdot e^{-\Delta_h}$, and $c := T_o \cdot e^{-\Delta_o}$. Since $\Delta_h \ll \Delta_e$, hard distractors contribute more strongly to the softmax denominator than easy distractors, so $b > a$. The paper shows that $$z_{i,j} = \frac{q_i^\top k_j}{\sqrt{d_k}},\qquad \alpha_{i,j} = \frac{\exp(z_{i,j})}{\sum_{\ell=1}^{T}\exp(z_{i,\ell})},\qquad o_i = \sum_{j=1}^{T}\alpha_{i,j} v_j.$$0 is strictly decreasing and strictly convex in $$z_{i,j} = \frac{q_i^\top k_j}{\sqrt{d_k}},\qquad \alpha_{i,j} = \frac{\exp(z_{i,j})}{\sum_{\ell=1}^{T}\exp(z_{i,\ell})},\qquad o_i = \sum_{j=1}^{T}\alpha_{i,j} v_j.$$1, yielding a front-loaded degradation curve rather than a linear one (Gao et al., 11 May 2026).

A parallel but simpler argument appears in driving. Under standard softmax attention over $$z_{i,j} = \frac{q_i^\top k_j}{\sqrt{d_k}},\qquad \alpha_{i,j} = \frac{\exp(z_{i,j})}{\sum_{\ell=1}^{T}\exp(z_{i,\ell})},\qquad o_i = \sum_{j=1}^{T}\alpha_{i,j} v_j.$$2 surrounding agents, “each agent receives approximately $$z_{i,j} = \frac{q_i^\top k_j}{\sqrt{d_k}},\qquad \alpha_{i,j} = \frac{\exp(z_{i,j})}{\sum_{\ell=1}^{T}\exp(z_{i,\ell})},\qquad o_i = \sum_{j=1}^{T}\alpha_{i,j} v_j.$$3 attention weight when the number of surrounding agents becomes large,” so risk-bearing agents are numerically overwhelmed by harmless ones (Ding et al., 7 Apr 2026). Across domains, the common mechanism is denominator capture: the scarce mass of a normalized allocation is consumed by the wrong competitors.

3. Training-time attention dilution in long-context supervised fine-tuning

The most developed use of the term appears in “FocuSFT: Bilevel Optimization for Dilution-Aware Long-Context Fine-Tuning” (Pei et al., 11 May 2026). There, attention dilution is defined as a training-time bottleneck: positional biases and attention sinks cause the model to spend most of its attention on positionally privileged tokens rather than semantically relevant context tokens, weakening the gradient signal needed for robust long-context learning. The paper ties the effect to four mechanisms: positional privilege, attention sinks, causal asymmetry induced by masking, and optimization feedback that locks shallow position-driven attention patterns into the model.

The paper’s representative 4096-token agentic example operationalizes this definition. Standard SFT sends 30.1% of the response-query attention mass to the first five positions, while the entire useful context content receives only 13.5%. Under FocuSFT, context engagement rises to 41.4%. In the appendix sink diagnostic, standard SFT has average sink mass 0.301 and FocuSFT reduces it to 0.0006, a 529\times reduction. The paper further reports that the sink is pervasive across all 28 layers, not a localized artifact.

The proposed remedy is a bilevel optimization framework with fast-weight parameters $$z_{i,j} = \frac{q_i^\top k_j}{\sqrt{d_k}},\qquad \alpha_{i,j} = \frac{\exp(z_{i,j})}{\sum_{\ell=1}^{T}\exp(z_{i,\ell})},\qquad o_i = \sum_{j=1}^{T}\alpha_{i,j} v_j.$$4, implemented as LoRA adapters reinitialized at every training step: $$z_{i,j} = \frac{q_i^\top k_j}{\sqrt{d_k}},\qquad \alpha_{i,j} = \frac{\exp(z_{i,j})}{\sum_{\ell=1}^{T}\exp(z_{i,\ell})},\qquad o_i = \sum_{j=1}^{T}\alpha_{i,j} v_j.$$5 The inner loop adapts $$z_{i,j} = \frac{q_i^\top k_j}{\sqrt{d_k}},\qquad \alpha_{i,j} = \frac{\exp(z_{i,j})}{\sum_{\ell=1}^{T}\exp(z_{i,\ell})},\qquad o_i = \sum_{j=1}^{T}\alpha_{i,j} v_j.$$6 on the current training context to form a transient parametric memory, and the outer loop performs response-token SFT conditioned on this sharpened representation. Both loops apply bidirectional attention among context tokens while preserving causal masking for response tokens. The paper argues that this reduces the causal asymmetry that gives rise to attention sinks and maintains inner-outer consistency.

Empirically, the method improves BABILong accuracy by +14.2, +10.2, +10.2, and +9.6 percentage points at 4K, 8K, 16K, and 32K, respectively. On RULER, CWE rises from 72.9% to 81.1% at 16K. On GPQA Diamond with multi-turn agentic tool use, pass@1 rises from 15.6 to 19.4, which the abstract also describes as a 24% relative gain. The ablation study is mechanistically important: “SFT + Bidir.” hurts on BABILong at shorter lengths, “Causal Bilevel” provides most of the benefit, and full FocuSFT adds further gains, especially at 32K. The paper also notes a 1.71× training wall-time cost and leaves open why FFN LoRA adaptation induces the observed attention sharpening in later layers.

4. Global attention dilution in multi-agent planning

In “Not All Agents Matter: From Global Attention Dilution to Risk-Prioritized Game Planning,” the same structural problem is recast for autonomous driving (Ding et al., 7 Apr 2026). The paper argues that standard global attention over all surrounding agents allocates attention too uniformly in crowded scenes, so agents with geometric conflict do not stand out enough. This is framed as a risk-aware interaction problem rather than a generic perception problem: collision risk is sparse, but dense softmax attention treats all visible agents as comparable competitors.

GameAD addresses the effect through four components: Risk-Aware Topology Anchoring (RTA), Strategic Payload Adapter (SPA), Minimax Risk-Aware Sparse Attention (MRSA), and Risk Consistent Equilibrium Stabilization (RCES). The central mechanism is MRSA. It computes a worst-case risk matrix $$z_{i,j} = \frac{q_i^\top k_j}{\sqrt{d_k}},\qquad \alpha_{i,j} = \frac{\exp(z_{i,j})}{\sum_{\ell=1}^{T}\exp(z_{i,\ell})},\qquad o_i = \sum_{j=1}^{T}\alpha_{i,j} v_j.$$7 over ego planning mode $$z_{i,j} = \frac{q_i^\top k_j}{\sqrt{d_k}},\qquad \alpha_{i,j} = \frac{\exp(z_{i,j})}{\sum_{\ell=1}^{T}\exp(z_{i,\ell})},\qquad o_i = \sum_{j=1}^{T}\alpha_{i,j} v_j.$$8, agent $$z_{i,j} = \frac{q_i^\top k_j}{\sqrt{d_k}},\qquad \alpha_{i,j} = \frac{\exp(z_{i,j})}{\sum_{\ell=1}^{T}\exp(z_{i,\ell})},\qquad o_i = \sum_{j=1}^{T}\alpha_{i,j} v_j.$$9, agent motion modes, and time horizon, then retains only the top-$j \le i$0 highest-risk agents per planning mode. The selected risk values are injected into attention logits: $j \le i$1 This changes weighting relative to standard attention by combining learned semantic compatibility with explicit geometric collision risk.

The framework’s evaluation uses both open-loop and closed-loop metrics. On nuScenes, GameAD reports Avg. L2: 0.59 m, Avg. Collision Rate: 0.08%, and Avg. PRE: 3.94. On Bench2Drive, it reports Driving Score: 48.35, Success Rate: 19.43%, Efficiency: 178.96, and Comfort: 53.11. Relative to baselines, the paper highlights an 11% reduction in average collision rate relative to BridgeAD, an 11% PRE improvement over MomAD, a 16.3% success-rate improvement over SparseDrive, and 7.3% over MomAD.

The ablation results identify MRSA as the component most directly responsible for resolving dilution. Full GameAD yields Avg. PRE: 3.94, Avg. Collision Rate: 0.08, and Avg. TPC: 0.51; removing MRSA changes these to 4.02, 0.15, and 0.53. Qualitatively, the paper describes SparseDrive attention as dispersed across many agents with little variation across planning modes, whereas GameAD produces sparse, mode-differentiated attention focused on a small subset of safety-critical agents. In this literature, attention dilution is therefore not merely low entropy or high entropy in the abstract; it is the mismatch between uniform all-agent reasoning and the sparse structure of collision risk.

5. Related variants: hard-distractor competition and semantic fragmentation

A closely related long-context phenomenon is the “First Drop of Ink” effect (Gao et al., 11 May 2026). The setting fixes total context length while varying the proportion of hard distractors, defined as BM25-retrieved passages that are semantically related to the query but do not contain the answer. The paper reports that the initial increase from 0% to 10% hard distractors causes a disproportionate share of the total degradation. It quantifies this with the Drop Ratio: $j \le i$2 On nq_easy at 128K with Qwen2.5-7B-Instruct, the drop ratio is 0.58, meaning 58% of the total degradation from 0% to 100% hard distractors occurs in just the first 10%. The margin analysis gives $j \le i$3, implying $j \le i$4; the paper interprets this as each hard distractor token contributing about 340× more to the softmax denominator than each easy distractor token. This is an attention-dilution account in denominator form: a small amount of misleading relevance can consume most of the competitive mass.

A different but related formulation appears in “Overcoming Semantic Dilution in Transformer-Based Next Frame Prediction” (Nguyen et al., 28 Jan 2025). Here the dilution is not over tokens in context but over semantic content in multi-head representation splitting. The paper argues that standard MHSA divides an embedding $j \le i$5 into $j \le i$6 chunks,

$j \le i$7

so each head processes only a fraction of the original embedding. SCMHSA replaces this with full-embedding projections for every head,

$j \le i$8

and adds a Semantic Similarity loss to discourage head redundancy.

The empirical evidence is strongest on larger datasets. On UCSD, SCMHSA changes MSE 111.71 \to 86.71 and PSNR 27.65 \to 28.75; on UCF Sports, MSE 145.57 \to 79.63 and PSNR 26.50 \to 29.12; on Penn Action, MSE 171.03 \to 109.92 and PSNR 25.80 \to 27.72. The paper also reports 42.7M parameters with SCMHSA versus 31.4M without it, so increased capacity is an explicit confound. This variant should therefore be distinguished from token-level attention dilution: it concerns architectural fragmentation of embedding semantics across heads rather than starvation of relevant tokens in a softmax over long context.

6. Analogies, neighboring dilution models, and limits of transfer

Several papers outside mainstream transformer analysis clarify what does and does not count as attention dilution. “Group Testing in the High Dilution Regime” models independent per-defective dropout before pooled OR aggregation rather than cognitive or machine attention (Arpino et al., 2021). If a test contains $j \le i$9 defectives, it is positive with probability

$p$0

The paper explicitly presents this as only an analogy to an “attention dilution effect”: a precise stochastic model of pre-aggregation signal loss, not a theory of limited-capacity attention.

“Less is More: The Dilution Effect in Multi-Link Wireless Sensing” is closer in mechanism, because it studies broad fusion over many candidate inputs when only a sparse subset is informative (Rodrigues et al., 11 Feb 2026). In a 9-node mesh with 72 sensing links, a single selected link achieved AUC 0.541 versus 0.489 for the full 72-link mesh, with Cohen’s $p$1. Soft attention performed worse, with AUC $p$2, and the learned attention weights remained nearly uniform, with max/min $p$3 around the uniform baseline $p$4. This is not transformer self-attention, but it is a strong fusion analogue: sparse relevance cannot be recovered by a weighting mechanism when the per-sample saliency signal is weak.

By contrast, some uses of “dilution” are terminological neighbors only. In weak-lensing cluster finding, the dilution effect is the suppression of shear signals by foreground galaxies and cluster member galaxies that are unlensed by the target cluster (Hamana et al., 2020). In neutrino cosmology, the “dilution-resistant effect” concerns rest-mass energy of light mediators redshifting more slowly than radiation and can increase $p$5 by 0.118 for scalars and 0.242 for vectors (Li et al., 2023). These are not attention mechanisms. Their relevance is taxonomic: they show that “dilution” names several distinct phenomena, so cross-domain transfer of the term requires caution.

7. Open questions and unresolved issues

The recent literature establishes the empirical reality of attention-like dilution effects more clearly than it establishes a unified theory. In FocuSFT, the proposed causal chain

$p$6

is strongly supported by diagnostics and interventions, but the paper itself notes open questions about why FFN LoRA adaptation induces the observed attention sharpening, how the method scales beyond Qwen2.5-7B, and why train-time bidirectional context attention transfers effectively to standard autoregressive inference (Pei et al., 11 May 2026).

The “First Drop of Ink” work sharpens the mechanism but also narrows the mitigation story (Gao et al., 11 May 2026). It shows that filtering gains often come mainly from context-length reduction rather than selective distractor removal, and that substantial recovery requires reducing hard-distractor proportion to near zero. Lowering softmax temperature worsens accuracy rather than fixing the problem. This implies that simple inference-time sharpening is not a general remedy for denominator competition.

The semantic-dilution literature remains partly interpretive (Nguyen et al., 28 Jan 2025). The paper does not provide a formal proof that splitting embeddings across heads necessarily distorts semantics in a harmful way, and its ablations do not fully separate the effect of full-embedding attention from the effect of moving from 31.4M to 42.7M parameters. Its evidence is task-specific to next-frame embedding prediction with ViT [CLS] embeddings and does not establish that the same diagnosis governs language-model attention.

The autonomous-driving formulation is similarly persuasive but partly implicit (Ding et al., 7 Apr 2026). The paper motivates dilution through the approximate $p$7 allocation under standard softmax and demonstrates strong gains from risk-aware sparse attention, yet it does not derive a general theorem for when dense global attention must fail. Its hyperparameter study also shows that risk should bias attention strongly but not absolutely: $p$8 works best, while too large a value hurts performance.

Taken together, these works suggest that attention dilution is not one pathology but a class of normalized-allocation failures. The shared structure is sparse relevance under competitive aggregation. The unresolved question is whether a general theory can unify training-time token starvation, hard-distractor denominator capture, cross-head semantic fragmentation, and multi-agent risk prioritization without collapsing distinctions that the current literature treats as substantive.