Attention Basin in Machine Learning

• Attention basin is a concept defining regions of uneven influence in model attention, highlighting variations in salience due to positional and contextual factors.
• It encompasses diverse manifestations such as RoPE-induced troughs, U-shaped attention over structured blocks, and sink-dominated routing in both language and diffusion models.
• Mitigation strategies like Attention Buckets and AttnRank redistribute information or reorder inputs to counteract low-salience regions, enhancing overall model performance.

Attention basin is a recent term in machine-learning research for systematically uneven regions of effective model influence. In LLMs, it has been used to denote either a position-sensitive trough-and-peak structure induced by rotary position embedding, or a U-shaped preference over structured context blocks in which boundary items receive more attention than middle items (Chen et al., 2023, Yi et al., 7 Aug 2025). Related papers use the same basin language for sink-dominated routing in attention layers, for a stable anchor around the first token, and for attraction regions in diffusion denoising that pull trajectories toward memorized samples (Fu et al., 1 Feb 2026, Liu et al., 11 Apr 2026, Jain et al., 2024). Across these usages, the term refers to a region of disproportionately high or low effective influence; this suggests that model behavior depends not only on semantic relevance but also on the geometry by which probability mass, attention weight, or denoising trajectories are routed.

1. Terminology and relation to basin-of-attraction theory

In nonlinear dynamics, a basin of attraction is the set of initial conditions leading to a specific asymptotic state, and basin entropy is a finite-resolution measure of final-state unpredictability (Daza et al., 2022, Daza et al., 2022). A riddled basin is an extreme case in which arbitrarily small neighborhoods contain points from multiple basins, so final-state sensitivity remains severe even without chaos (Yan et al., 2 Mar 2026). This classical vocabulary provides the backdrop for the newer machine-learning uses of “attention basin,” although the latter are not all literal basins of attraction in the dynamical-systems sense.

Usage	Core object	Representative paper
RoPE attention basin	Peak–trough variation in attention-score upper bounds across relative positions	(Chen et al., 2023)
Structured-context attention basin	U-shaped attention over document or example blocks	(Yi et al., 7 Aug 2025)
Sink-dominated routing basin	Stable concentration of routing mass on sink tokens or branches	(Fu et al., 1 Feb 2026, Luo et al., 18 May 2026)
Diffusion attraction basin	Region of denoising state space leading to a memorized sample	(Jain et al., 2024)

The distinction matters. In some papers, basin denotes a literal subset of state space defined by convergence behavior. In others, it denotes a topographic pattern of salience or routing. A plausible implication is that the term has become a unifying metaphor for nonuniform computational accessibility.

2. RoPE-induced positional basins in long-context LLMs

A central formulation appears in “Fortify the Shortest Stave in Attention,” which analyzes rotary position embedding (RoPE) and shows that the pre-softmax attention score has an inherent oscillatory structure (Chen et al., 2023). For a query at position $m$ and key at position $n$,

$$(R_{\theta,m}q_m)^\top (R_{\theta,n}k_n)=q_m^\top R_{\theta,n-m}k_n, \qquad \theta_i = B^{-\frac{2i}{d}}.$$

The appendix derives the upper bound

$$UB=\sum_{j=0}^{d/2-1} 2\cos\!\big((m-n)\theta_j\big)\ge q_m\cdot k_n.$$

This upper bound is not monotonic with relative distance. Instead, it exhibits long-term decay plus waveform fluctuations. The paper calls those fluctuations the attention waveform, and characterizes alternating peaks and troughs over contextual positions.

Within this formulation, an attention basin is the low-salience region associated with a trough. If a crucial token or token pair falls near a peak, the model tends to attend to it better; if the same information falls near a trough, the model may ignore it. The paper verifies this on an in-context key-value retrieval task. In Table 1, with $K=50$ and base $30{,}000$, peak accuracy is $51.8\%$ while trough accuracy drops to $24.4\%$. The authors describe this as a position-sensitive basin in attention salience.

The proposed mitigation is Attention Buckets, an inference-time, training-free method. The input context $\mathcal C$ is duplicated into multiple parallel copies, each processed with a different RoPE base $B_j$ chosen so that the resulting waveforms are complementary. One searched set is

$n$0

The intuition is that a trough under one base can be covered by a peak under another. Outputs from the parallel branches are then confidence-weighted rather than naively averaged.

Empirically, this mechanism improves tasks requiring precise use of long context. On ToolBench, Attention Buckets raises ToolLlama-7B to a pass rate of $n$1 and a win rate of $n$2, reported as state-of-the-art and comparable to GPT-4. On ToolAlpaca, ToolAlpaca-7B improves from $n$3 to $n$4, and ToolAlpaca-13B from $n$5 to $n$6. On open-domain QA with retrieved documents, Llama-2-7B improves on NQ from $n$7 to $n$8 and on WebQA from $n$9 to $$(R_{\theta,m}q_m)^\top (R_{\theta,n}k_n)=q_m^\top R_{\theta,n-m}k_n, \qquad \theta_i = B^{-\frac{2i}{d}}.$$0. In this line of work, the attention basin is therefore a position-dependent blind spot arising from RoPE geometry rather than from content irrelevance.

3. Boundary-biased attention over structured context blocks

A second major usage appears in “Attention Basin: Why Contextual Position Matters in LLMs,” where the basin is defined at the level of structured items such as retrieved documents or few-shot demonstrations rather than individual token pairs (Yi et al., 7 Aug 2025). For an input

$$(R_{\theta,m}q_m)^\top (R_{\theta,n}k_n)=q_m^\top R_{\theta,n-m}k_n, \qquad \theta_i = B^{-\frac{2i}{d}}.$$1

the query tokens allocate more attention to items near the beginning and end of the sequence than to items in the middle. The resulting profile is explicitly described as U-shaped: the first and last documents receive the most attention, while middle documents receive less.

This paper positions the attention basin as the mechanism behind the lost-in-the-middle symptom. Its empirical evidence is block-structured rather than token-local. Mean query-to-document attention yields a consistent U-shaped curve across 10 mainstream LLMs, including LLaMA-2, LLaMA-3, Mistral, Qwen 2.5, DeepSeek-LLM, DeepSeek-R1-Distill-Llama, and InternLM3. The effect largely disappears when punctuation, capitalization, and explicit delimiters such as “Document [1]” are removed. That result ties the basin to the model’s perception of segment boundaries rather than to absolute token position alone.

The paper estimates a model’s positional preference through a calibration profile

$$(R_{\theta,m}q_m)^\top (R_{\theta,n}k_n)=q_m^\top R_{\theta,n-m}k_n, \qquad \theta_i = B^{-\frac{2i}{d}}.$$2

using the shallowest attention layer because early layers preserve intrinsic positional bias more clearly. The profile stabilizes with relatively few samples, often around $$(R_{\theta,m}q_m)^\top (R_{\theta,n}k_n)=q_m^\top R_{\theta,n-m}k_n, \qquad \theta_i = B^{-\frac{2i}{d}}.$$3, and in some models even a single sample exposes the pattern.

The resulting inference-time method, AttnRank, is a two-stage, model-agnostic, training-free reranking framework. It first extracts the positional attention profile, then reorders retrieved documents or demonstrations so that the most relevant item is placed in the highest-attention position, the second-most relevant in the second-highest position, and so on. On HotpotQA, AttnRank reaches $$(R_{\theta,m}q_m)^\top (R_{\theta,n}k_n)=q_m^\top R_{\theta,n-m}k_n, \qquad \theta_i = B^{-\frac{2i}{d}}.$$4, compared with $$(R_{\theta,m}q_m)^\top (R_{\theta,n}k_n)=q_m^\top R_{\theta,n-m}k_n, \qquad \theta_i = B^{-\frac{2i}{d}}.$$5 for random ordering, $$(R_{\theta,m}q_m)^\top (R_{\theta,n}k_n)=q_m^\top R_{\theta,n-m}k_n, \qquad \theta_i = B^{-\frac{2i}{d}}.$$6 for descending similarity, $$(R_{\theta,m}q_m)^\top (R_{\theta,n}k_n)=q_m^\top R_{\theta,n-m}k_n, \qquad \theta_i = B^{-\frac{2i}{d}}.$$7 for ascending similarity, and $$(R_{\theta,m}q_m)^\top (R_{\theta,n}k_n)=q_m^\top R_{\theta,n-m}k_n, \qquad \theta_i = B^{-\frac{2i}{d}}.$$8 for the LIM baseline. On 2WikiMultiHopQA, it reaches $$(R_{\theta,m}q_m)^\top (R_{\theta,n}k_n)=q_m^\top R_{\theta,n-m}k_n, \qquad \theta_i = B^{-\frac{2i}{d}}.$$9, compared with $$UB=\sum_{j=0}^{d/2-1} 2\cos\!\big((m-n)\theta_j\big)\ge q_m\cdot k_n.$$0, $$UB=\sum_{j=0}^{d/2-1} 2\cos\!\big((m-n)\theta_j\big)\ge q_m\cdot k_n.$$1, $$UB=\sum_{j=0}^{d/2-1} 2\cos\!\big((m-n)\theta_j\big)\ge q_m\cdot k_n.$$2, and $$UB=\sum_{j=0}^{d/2-1} 2\cos\!\big((m-n)\theta_j\big)\ge q_m\cdot k_n.$$3 respectively. On MultiWOZ 2.1 and 2.4, it reports an average JGA of $$UB=\sum_{j=0}^{d/2-1} 2\cos\!\big((m-n)\theta_j\big)\ge q_m\cdot k_n.$$4, compared with $$UB=\sum_{j=0}^{d/2-1} 2\cos\!\big((m-n)\theta_j\big)\ge q_m\cdot k_n.$$5 for random ordering. In this literature, the basin is not a single trough in token-level attention geometry but a reproducible edge-heavy allocation over structured blocks.

4. Sink-based basins in attention routing and context anchoring

A third strand of work uses basin language for sink-dominated routing. “Attention Sink Forges Native MoE in Attention Layers” argues that the attention sink is the mechanism by which standard attention already behaves like a sparse Mixture-of-Experts (Fu et al., 1 Feb 2026). For Vanilla Attention, if the first token is the sink, the head output can be rewritten as

$$UB=\sum_{j=0}^{d/2-1} 2\cos\!\big((m-n)\theta_j\big)\ge q_m\cdot k_n.$$6

The factor

$$UB=\sum_{j=0}^{d/2-1} 2\cos\!\big((m-n)\theta_j\big)\ge q_m\cdot k_n.$$7

acts as a gate on the head’s useful content. In this account, the sink is not merely wasted probability mass; it is the complement that routes how much each head contributes. Head collapse is therefore interpreted as an MoE-style expert-collapse phenomenon, quantified by a coefficient-of-variation imbalance metric over head importance scores. The proposed sink-aware auxiliary load-balancing loss improves performance across Vanilla Attention, Sink Attention, and Gated Attention. For example, at 2B parameters, average score rises from $$UB=\sum_{j=0}^{d/2-1} 2\cos\!\big((m-n)\theta_j\big)\ge q_m\cdot k_n.$$8 to $$UB=\sum_{j=0}^{d/2-1} 2\cos\!\big((m-n)\theta_j\big)\ge q_m\cdot k_n.$$9 for Vanilla, from $K=50$0 to $K=50$1 for Sink, and from $K=50$2 to $K=50$3 for Gated Attention.

“SinkTrack” exploits a related but distinct observation: the first token, typically <BOS>, remains unusually salient even as attention to other early tokens decays during generation (Liu et al., 11 Apr 2026). The paper treats hallucination and context forgetting as consequences of attention drift toward newly generated tokens and uses <BOS> as an information anchor. Its final mechanism applies cross-attentive injection to the sink token,

$K=50$4

while leaving the remaining tokens on the standard causal path. Injection every 5 layers is reported to work best. The method is training-free, plug-and-play, and adds negligible overhead. Reported gains include $K=50$5 on SQuAD2.0 with Llama3.1-8B-Instruct and $K=50$6 on M3CoT with Qwen2.5-VL-7B-Instruct.

A more architectural version appears in “Attention Sinks and Outliers in Attention Residuals,” which studies AttnResidual under dual normalization: token-level softmax and depth-level softmax (Luo et al., 18 May 2026). In that paper, an attention basin is a dynamical concentration basin in which mass repeatedly falls into sink tokens and sink-like residual routes because the architecture lacks an explicit null path. OASIS introduces a Softmax$K=50$7-based null state,

$K=50$8

and couples token-level null evidence to depth routing. Averaged over the evaluated settings, OASIS reports a $K=50$9 reduction in maximum infinity norm and a $30{,}000$0 reduction in average kurtosis, while lowering perplexity by $30{,}000$1 under W8A8 and improving GSM8K Pass@1 by $30{,}000$2 under W4A4. These sink-based papers therefore treat the basin either as a useful anchor, an implicit gate, or a pathological attractor that should be given an explicit null alternative.

5. Attraction basins in diffusion-model memorization

The most literal machine-learning use of basin terminology appears in “Classifier-Free Guidance inside the Attraction Basin May Cause Memorization,” which frames diffusion-model memorization as a dynamical-systems phenomenon (Jain et al., 2024). For a memorized training image $30{,}000$3, the paper defines an $30{,}000$4-attractor through convergence to a perceptual neighborhood

$30{,}000$5

and defines the attraction basin as

$30{,}000$6

Informally, this is the set of latent states and denoising times from which classifier-free guidance (CFG) pulls the trajectory toward the memorized target.

A central empirical indicator of the basin is the conditional-versus-unconditional noise difference,

$30{,}000$7

For memorized samples, this quantity is unusually large early in denoising and sharply drops when the trajectory exits the basin. The paper states explicitly that if $30{,}000$8 is inside the attraction basin, then applying CFG leads to the memorized output; if $30{,}000$9 is outside the basin, then applying CFG with any guidance weight $51.8\%$0 is unlikely to yield that memorized image.

The mitigation is to delay guidance until after the basin-exit transition point. In the zero-CFG phase,

$51.8\%$1

and the algorithm tracks

$51.8\%$2

The first local minimum,

$51.8\%$3

is used as the dynamic transition point. The paper also introduces opposite guidance, which flips the sign of the CFG term during the early denoising phase so that the trajectory exits the basin sooner. In one static-transition example, SDv2.1 fine-tuned on LAION-10k has a universal transition at about $51.8\%$4. The broader conclusion is that basin language can describe not only attention allocation but also memorization as a trajectory-level trapping phenomenon.

6. Conceptual synthesis, misconceptions, and significance

A common misconception is that attention basin denotes a single, standardized object. The literature instead uses the phrase for several non-equivalent constructs: a RoPE-induced trough in positional accessibility, a U-shaped block-level attention profile over structured context, a sink-dominated routing attractor, and a denoising basin around a memorized image (Chen et al., 2023, Yi et al., 7 Aug 2025, Fu et al., 1 Feb 2026, Jain et al., 2024). What these share is not a single formalism but the idea that effective influence is topographically uneven.

A second misconception is that basin structure is necessarily pathological. Some works try to compensate for low-salience regions by superimposing complementary attention landscapes, as in Attention Buckets, or by moving relevant evidence into naturally high-attention positions, as in AttnRank (Chen et al., 2023, Yi et al., 7 Aug 2025). Other works repurpose the sink itself as useful structure: SinkTrack turns <BOS> into a context anchor, and the native-MoE analysis interprets sink mass as an implicit gate rather than pure waste (Liu et al., 11 Apr 2026, Fu et al., 1 Feb 2026). By contrast, OASIS adds an explicit null channel to prevent sink-dominated routing, and the diffusion memorization work delays or reverses guidance to escape an attraction basin (Luo et al., 18 May 2026, Jain et al., 2024). This suggests that the practical issue is not whether a basin exists, but whether its geometry aligns with the intended computation.

The classical basin literature clarifies why the metaphor has been attractive. Basin entropy,

$51.8\%$5

measures the uncertainty of final-state prediction at finite resolution, while riddled and fractal basins describe progressively more severe boundary complexity (Daza et al., 2022, Daza et al., 2022). In coupled phase oscillators, the box-counting dimension of basin boundaries on a 2D slice approaches the full slice dimension $51.8\%$6 as $51.8\%$7, motivating the conjecture of riddling (Yan et al., 2 Mar 2026). A plausible implication is that recent machine-learning papers borrow the basin vocabulary because it captures the same underlying concern: high sensitivity of final behavior to where mass, evidence, or initialization lands within a nonuniform landscape.

Within current machine-learning usage, “attention basin” therefore names a family of geometric explanations for failures of context use, routing balance, and generative faithfulness. Its significance lies less in any single definition than in a recurring methodological claim: model performance can often be improved by measuring the geometry of influence, then either redistributing information across that geometry or exploiting its stable attractors.