DoPE: Denoising Rotary Position Embedding (2511.09146v1)

Abstract: Rotary Position Embedding (RoPE) in Transformer models has inherent limits that weaken length extrapolation. We reinterpret the attention map with positional encoding as a noisy feature map, and propose Denoising Positional Encoding (DoPE), a training-free method based on truncated matrix entropy to detect outlier frequency bands in the feature map. Leveraging the noise characteristics of the feature map, we further reparameterize it with a parameter-free Gaussian distribution to achieve robust extrapolation. Our method theoretically reveals the underlying cause of the attention sink phenomenon and its connection to truncated matrix entropy. Experiments on needle-in-a-haystack and many-shot in-context learning tasks demonstrate that DoPE significantly improves retrieval accuracy and reasoning stability across extended contexts (up to 64K tokens). The results show that the denoising strategy for positional embeddings effectively mitigates attention sinks and restores balanced attention patterns, providing a simple yet powerful solution for improving length generalization. Our project page is Project: https://The-physical-picture-of-LLMs.github.io

• The paper introduces DoPE, a training-free approach leveraging truncated matrix entropy to identify and denoise low-rank attention heads.
• It masks or replaces anomalous RoPE frequency bands to suppress attention sinks and enhance retrieval and reasoning in long contexts.
• Experimental results demonstrate up to a 10-point improvement in top-1 retrieval and increased robustness in many-shot in-context learning tasks.

Denoising Rotary Position Embedding via Truncated Matrix Entropy

Background and Motivation

Position encodings remain a critical facet of Transformer architectures tasked with modeling sequences of variable and extended length. Among prevailing strategies, Rotary Position Embedding (RoPE) is ubiquitously adopted in contemporary LLMs owing to its operational simplicity and successful encoding of relative positions within the inner product computation of attention. RoPE's formulation rotates query/key vectors in multiple low-dimensional planes using a well-defined base frequency schedule, aligning the attention score with relative positional offsets. Despite its widespread integration in models such as LLaMA3, Qwen3, and Mistral, RoPE—like other position encodings—faces fundamental inefficiencies when extrapolated beyond the pretraining context length, leading to severe degradation of retrieval accuracy, reasoning stability, and the emergence of attention sink artifacts.

Previous work (e.g., DAPE, NTK-based methods, and ALiBi) explored various mechanisms to circumvent these context length constraints, either by introducing additional learnable layers or adaptive frequency scaling. Nevertheless, the underlying causes of extrapolation failure and persistent low-rank anomalies in self-attention maps are poorly understood. This paper establishes a direct connection between RoPE-induced low-rank artifacts and spectral properties of the attention heads, introducing a training-free, parameter-free approach—Denoising Positional Encoding (DoPE)—to robustly identify and denoise affected heads through principled truncated matrix entropy analysis.

Truncated Matrix Entropy for Outlier Head Detection

The crux of DoPE is a spectral analysis of RoPE components at the attention head level. Empirically, only a minority of frequency bands contribute abnormally large $l_2$ norm artifacts, elucidated as row/column "bright bands" in the attention matrix—these channels exhibit pronounced low-rank structures. Formally, the Gram matrix of projected keys or queries for each band is analyzed, and its spectral mass concentration is quantified through matrix entropy. For each attention head, both the full matrix entropy and its truncated variant (retaining the first $r$ principal singular values) are computed:

$$H_{h,f} = - \text{tr}(E_{h,f} \log E_{h,f}), \quad p_{h,f} = \exp(H_{h,f})$$

Low entropy (and thus low effective rank) signals the dominance of a coherent, nearly rank-one direction, precisely the case where RoPE's periodic spectrum collapses and attention sinks arise. Truncated matrix entropy ($p_h$) is particularly adept at differentiating heads with isolated spectral spikes—critical for large context extrapolation tasks—while avoiding over-pruning isotropic, denoised representations.

Heads are selected for masking based on $p_h$ thresholds (either via quantile cuts or explicit $r$ settings), with denoising performed at the band or head level. This yields a series of strategies:

• DoPE-by-parts: Attenuate or remove RoPE bands for heads where $p_h$ falls below threshold;
• DoPE-by-all: Mask entire positional encoding for selected low-entropy heads;
• DoPE-by-Gaussian: Replace masked RoPE components with i.i.d. Gaussian noise matching empirical variance, restoring isotropy and acting as stochastic regularization.

This procedure is entirely training-free and parameter-free, relying exclusively on forward-pass statistics of query/key activations.

Theoretical Analysis

Through derivation, the paper exposes the spectral lower bound for attention scores under band-wise cone conditions. For a RoPE band acting on 2D subspaces, singular value analysis reveals the following:

• When RoPE frequencies are sufficiently low (i.e., rotate vectors within a narrow cone), the principal eigenvalue of the Gram matrix grows linearly with context length $N$.
• Alignment of left/right singular directions further yields rank-one artifacts in the score matrix, manifesting as sharp bright bands ("attention sinks") in self-attention maps.
• These spectral anomalies persist and amplify with longer context, confirming the prevailing recency bias and loss of retrieval/attention diversity.

Truncated matrix entropy directly quantifies this phenomenon by measuring spectral collapse, offering a precise operational definition of "noisy" heads.

Experimental Results

Needle-in-a-haystack Retrieval:

Experiments with Qwen2.5-Math-7B, LLaMA-3-8B, and Qwen-1.5-7B on contexts up to 64K tokens confirm that:

• Models exhibit abrupt degradation in retrieval accuracy post-extrapolation, with baseline RoPE/NTK approaches failing under noisy setups.
• DoPE strategies, particularly DoPE-by-Gaussian, improve top-1 retrieval scores by up to 10 points over Dynamic NTK baseline under 24K noisy setting (84.4 vs 75.4).
• For ultra-long contexts (64K), truncated matrix entropy with $r=1$ (spectral norm) consistently yields maximal gains, underscoring the utility of pinpointing rank-one heads for denoising.

Many-Shot In-Context Learning (MICL):

Model robustness in reasoning over the MATH benchmark reveals:

• Under 8K/16K context extension, DoPE variants outperform baseline by up to 0.04 in accuracy (0.393 vs 0.370 on needle-insert 8K setting).
• Head selection transferability across datasets (MATH vs NIH) demonstrates that entropy-based denoising remains effective, with no significant loss between tasks.
• Longer contexts induce a marked drop in performance, supporting the claim that reasoning complexity is bottlenecked by context extrapolation ability, not merely by available exemplars.

Attention Visualization and Ablation Studies:

Selective masking on low-entropy heads demonstrably suppresses sink artifacts and recency bias, restoring coherent attention distributions and enabling needle retrieval irrespective of context depth or local noise. Analysis of cosine similarity in query/eigenvector representations further confirms the low-rankness and periodicity of heads targeted by truncated matrix entropy, distinct from those identified by vanilla entropy metrics.

Practical Implications and Implementation Considerations

DoPE is notable for strict efficiency: it can be applied post hoc to any RoPE-encoded Transformer without retraining, minimal added computational overhead, and compatibility with existing FlashAttention and tensor-parallel inference backends. The primary cost is the up-front SVD computation for matrix entropy, tractable given low per-head dimensionality ($d_h \sim 64$–$128$).

This method addresses persistent long-context generalization failures in current Transformer models, essentially regularizing attention sinks without loss of capacity for context-dependent retrieval and reasoning. DoPE is modular and hyperparameter-free, allowing dynamic adaptation to deployed sequence lengths and target applications.

Implications and Future Directions

The theoretical framework establishes a new axis for position encoding analysis, connecting spectral entropy directly to extrapolation robustness. For practitioners, truncated matrix entropy offers an operational diagnostic for attention anomalies, enabling systematic debugging and auto-tuning of Transformer positional representations.

Future research may explore adaptive, data-dependent mask selection, integration with online attention statistics, and dynamic entropy computation during inference for highly variable-length tasks. Extensions to multi-modal and cross-modal models employing RoPE-like encodings present additional avenues, as does the investigation of entropy-based regularization in fine-tuning regimes.

Moreover, results suggest that exploiting the inherent Gaussian accumulation of layer-wise positional noise could be leveraged for unsupervised adaptation of position encodings, potentially improving generalization to non-natural sequence distributions and adversarial environmental perturbations.

Conclusion

Denoising Rotary Position Embedding using truncated matrix entropy provides an effective, training-free solution to the longstanding attention sink and recency bias problems in long-context Transformers. By targeting and suppressing outlier frequency bands at the attention head level, DoPE restores balanced attention, improves retrieval and reasoning stability, and enables robust length extrapolation in practice. The spectral entropy methodology delineated herein offers a principled path toward principled context-length extension in future LLM architectures.