Denoising Positional Encoding (DoPE)

• Denoising Positional Encoding (DoPE) is a training-free method that reinterprets Transformer attention maps as noisy feature maps to identify and correct outlier frequency bands.
• It uses truncated matrix entropy to detect low-entropy, spiky bands and applies parameter-free Gaussian reparameterization to replace problematic RoPE features with isotropic noise.
• Empirical results on long-context tasks show that DoPE effectively restores uniform attention distributions and enhances performance compared to baseline methods without model retraining.

Denoising Positional Encoding (DoPE) is a training-free methodology introduced to address the limitations in length extrapolation of Transformer models that utilize Rotary Position Embedding (RoPE). By reinterpreting the attention map with positional encoding as a noisy feature map, DoPE provides a principled approach based on truncated matrix entropy to detect and correct outlier frequency bands responsible for pathological attention behaviors—most notably, attention sinks. The method is parameter-free, leveraging Gaussian reparameterization to restore robustness in long-context inference without any model retraining or fine-tuning.

1. RoPE Attention as a Noisy Feature Map

In standard multi-head self-attention, each head operates on linearly projected queries and keys $Q, K \in \mathbb{R}^{n \times d_h}$, with positional information injected through RoPE rotations. The RoPE operator $R(\theta_i)$, block-diagonalized over frequency bands $f=0..\frac{d_h}{2}-1$, rotates each 2D band by an angle determined by position $i$. Thus: $$\mathrm{QR}_i = R(\theta_i) Q_i,\quad \mathrm{KR}_j = R(\theta_j) K_j,\quad \theta_m = m \omega$$ The causal attention score is

$$A_{ij} = \mathrm{softmax}_k \left( \frac{\langle \mathrm{QR}_i, \mathrm{KR}_j \rangle}{\sqrt{d_h}} + M_{ij} \right)$$

Decomposing the inner product over bands yields: $$\langle \mathrm{QR}_i, \mathrm{KR}_j \rangle = \sum_{f=0}^{d_h/2-1} \langle P_f \mathrm{QR}_i, P_f \mathrm{KR}_j \rangle$$ with $P_f$ the projection to the $f^{th}$ 2D band ($Q'_f, K'_f \in \mathbb{R}^{n\times2}$). Each band thus contributes its own block to the overall attention map. Spectral analysis indicates that in low-frequency bands (bands where rotational phases vary slowly, establishing a "cone condition"), the Gram matrix

$E_f = \frac{1}{N} \sum_{j=1}^N K'_{f,j} K'_{f,j}^\top$

exhibits one dominant eigenvalue $\gtrsim N B_\mathrm{min}^2 \cos^2 \alpha$, leading to a near-rank-one spike in $Q'_f K'_f{}^\top$ of magnitude $O(N/\sqrt{d_h})$, which remains prominent under softmax normalization. These spikes, termed "attention sinks," persist as $N$ grows, representing noise in the attention feature map arising from outlier RoPE bands.

2. Truncated Matrix Entropy for Outlier Detection

To identify spiky, low-entropy frequency bands, DoPE computes the 2×2 band-wise Gram matrix per head: $E_{h,f} = \frac{1}{N} K'_{h,f} K'_{h,f}^\top$ Normalizing to unit trace,

$$\widetilde{E}_{h,f} = \frac{E_{h,f}}{\mathrm{tr}(E_{h,f})}$$

the matrix entropy for each band is

$$H_{h,f} = -\mathrm{tr} \Bigl( \widetilde{E}_{h,f} \log \widetilde{E}_{h,f} \Bigr),\qquad H_{h,f} \in [0, \log 2]$$

A mean across bands yields the head-level entropy $H_h$. However, to isolate dominant modes, DoPE employs a truncated effective rank based on the top $r$ eigenvalues $\lambda_1 \ge \cdots \ge \lambda_r$: $$p_{h,f}^{(r)} = \exp\left( -\sum_{i=1}^r \tilde{\lambda}_i \log \tilde{\lambda}_i \right),\quad \tilde{\lambda}_i = \frac{\lambda_i}{\sum_{j=1}^r \lambda_j}$$ $p_{h,f}^{(r)} \approx 1$ signals a single-eigenvalue spike (nearly rank-one), while values near $r$ indicate isotropy. Aggregating $p_{h,f}^{(r)}$ over bands provides a per-head score $p_h^{(r)}$, identifying heads with persistent attention sinks.

3. Parameter-Free Gaussian Reparameterization

Upon determining a binary mask $m_h \in \{0,1\}$ for each attention head $h$ by thresholding $p_h^{(r)}$ above a selected quantile, DoPE replaces RoPE features in spiky ("bad") heads ($m_h=0$) by isotropic Gaussian noise: $$\varepsilon_{Q,h}, \varepsilon_{K,h} \sim \mathcal{N}(0, \sigma_h^2 I_{n \times d_h})$$ with $\sigma_h^2$ assigned as the empirical variance of non-spiky RoPE features in head $h$. The revised (denoised) queries and keys are: $$Q_h^{\mathrm{D}} = m_h Q_h + (1-m_h) \varepsilon_{Q,h},\qquad K_h^{\mathrm{D}} = m_h K_h + (1-m_h) \varepsilon_{K,h}$$ This modification eliminates persistent spikes by restoring zero-mean, isotropic positional variation in problematic heads, requiring neither new learnable parameters nor any form of retraining.

4. Theoretical Analysis: Attention Sinks and Truncated Entropy

Spike analysis (Lemma A.1 & Corollary A.3) formalizes the equivalence between low truncated entropy and attention sinks; if a band $f$ has one dominating eigenvalue $\lambda_{\max}(E_f) \gtrsim O(N)$, then $Q'_f K'_f{}^\top$ exhibits entries of $O(N/\sqrt{d_h})$. After softmax, these retain $\Omega(1)$ magnitude as $N \to \infty$, causing attention mass to collapse onto a few recent positions. The scalar $p_{h,f}^{(r)} \approx 1$ precisely selects such spike bands. Masking or reparameterizing these bands removes the $\Omega(1)$ spikes, restoring uniform, well-distributed attention over the entire context.

5. Algorithmic Implementation Summary

For a single forward pass, the DoPE algorithm proceeds as follows:

The approach is applicable post hoc to existing Transformer models, imposing only a minor computational overhead for Gram-matrix eigendecomposition.

6. Empirical Results on Long-Context Tasks

All experiments apply DoPE exclusively at inference without fine-tuning, on standard pre-trained checkpoints (LLaMA-3-8B, Qwen2.5-Math-7B). The "Dynamic-NTK" RoPE rescaling baseline (emoZilla, 2023) serves as comparison. Results cover both the needle-in-a-haystack (NIH) task and many-shot in-context learning (ICL) on the MATH dataset.

Task	Baseline (Dynamic-NTK)	DoPE (Best Variant)
NIH @24K (LLaMA-3-8B)	75.42%	84.35% (+8.9 points)
NIH @64K (LLaMA-3-8B)	40.42%	40.88% (+0.46 pts)
ICL @8K (many-shot, Math-7B, needle-insert)	0.370	0.393 (+0.023)
ICL @8K (skip)	0.370	0.410 (+0.040)
ICL @16K (needle-insert)	0.240	0.228 (comparable)
ICL @16K (skip)	0.240	0.250 (+0.010)

DoPE consistently mitigates the degradation effects induced by attention sinks, preserves balanced attention distributions up to 64K tokens, and operates entirely without model modification or additional training. The computational cost is dominated by per-head, per-band Gram-matrix eigendecomposition (dimension 2×2), which is negligible at inference time in common Transformers.

7. Significance and Implications

Denoising Positional Encoding directly addresses inherent extrapolation weaknesses in RoPE-based Transformers, offering a practical, parameter-free solution to the attention sink pathology—a phenomenon characterized by persistent attention hot spots caused by a few corrupt frequency bands. By reframing the attention map as a noisy feature map and utilizing truncated matrix entropy for outlier detection, DoPE theoretically and empirically elucidates the connection between low-entropy bands and attention sinks. Its success in robust length extrapolation without retraining or added parameters marks it as a notable advancement in positional encoding methodologies for long-context LLMs. A plausible implication is that similar spectral analysis and denoising strategies may prove effective in mitigating analogous failures in other componentwise or frequency-decomposed architectures.