Why Do Accumulated Transformations Extrapolate?

Abstract: PaTH Attention showed that replacing RoPE's position-indexed rotations with accumulated data-dependent Householder reflections yields strong length extrapolation, though performance degrades at extreme context lengths. We ask whether this depends on Householder-specific structure or reflects a general property of accumulated transformations along source-to-query paths. We study a simpler variant keeping RoPE's block-diagonal SO(2) rotations but replacing position-indexed angles with accumulated token-dependent ones. It shows the same pattern: improved extrapolation then degradation at long contexts. We prove the result extends to accumulated orthogonal transformations satisfying certain regularity conditions: their products become incoherent after finitely many steps, suppressing attention to distant tokens. Accumulated rotations of queries and keys create a finite mixing window independent of context length; per-token suppression learned in training transfers unchanged to any evaluation length, and high-dimensional concentration produces a score gap suppressing far tokens while near-route transport preserves the target signal. Conversely, a lower bound shows accumulated rotations must eventually degrade: as the far set grows, no rotations preserve the near signal without explicit far-mass control. For SO(2) rotations, rotating values too makes residual far contributions combine incoherently, extending the range. Controlled experiments support these predictions: random accumulated rotations substantially improve extrapolation over RoPE, learned token-dependent rotations maintain near-training-length perplexity far beyond the training context, and rotating values helps over queries and keys alone. Rotation-only models still degrade at extreme lengths, while ALiBi stays length-stable, consistent with the need for far-mass control.

• The paper demonstrates that accumulated orthogonal transformations create a fixed mixing window via score-side decoherence, which is crucial for length extrapolation.
• Experiments reveal that integrating both query/key and value rotations dramatically reduces long-context perplexity compared to standard RoPE.
• The study underscores that explicit far-mass control is necessary for unbounded extrapolation, unifying various architectural approaches under one framework.

Mechanistic Underpinnings of Length Extrapolation with Accumulated Transformations

Introduction

This work rigorously investigates why transformers equipped with accumulated, data-dependent orthogonal transformations in their attention mechanisms exhibit empirical length extrapolation. Focusing initially on the findings of "PaTH Attention" (Yang et al., 2025), which replaces traditional RoPE encodings (Su et al., 2024) with cumulative Householder reflections, the study generalizes this framework to a broader class of orthogonal transformations, including block-diagonal SO(2) rotations. The central question is whether improved extrapolation emerges from Householder-specific properties or from a more universal mechanism attached to the nature of accumulated transport along attention paths.

Core Theoretical Findings

Score-Side Decoherence via Random Matrix Products

The exposition begins with a clear, operator-theoretic treatment of how accumulated transformations along source-to-query paths induce a finite "mixing window" that is independent of the context length. Given a per-token orthogonal transformation $M_t \in \mathrm{O}(d)$ (potentially content-dependent), the product over a route (i.e., a source-to-query interval) is $P_{j \to i} = M_{i-1}^{-1} \cdots M_j^{-1}$. If the step distribution admits a strict spectral gap ($\| \mathbb{E}[M_t] \|_{\mathrm{op}} < 1$), the expected product norm decays exponentially with distance, leading to rapid score-side decoherence for sufficiently separated tokens.

High-dimensional concentration and TV-mixing arguments demonstrate that, in the far regime, these accumulated products act nearly Haar-randomly on the sphere, ensuring that transported keys become incoherent with queries, thus suppressing the selection of distant tokens. The boundary between the "near" and "far" regimes—established by the mixing window—remains fixed across varying context lengths, creating a train/eval distributional match necessary for extrapolation.

Fundamental Limitation: Far-Mass Lower Bound

A key negative result established is that rotation-based suppression, while effective at moderate lengths, cannot structurally guarantee bounded far-token attention under the full softmax when the candidate far set grows. The paper rigorously proves that, even with arbitrarily strong per-token suppression, the absence of explicit far-mass control leads to total attention mass eventually leaking to far tokens as context grows ($\rho_D \to 1$ as $M_L\to\infty$). Thus, architectures such as ALiBi that directly bias against distant tokens provide superior (flat) extrapolation compared to any rotation-only scheme.

Value-Side Decoherence with Commuting Rotations

The SO(2) specialization permits a secondary, value-path (V) analysis unavailable for general non-commuting orthogonals. By also rotating values with the route product, residual far-mass contributions, even if present after score-side selection, mix incoherently due to the commutative/additive phase structure, which is precisely analysable via per-block Fourier/statistical decoupling. This extends the length regime before degradation manifests and tightens spectral bounds on far-value covariance. The same-layer limitation of this mechanism for non-commuting (e.g., Householder) settings is noted—the value-side suppression analysis is presently unique to the commutative SO(2) case.

Near-Route Signal Preservation

The preservation of the signal of interest for "near" tokens boils down to the smallness of accumulated deviation from the identity on short routes. Sufficiently well-aligned accumulated maps (with tight concentration of route transformations around the identity for the near window) allow robust recovery of the latent signal, even post-aggregation, provided far-mass suppression remains effective.

Experimental Evidence

The study deploys controlled ablations using decoder-only transformers trained on OpenWebText with context length 512, evaluating up to 65,536 tokens. Four primary models are contrasted: standard RoPE (position-indexed), accumulated random rotations (fulfilling spectral gap conditions by construction), learned token-dependent rotations, and the ALiBi distance-bias baseline.

Findings include:

• Substantial reduction in long-context perplexity blow-up: Accumulated random rotations (Q/K) reach a $7.5\times$ perplexity increase at maximum context, vs. $38.5\times$ for RoPE. With value rotation (Q/K/V), degradation reduces further to $1.6\times$.
• Learned token rotations: These match RoPE at training length but maintain moderate degradation ($5.9\times$ for Q/K at longest context). Adding value rotation provides an additional benefit.
• ALiBi maintains near-flat perplexity ($0.96\times$), confirming the necessity of explicit far-mass control.
• Rotating only queries and keys improves extrapolation, but rotating values yields robust suppression of far-token interference, in line with theoretical predictions.

Implications

Theoretical

These results solidify the perspective that general random product phenomena and high-dimensional concentration suffice for explaining observed length extrapolation under accumulated, content-dependent orthogonal transformations—provided strong mixing (spectral gap, TV-mixing) is present. This unifies disparate schemes (PaTH, randomized RoPE, accumulated SO(2), etc.) under a single framework. The insight that commutativity/structure enables analytic value-side suppression, while non-commutativity currently lacks this, points to a new technical barrier for mechanistic interpretability.

Crucially, score-side randomization alone does not guarantee unbounded extrapolation; explicit architectural design for far-mass control is structurally required (as in ALiBi, FoX, or gating mechanisms).

Practical

From an engineering standpoint, rotation-based approaches provide orders-of-magnitude improvement in extrapolation window but are still not fully stable for extreme lengths unless supplemented by methods that directly enforce far-mass suppression. Combining these mechanisms is highlighted as a promising avenue (e.g., PaTH-FoX). The commutative SO(2) variant offers a tractable test-bed for theoretical analysis and ablations, and may inform future architectural choices for both scaling and interpretability.

Limitations and Future Directions

• Non-commuting Value-Side Decoherence: Extending value-rotation-induced incoherence results to accumulated Householder or more general orthogonal transformations is an open problem, requiring new concentration techniques for non-commutative random matrix products.
• Empirical Mechanistic Probes: Direct measurement of empirical spectral gaps, attention allocations, and value-path statistics in large-scale models is necessary to bridge theory and practice.
• Multi-Layer Dynamics: All current analysis is single-layer; compositional effects from stacking remain unresolved and represent a gap between stylized model predictions and observed gradual degradation.
• Some generalization results (e.g., for Position Interpolation, Selective RoPE) do not apply, as these use deterministic transport steps, lacking the required mixing properties.

Conclusion

The paper provides a rigorous, comprehensive answer to the mechanism underlying length extrapolation via accumulated transformations in transformer attention. Suppression of far-token attention emerges from universal properties of random matrix products subject to a spectral gap, but complete length stability further requires explicit far-mass control. Value-side decoherence (for SO(2)) extends the regime of stable extrapolation. Overall, a principled architectural decomposition into train/test-matched mixing, score suppression, and value incoherence is advanced, with empirical support and explicit limitations, charting a concrete path for future theoretical and applied research in long-context transformers.