Depth-Aware Rotary Positional Encoding

• Depth-Aware Rotary Positional Encoding is a method that extends standard RoPE by encoding hierarchical, spatial, and multi-dimensional relationships in transformers.
• It leverages layer-specific scaling and multi-axis rotations to capture diverse positional cues, thereby enhancing long-range dependency modeling.
• This approach improves transformer robustness and performance in multi-modal tasks by adapting positional encoding to both depth and geometric structure.

Depth-aware rotary positional encoding (DA-RoPE) refers to a class of positional encodings for transformer architectures that generalize the core idea of rotary position embedding (RoPE) to capture not only sequential position but also additional axes of structure—such as hierarchical (depth), multi-dimensional (e.g., 2D/3D), or layer-dependent relationships. The canonical RoPE directly injects position via a block-diagonal rotation matrix applied multiplicatively to query and key vectors in attention; DA-RoPE extends this in several directions: adaptation to multiple spatial axes, explicit parameterization of encoding transformations as a function of network depth (layer index), and the embedding of structured geometric or hierarchical features. Recent research demonstrates that such generalizations are critical for robust modeling in multi-modal, long-range, or hierarchical settings, especially where classic 1D or static positional encodings are insufficient.

1. Core Principles of Rotary Positional Encoding

Rotary positional encoding introduces absolute positional information through a rotation matrix applied to queries and keys prior to attention calculation (Li et al., 2021). For an input vector $x_m$ at position $m$, and projection matrices $W_q$, $W_k$, the position-modulated representations are: $$f_{(q)}(x_m, m) = R_{(\Theta, m)}^d \cdot (x_m W_{(q)}), \quad f_{(k)}(x_n, n) = R_{(\Theta, n)}^d \cdot (x_n W_{(k)}).$$

$R_{(\Theta, m)}^d$ is a block-diagonal rotation matrix with 2D rotational blocks: $$M_i = \begin{bmatrix} \cos(m\theta_i) & -\sin(m\theta_i) \ \sin(m\theta_i) & \cos(m\theta_i) \end{bmatrix}, \quad \theta_i = 10000^{-2(i-1)/d}.$$

The key property is that attention is a function of relative distance (i.e., $m-n$) owing to the group property of rotations: $$\langle f_{(q)}(x_m, m), f_{(k)}(x_n, n) \rangle = g(x_m, x_n, m-n).$$

RoPE is multiplicative, efficient, and naturally incorporates relative position into the transformer, outperforming classical absolute position encodings in domains such as speech recognition (Li et al., 2021).

2. Extension to Depth, Hierarchy, and Multi-Dimensional Inputs

Depth-aware and multi-dimensional generalizations of RoPE seek to encode more than just linear sequence order.

2.1 LieRE and Full Rotation Matrices

Lie Relative Encodings (LieRE) generalize RoPE by moving from block-diagonal 2D rotations to full $n\times n$ rotations parameterized via Lie group generators (Ostmeier et al., 14 Jun 2024): $R_{\text{LieRE}} = \exp(A\mathbf{x}),$ with $A$ a learned skew-symmetric matrix and $\mathbf{x} \in \mathbb{R}^d$ the (possibly multidimensional) token position. The attention between two tokens at positions $x$ and $y$ becomes: $$\text{Attention}(x, y) : \langle q, k \rangle = q^T \exp[A(y - x)] k,$$ encoding relative positions (and thereby "depth" or spatial axes) generally, supporting 2D/3D spatial and temporal data.

2.2 Axial, Depth-Partitioned, and Joint Spatial-Temporal RoPE

Depth-aware behavior is also achieved by splitting the embedding/channel space so that each axis (e.g., $x$, $y$, $t$, $z$, or even layer index) receives its own RoPE module (Zivanovic et al., 26 May 2025, Bai et al., 23 Oct 2025, Wang et al., 17 Jun 2025). For 2D/3D/temporal data, the embedding is divided into chunks such that

$$\text{Embedding}_\text{token} = [\text{Chunk}_x,\,\text{Chunk}_y,\,\text{Chunk}_z],$$

and for each chunk (axis), a separate RoPE rotation is applied.

In advanced DA-RoPE schemes, such as those for video-LLMs or 3D-aware diffusion transformers (Wang et al., 17 Jun 2025, Bai et al., 23 Oct 2025, Feng et al., 24 Mar 2025), spatial-temporal interactions are encoded by taking multiplicative compositions of RoPE along spatial and temporal axes: $R_{xy+t} = R_{xy} \cdot R_{t},$ where $R_{xy}$ and $R_t$ are rotation matrices derived from 2D patch position and time index, respectively. This composition allows attention to flexibly aggregate features from different spatial and temporal "depths" within a scene.

3. Layer-Adaptive and Depth-Modulated Rotary Encodings

Practical DA-RoPE approaches adapt the rotation mechanism based on the network layer (depth).

Recent work introduces layer-specific scaling factors $s_i$ for each transformer layer $i$ (Wang et al., 6 Mar 2025). Instead of globally applying the same positional frequency, each layer's RoPE is scaled according to a Bézier curve (parameterized by a small set of control points) and optimized via a genetic algorithm. This reduces the "lost-in-the-middle" effect by altering the decay rate of positional signal at each depth, sharpening attention in later layers and broadening context coverage in early ones. The scaling per layer is expressed as: $$\mathbf{S} = B(t),\quad t \in \left\{ \frac{k}{n} \mid k=0,1,\dotsc,n \right\},$$ with $B(t)$ the Bézier curve, and $n$ is number of layers.

Similar principles can be applied to set block-wise or trainable frequency parameters, so that $\theta_i$ becomes a learned function of model depth, position, or data modality (Yu et al., 4 Jun 2025, Ostmeier et al., 14 Jun 2024).

4. Geometry, Volumetric, and Hierarchical Structure: 3D- and Depth Encoding

Applications requiring explicit volumetric or multi-view modeling benefit from depth-aware rotary positional encoding.

4.1 3D-Aware Rotary Embedding in Vision and Texture Synthesis

For tasks such as 3D-aware texture synthesis, DA-RoPE encodes each token's 3D position (e.g., voxel grid or mesh coordinate) via rotary embedding (Feng et al., 24 Mar 2025): $$\varphi(i,j) = f_{\text{PE}}\{ \mathcal{V}^l[ \text{round}(pos_x^l(i,j) R^l), \text{round}(pos_y^l(i,j) R^l), \text{round}(pos_z^l(i,j) R^l)] \},$$ supplementing the latent token with a 3D positional code.

In diffusion Transformers for vision, Positional Encoding Field (PE-Field) augments standard 2D RoPE to a 3D field by allocating subspaces of the embedding to $(x, y, z)$, and attaches a distinct RoPE rotation to each (Bai et al., 23 Oct 2025). Hierarchical (multi-scale) encodings assign different spatial resolutions to different heads or subspaces, affording fine-grained control at subpatch levels and facilitating novel view synthesis and volumetric spatial editing.

4.2 Decoupling Cross-Modal Positional Bias

For cross-modal transformers (e.g., vision-LLMs), DA-RoPE can mitigate cross-modality biases by geometrically decoupling the positional trajectories (e.g., placing image tokens on a circle orthogonal to text tokens, forming a cone-like configuration) so that every text token is equidistant from all image tokens, while still preserving intra-image structure (Wang et al., 22 May 2025).

5. Depth-Aware RoPE and Generalization: Multiscale, Spectral, and Extrapolation Properties

DA-RoPE facilitates improved generalization, multi-scale analysis, and extrapolation to long contexts or out-of-distribution scenarios.

Wavelet-based positional encodings show that standard RoPE is akin to a fixed-scale, Haar-like wavelet transform, which limits extrapolation (Oka et al., 4 Feb 2025). By adopting variable-scale wavelet transforms on positional differences, depth-aware systems can integrate local and global context across resolutions, naturally generalizing to longer or irregular sequences:

$$p_{m,n} = \left(1 - \left( \frac{m-n-b}{a} \right)^2 \right) \exp \left( -\frac{1}{2} \left( \frac{m-n-b}{a} \right)^2 \right),$$

where scale $a$ and shift $b$ are sampled over a range of window sizes.

Spectral analyses further reveal that RoPE (and its generalized forms) contract the spectrum of the attention logit matrix due to the Hadamard product with the underlying Toeplitz (circulant) structure induced by relative rotary rotations, leading to improved optimization dynamics and stability on position-sensitive tasks (Gu et al., 19 May 2025). This spectral regularization—enhanced by depth-aware adaptations—enables better length generalization and robust performance on extended contexts.

6. Challenges, Limitations, and Adaptive Designs

Despite its strengths, DA-RoPE presents certain challenges in both design and extrapolation.

• Fixed-frequency (static) RoPE can limit extrapolation and induce U-shaped or oscillatory attention decay at long distances, motivating high-frequency and component-suppressed variants such as HoPE (Chen et al., 28 Oct 2024) or monotonic decay via hyperbolic rotations (Hyperbolic Rotary Positional Encoding) (Dai et al., 5 Sep 2025).
• Low-frequency rotary pairs can lead to attention sinks or large outlier features that dominate attention, especially in deep or quantized models, suggesting the importance of frequency selection and outlier-aware schemes (Jonasson, 3 Mar 2025).
• Rigidity in the positional encoding can hinder numerical reasoning and arithmetic generalization, while augmenting token representations with random position tags or adaptive, hierarchical encodings enhances compositionality and depth sensitivity (Shen et al., 2023).

Adaptive and learnable generalizations—e.g., trainable commuting rotation matrices in ComRoPE (Yu et al., 4 Jun 2025) or content-aware phase encodings as in TAPA (Yu et al., 16 Sep 2025)—further increase robustness, scalability, and flexibility, addressing long-context modeling and multi-dimensional data requirements in practice.

7. Empirical Impact, Modalities, and Benchmarks

DA-RoPE approaches, in their diverse forms, consistently outperform static, absolute, or non-adaptive positional encoding baselines on a wide range of modalities and tasks.

• In end-to-end speech recognition, RoPE-augmented conformer models yield 8.7% and 7.27% relative WER reductions on LibriSpeech and ~4% relative CER reduction on AISHELL-1 (Li et al., 2021).
• LieRE and ComRoPE achieve state-of-the-art or superior accuracy on 2D and 3D image classification benchmarks, highlighting the value of generalizing to learnable, high-dimensional rotations (Ostmeier et al., 14 Jun 2024, Yu et al., 4 Jun 2025).
• Video and video-LLMs equipped with joint spatial-temporal or 3D-aware RoPEs attain new performance benchmarks and perceptual robustness, as evidenced by improvements in retrieval accuracy, CLIP-based scores, and human user studies (Wang et al., 17 Jun 2025, Feng et al., 24 Mar 2025).
• On long-context language modeling and memory tasks, adaptive schemes such as high-frequency RoPE, layer-specific scaling, or monotonic decay outperform traditional RoPE, ALiBi, and fixed schemes, particularly in perplexity and context retention (Chen et al., 28 Oct 2024, Wang et al., 6 Mar 2025, Dai et al., 5 Sep 2025).

The breadth of demonstrated improvements underscores DA-RoPE's centrality in scaling, generalizing, and specializing transformers across increasingly complex and high-dimensional domains.

---

In sum, depth-aware rotary positional encoding is a unifying abstraction for the latest advances in transformer position encoding: it enables multiplicative, relative, and adaptable representations that cover multiple dimensions or hierarchies, support structured spatial or volumetric reasoning, adapt to network depth, and excel on long-range dependencies and multi-modal integration. As such, DA-RoPE—manifested through block or full rotation matrices (possibly trainable), adaptive scaling, multi-axis coupling, and hierarchical design—constitutes a foundational mechanism in contemporary and emerging transformer architectures.