Resonance 2D RoPE: Foundations & Applications

• Resonance 2D RoPE is a two-dimensional extension of rotary position embeddings that uses Lie theory to maintain relative and reversible spatial encodings.
• It employs a maximal abelian subalgebra approach in so(4) to construct axis-aligned and learned bases, facilitating independent and inter-dimensional frequency interactions.
• This method is significant for applications in image, video, and grid-structured data, offering precise periodicity and resonance through adaptable frequency parameters.

Resonance 2D RoPE refers to a mathematically principled extension of Rotary Position Embedding (RoPE) to two spatial dimensions, as formulated within a Lie-theoretic framework. This approach provides a foundation for 2D and N-dimensional position encoding in transformer models, ensuring properties critical to neural attention: relativity, reversibility, and, notably, the ability to capture resonant and periodic spatial interference effects. The core constructions are grounded in the identification of rotary encodings as elements of a maximal abelian subalgebra (MASA) of the special orthogonal Lie algebra, with explicit mechanisms for frequency resonance via axis-aligned and basis-learned rotations.

1. Mathematical Foundation: Core Properties and 2D Specialization

Two central properties define valid 2D RoPE:

1. Relativity: The attention similarity function under RoPE depends only on positional differences. Formally, for $x_1, x_2 \in \mathbb{R}^2$ and associated rotation matrices $R_{x_1}, R_{x_2} \in \mathrm{SO}(4)$,

$$(R_{x_1} q)^\mathrm{T}(R_{x_2} k) = q^\mathrm{T} R_{x_1}^\mathrm{T} R_{x_2} k,$$

which must depend only on $x_2 - x_1$: $R_{x_1}^\mathrm{T} R_{x_2} = R_{x_2 - x_1}$. In two dimensions, this condition specializes to

$$R_{(x_1, y_1)}^\mathrm{T} R_{(x_2, y_2)} = R_{(x_2 - x_1, y_2 - y_1)}.$$

2. Reversibility (Injectivity): The map from coordinates $(x, y)$ to rotation matrices $R_{(x, y)}$ must be injective: $R_{x_1}=R_{x_2} \implies x_1=x_2$. In practice, injectivity holds within each $2\pi/\omega$ period for base frequency $\omega$, thus frequency selection governs the feasible domain.

These properties are essential for maintaining the integrity of positional information and ensuring that transformers learn meaningful relative and absolute spatial relationships (Liu et al., 7 Apr 2025).

2. RoPE and the Structure of so(4): MASA and Basis Construction

In dimension $d=4$, the Lie algebra $\mathfrak{so}(4)$ has rank 2, enabling a two-dimensional MASA for generator construction. There are two canonical approaches:

• Toral (Axis-Aligned) Basis: The standard basis is built from the commuting skew-symmetric matrices $E_{12}$ and $E_{34}$, which correspond to rotations in the $(1, 2)$ and $(3, 4)$ planes. Explicitly,

$$E_{12} = \begin{pmatrix} 0 & -1 & 0 & 0 \ 1 & 0 & 0 & 0 \ 0 & 0 & 0 & 0 \ 0 & 0 & 0 & 0 \end{pmatrix}, \quad E_{34} = \begin{pmatrix} 0 & 0 & 0 & 0 \ 0 & 0 & 0 & 0 \ 0 & 0 & 0 & -1 \ 0 & 0 & 1 & 0 \end{pmatrix}.$$

The basis $B_1 = \omega_1 E_{12}$, $B_2 = \omega_2 E_{34}$ encodes independent rotations along $x$ and $y$ axes for frequencies $\omega_1$, $\omega_2$.

• General MASA via Learned Basis: More expressive RoPE can be obtained by learning an orthonormal change of basis $Q \in \mathrm{SO}(4)$. Any MASA basis is $$B_i = Q \, \mathrm{diag}(J(\lambda_i), J(\mu_i)) Q^\mathrm{T}$$, with $$J(\lambda) = \lambda \begin{bmatrix} 0 & -1 \ 1 & 0 \end{bmatrix}$$. Mixing the axes in this way enables modeling of inter-dimensional frequency interactions.

A summary table organizing axis-aligned and learned-basis variants:

Approach	Generator Construction	Properties
Axis-aligned	$E_{12}, E_{34}$	Independent axes
Learned (Q)	$$B_i = Q \,\mathrm{diag}(J(\lambda), J(\mu)) Q^\mathrm{T}$$	Inter-dimensional interaction

The general framework supports both predefined and learned frequency bases, while ensuring commutativity and invertibility.

3. 2D RoPE: Generators, Frequencies, and Rotation Operator Formulation

Given a 2D coordinate $(x, y) \in \mathbb{R}^2$, the 2D RoPE rotation is parameterized as follows:

• Generators: $G_1 = E_{12}$, $G_2 = E_{34}$.
• Angles: $\theta_1(x) = \omega_1 x$, $\theta_2(y) = \omega_2 y$.
• Rotation Operator:

$$R(x, y) = \exp\big[\theta_1(x) G_1 + \theta_2(y) G_2\big] = \begin{pmatrix} \cos(\omega_1 x) & -\sin(\omega_1 x) & 0 & 0 \ \sin(\omega_1 x) & \cos(\omega_1 x) & 0 & 0 \ 0 & 0 & \cos(\omega_2 y) & -\sin(\omega_2 y) \ 0 & 0 & \sin(\omega_2 y) & \cos(\omega_2 y) \end{pmatrix}.$$

With a learned basis $Q$, this operator becomes $$R(x, y) = Q \exp[\theta_1(x) G_1 + \theta_2(y) G_2] Q^\mathrm{T}$$, allowing for axis mixing. The formulation ensures rapid computation of the rotary embedding via block-diagonal sine/cosine rotations, preserving the relativity and reversibility properties (Liu et al., 7 Apr 2025).

4. Resonance, Periodicity, and Multi-Frequency Interference

Resonance in 2D RoPE arises from specific frequency choices and their ratios, leading to periodic or interference patterns over the spatial domain.

• Periodicity: The mapping is periodic along each axis, with period $2\pi/\omega_1$ or $2\pi/\omega_2$ respectively. When both $\omega_1 x \equiv 0 \pmod{2\pi}$ and $\omega_2 y \equiv 0 \pmod{2\pi}$, the rotation reduces to identity.
• Resonance and Interference: If $\omega_2/\omega_1 = p/q \in \mathbb{Q}$, then the rotation operator is periodic on a lattice where $p x - q y \in \mathbb{Z} (2\pi/\omega_1)$. For multi-frequency encodings, stacking $k$ blocks per axis with frequencies $\{\omega_1^{(j)}, \omega_2^{(j)}\}$ yields combined encodings; constructive interference arises at points where all base phases align.
• Frequency Selection: Geometric progression of frequencies $\{\omega_i^{(j)}\}$ covers multiple spatial scales. Lower $\omega$ offers coarser position resolution, while higher $\omega$ gives finer resolution. To avoid aliasing, it is essential that $$\text{max position} \times \text{max} \; \omega < 2\pi$$.

This structure enables transformer models to represent complex spatial periodicities and resonances, crucial for tasks involving 2D or grid-structured data.

5. Implementation and Training Guidelines

In transformer architectures employing 2D RoPE, application involves procedural steps supported by the theoretical guarantees:

1. Frequency Selection: Practitioners choose base frequencies $(\omega_1, \omega_2)$, optionally with multi-scale (vector-valued) frequencies for each axis.
2. Orthonormal Basis Learning: Optionally, a mixing matrix $Q \in \mathrm{SO}(4)$ is initialized and trained to model cross-axis correlations. $Q$ must remain orthonormal, which can be enforced via Cayley transform ($Q = (I-A)(I+A)^{-1}$ with $A^\mathrm{T} = -A$), exponential map ($Q = \exp(A)$), or Givens rotations (parameterize as sequential plane rotations).
3. Computation: For each batch or token, compute per-axis phases $\varphi_1 = \omega_1 X$, $\varphi_2 = \omega_2 Y$, then sines and cosines. The RoPE operator is applied to split vectors along $(0,1)$ and $(2,3)$ dimensions, with optional $Q$ transforms before and after.
4. Transformer Layer Integration: The attention mechanism computes similarity using RoPE-transformed queries and keys, such that attention weights depend only on relative positions, satisfying both core properties.

The pseudocode officially provided in (Liu et al., 7 Apr 2025) captures batched vectorized processing and ensures basis orthogonality throughout optimization.

6. Applications and Significance

Resonant 2D RoPE generalizes the utility of rotary position embeddings for image, video, and grid-structured data modalities, where spatial locality and periodicity are essential. The mathematical foundation enables principled extension to $N$ dimensions and supports adaptive learning of coordinate interactions via basis mixing. This unifying view reconciles specialized variants of RoPE and informs frequency selection, basis learning, and implementation practices for robust spatial representation in large-scale neural attention systems (Liu et al., 7 Apr 2025).

A plausible implication is that this construction, by enforcing relativity, reversibility, and resonance structure, provides a blueprint for generalized, provably valid positional encodings in any domain where transformers handle structured or geometric input.