AS2DRoPE: 2D Rotary Positional Encodings

• Two-Dimensional Rotary Positional Encodings (AS2DRoPE) are a 2D generalization of 1D rotary embeddings that use axis-separable rotations to encode spatial coordinates.
• They apply SO(2) rotations with learned or geometric frequencies for each 2D subspace, maintaining translation invariance in the embedding space.
• AS2DRoPE is integrated into multi-head self-attention layers in Vision Transformers, ensuring efficient and explicit encoding of spatial displacements.

Two-dimensional rotary positional encodings (AS2DRoPE) generalize 1D Rotary Position Embeddings (RoPE) to structured tensors with spatial coordinates, primarily for use in Vision Transformers and agent-based modeling frameworks. AS2DRoPE extends the translation-invariant, relative-position properties of RoPE to 2D, enabling transformers to encode spatial displacements explicitly and efficiently in multi-head self-attention. The term AS2DRoPE is used for canonical “axial-separable” 2D rotary schemes in several works, and forms the reference implementation for practical 2D relative position encoding (Liu et al., 3 Feb 2026, Yao et al., 4 Dec 2025, Zhao et al., 19 Mar 2025, Schenck et al., 4 Feb 2025).

1. Mathematical Formulation

Let $d$ be the embedding dimension of a token. In the standard axis-separable 2D RoPE (“AS2DRoPE”), the $d$-dimensional token embedding is divided into $d/2$ contiguous 2D subspaces. For token position $(x, y)$, each block is rotated in the complex plane by an angle that is a linear function of $x$ and $y$:

$$\mathrm{Rot}(x, y) = \bigoplus_{n=1}^{d/2} \rho(x\,\theta_n^x + y\,\theta_n^y),$$

where $$\rho(\phi)=\begin{bmatrix}\cos \phi & -\sin \phi \ \sin \phi & \cos \phi\end{bmatrix}$$. The frequencies $\theta_n^x, \theta_n^y$ are set per block (e.g., as geometric sequences or learned parameters).

This construction can also be seen as the composition of two commuting rotation blocks:

$$\mathrm{Rot}(x, y) = \mathrm{Rot}_x(x) \cdot \mathrm{Rot}_y(y),$$

which ensures that the embedding is translation-invariant in 2D:

$$\mathrm{Rot}(x_i, y_i)^\top \mathrm{Rot}(x_j, y_j) = \mathrm{Rot}(x_j - x_i, y_j - y_i).$$

In practical variants (Schenck et al., 4 Feb 2025, Zhao et al., 19 Mar 2025), the $d$-dimensional embedding is split evenly into $x$ and $y$ blocks, each block rotated independently by the respective spatial coordinate multiplied by a frequency or a global scaling parameter. The attention score between two tokens is then a function only of their relative displacement, preserving the essential property of RoPE.

2. Practical Implementation

AS2DRoPE inserts the 2D rotary operation directly into the attention mechanism of Multi-Head Self-Attention (MHSA) layers. For each query/key vector:

1. Project to embedding dimension $d$.
2. Partition into $d/2$ blocks (size 2).
3. For block $n$, apply the SO(2) rotation defined by the token’s 2D coordinates.

Efficient implementation leverages inplace block-wise rotations, with $O