---
title: 'LieRE 3D: Rotational Positional Encoding'
url: https://www.emergentmind.com/topics/liere-3d
type: topic
---

# LieRE 3D: Rotational Positional Encoding

LieRE 3D (Lie Rotational Positional Encodings in Three Dimensions) is a positional encoding scheme that generalizes the rotary position encoding (RoPE) mechanism to noncommutative Lie group rotations in three dimensions, enabling transformer models to effectively encode relative positions in 3D grid-structured data. LieRE is built upon the mathematical structure of SO(3), the Lie group of 3D rotations, and leverages its associated Lie algebra for high-capacity, geometry-aware encodings that are crucial for 3D vision, volumetric imaging, and video representation tasks [2406.10322].

## 1. Limitations of Previous Positional Encoding Mechanisms

Traditional rotary positional encoding (RoPE) injects relative position by applying block-diagonal 2D rotations to attention keys and queries. RoPE is effective for 1D sequence modeling, enabling attention scores to depend only on token offset $j - i$. However, key limitations arise in higher dimensions:

- **Dimensionality Constraint**: RoPE inherently operates on 1D sequences; naive extensions to 2D/3D (independent block rotations) impose a commuting structure, which fails to capture path-dependent relationships. In 3D data, this prevents distinction between different routes between grid points (e.g., “move up then right” vs “right then up”).
- **Limited Representational Expressivity**: The block structure of 2D RoPE cannot represent general 3D rotations or encode orientation-dependent cues essential for spatial or spatiotemporal data.
- **Loss of Spatial Locality**: Flattening multidimensional grids into sequences discards natural adjacency and undermines geometric inductive biases [2406.10322].

## 2. Foundations in Lie Groups and Lie Algebras

LieRE leverages the properties of the Lie group SO(3) and its Lie algebra so(3):

- **SO(3)**: The group of all $3 \times 3$ real orthogonal matrices with determinant $+1$, representing all proper rotations in $\mathbb{R}^3$.
- **so(3)**: The Lie algebra of SO(3), consisting of all $3 \times 3$ real skew-symmetric matrices. Any $\omega \in \mathbb{R}^3$ is mapped to a skew-symmetric matrix:
  $$
  \Omega = \begin{pmatrix}
   0 & -\omega_3 & \omega_2 \\
   \omega_3 & 0 & -\omega_1 \\
   -\omega_2 & \omega_1 & 0
  \end{pmatrix}
  $$
- **Exponential and Logarithm Maps**: $\exp: \mathrm{so}(3) \to \mathrm{SO}(3)$ provides a local diffeomorphism; for small $X, Y \in \mathrm{so}(3)$, $\exp(X)\exp(Y) \approx \exp(X+Y)$, supporting relative positional encoding.
- **Axis-Angle Parameterization**: Any rotation in SO(3) can be parameterized by a vector $\theta \in \mathbb{R}^3$, with the Rodrigues formula providing a closed-form matrix exponential:
  $$
  R(\theta) = I + \frac{\sin\|\theta\|}{\|\theta\|} \Omega(\theta) + \frac{1 - \cos\|\theta\|}{\|\theta\|^2} \Omega(\theta)^2
  $$
  where $\Omega(\theta)$ is the skew-symmetric matrix generated by $\theta$.

## 3. Construction of LieRE 3D Positional Encoding

LieRE maps each token’s 3D coordinate $p_i = (x_i, y_i, z_i) \in \mathbb{Z}^3$ to the Lie algebra so(3) using a learned linear generator:
$$
A(p_i) = x_i A_x + y_i A_y + z_i A_z
$$
where $A_x$, $A_y$, $A_z$ are learned $3 \times 3$ skew-symmetric matrices.

Each $A(p_i)$ is exponentiated to obtain a rotation matrix:
$$
R_i = \exp(A(p_i)) \in \mathrm{SO}(3)
$$

The rotation $R_i$ is then applied to the query and key representations:
$$
Q'_i = R_i Q_i,\quad K'_i = R_i K_i
$$

When attention is computed between tokens $i$ and $j$:
$$
(Q'_i)^\top K'_j = Q_i^\top R_i^\top R_j K_j = Q_i^\top \exp(A(p_j) - A(p_i)) K_j
$$
The inner product is thus a function only of their position difference $p_j - p_i$, ensuring strict translational equivariance in 3D space.

## 4. Integration into Transformer Architectures

Within a multi-head self-attention module, LieRE applies the encoding as follows:

- For $N$ tokens with 3D positions $p_1, \dots, p_N$,
  - Compute $R_s = \exp(A(p_s))$ for each source token,
  - Compute $R_t = \exp(A(p_t))$ for each target token.
  
- Rotated queries and keys are used in the attention computation:
  $$
  \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\left((R_s Q) (R_t K)^\top / \sqrt{d}\right) V
  $$

- GPU implementation uses vectorized, closed-form Rodrigues computation for all token positions in batch. For a batch of 64 tokens (12-layer ViT-B, 12 heads), memory requirements are approximately 40 GB for a single forward/backward pass [2406.10322].

## 5. Empirical Evaluation in 3D Vision and Temporal Tasks

LieRE was empirically evaluated on volumetric and spatiotemporal benchmarks:

- **Datasets**: UCF101 (3D video, 101 classes), RSNA (3D CT, binary brain hemorrhage detection).
- **Baseline Comparisons**: Absolute Position Embedding (APE), RoPE-Mixed (separate RoPE on spatial and temporal axes), and LieRE (full SO(3)).
- **Results—Classification Accuracy**:

  | Method              | UCF101 (%) | RSNA (%) |
  |---------------------|:----------:|:--------:|
  | Absolute Pos. Emb.  |   44.4     |  80.7    |
  | RoPE-Mixed          |   48.6     |  81.9    |
  | LieRE (SO(3))       |   51.1     |  82.7    |

  LieRE delivered improvements of +6.7 percentage points over APE and +2.5 over RoPE-Mixed on UCF101; on RSNA, +2.0 and +0.8 points, respectively [2406.10322].

- **Robustness and Inductive Bias**: Under random patch shuffling at inference, LieRE suffered a 36.9 percentage point drop in accuracy, compared to 24.7 for RoPE-Mixed and 0.1 for APE, indicating stronger reliance on precise 3D relative cues.

## 6. Implementation Considerations and Computational Efficiency

- **Closed-Form Exponentials**: All matrix exponentials are explicit via Rodrigues’ formula, eliminating the need for general-purpose matrix-exponential routines.
- **Batching**: All position-to-generator mappings and exponentials are vectorized per batch, with negligible compute overhead for standard Vision Transformer (ViT) memory capacities.
- **Hyperparameters**: Cosine learning rate decay from 1e-4, Adam optimizer ($\beta_1=0.9, \beta_2=0.999, \epsilon=1\mathrm{e}{-8}$), patch size 4×16×16 on 32×224×224 video inputs, dropout 0.1, and no positional-encoding–specific tuning.
- **Scalability**: Generator matrix block size (from $2\times2$ to $64\times64$) monotonically increases accuracy on UCF101, with even $4\times4$ blocks capturing most of the benefit.

## 7. Significance, Impact, and Potential Extensions

LieRE establishes a unifying, group-theoretic framework for 3D positional encoding in attention-based architectures, offering several critical capabilities:

- **Strict Relative Position Dependence**: Attention depends only on $p_j - p_i$ in $\mathbb{Z}^3$, enforcing homogeneous locality priors in 3D grids.
- **Increased Geometric Expressivity**: The noncommutative nature of SO(3) permits modeling of path-dependent transformations and more faithful spatial cues.
- **Empirical Gains**: Outperforms both absolute and commutative block-diagonal (2D) positional encodings in 3D classification accuracy.
- **Practicality**: Computationally efficient, minimal impact on standard ViT memory and runtime budgets, and readily implementable via closed-form operations.
- **Scalability Knob**: Generator matrix block size offers a tunable trade-off between expressivity and compute.

A plausible implication is that the adoption of LieRE in video, volumetric 3D, or multi-modal transformer architectures can replace less expressive positional encoding schemes to achieve higher fidelity in geometric structure modeling and improved task generalization [2406.10322].

Source: https://www.emergentmind.com/topics/liere-3d