Multiplicative GRAPE: Lie Group Position Encoding

• Multiplicative GRAPE is a Lie group-based positional encoding that leverages multiplicative rotations to provide norm-preserving, relative, and compositional mappings.
• It employs a rank-2 skew generator and closed-form Rodrigues-type solutions to enforce exact relative attention laws with efficient O(d) complexity.
• Empirical results demonstrate stable training and improved generalization in long-context models compared to traditional block-diagonal approaches.

Multiplicative GRAPE (GRAPE-M) is a positional encoding mechanism grounded in Lie group actions, specifically utilizing multiplicative rotations from the special orthogonal group $\mathrm{SO}(d)$. It provides a rigorous, parameterized family of norm-preserving, relative, and compositional position mappings for attention-based neural architectures. GRAPE-M generalizes and subsumes the Rotary Position Embedding (RoPE) by extending beyond canonical block-diagonal rotations to include learned multi-subspace rotations and compact mixtures of non-commuting group actions with tractable computational overhead. It is a central component of the GRAPE (Group RepresentAtional Position Encoding) framework for positional geometry in long-context models, admitting closed-form solutions and strict relative attention laws (Zhang et al., 8 Dec 2025).

1. Mathematical Construction: Rank-2 Skew Generator

At the core of Multiplicative GRAPE is a rank-2 skew-symmetric generator $L(a,b) = a b^\top - b a^\top \in \mathfrak{so}(d)$, where $a, b \in \mathbb{R}^d$. For $a, b$ fixed:

• Compute $\alpha = \|a\|^2$, $\beta = \|b\|^2$, $\gamma = a^\top b$
• Define $\Delta = \alpha\beta - \gamma^2 \geq 0$, $s = \sqrt{\Delta}$
• $L$ acts nontrivially only on $L(a,b) = a b^\top - b a^\top \in \mathfrak{so}(d)$0 and satisfies $L(a,b) = a b^\top - b a^\top \in \mathfrak{so}(d)$1, where $L(a,b) = a b^\top - b a^\top \in \mathfrak{so}(d)$2 is the orthogonal projector onto $L(a,b) = a b^\top - b a^\top \in \mathfrak{so}(d)$3

The spectrum of $L(a,b) = a b^\top - b a^\top \in \mathfrak{so}(d)$4 is $L(a,b) = a b^\top - b a^\top \in \mathfrak{so}(d)$5, ensuring that $L(a,b) = a b^\top - b a^\top \in \mathfrak{so}(d)$6 is a rotation in $L(a,b) = a b^\top - b a^\top \in \mathfrak{so}(d)$7 for frequency $L(a,b) = a b^\top - b a^\top \in \mathfrak{so}(d)$8 and token index $L(a,b) = a b^\top - b a^\top \in \mathfrak{so}(d)$9 (or $a, b \in \mathbb{R}^d$0 for continuous positions).

2. One-Parameter Subgroup, Relative Law, and Closed-Form Exponential

GRAPE-M defines the multiplicative positional map as $a, b \in \mathbb{R}^d$1. The exponential map generates a one-parameter subgroup:

• $a, b \in \mathbb{R}^d$2
• $a, b \in \mathbb{R}^d$3
• $a, b \in \mathbb{R}^d$4

Applied to attention, if $a, b \in \mathbb{R}^d$5 and $a, b \in \mathbb{R}^d$6 are query and key embeddings at positions $a, b \in \mathbb{R}^d$7 and $a, b \in \mathbb{R}^d$8, GRAPE-M enforces

$a, b \in \mathbb{R}^d$9

giving strictly relative positional dependence.

The matrix exponential yields a closed-form “Rodrigues”-type solution:

$a, b$0

This formula achieves a planar rotation by $a, b$1 in $a, b$2 and identity outside $a, b$3.

3. Recovery of RoPE and the Canonical Case

RoPE arises as a special case where $a, b$4, with $a, b$5 the canonical block-diagonal complex structure ($a, b$6; 90° blocks). Taking $a, b$7 and $a, b$8 such that $a, b$9, $\alpha = \|a\|^2$0, and $\alpha = \|a\|^2$1, the generator satisfies $\alpha = \|a\|^2$2 and $\alpha = \|a\|^2$3. The resulting position map is

$\alpha = \|a\|^2$4

For $\alpha = \|a\|^2$5 disjoint planes (canonical coordinate pairs) and per-plane frequencies $\alpha = \|a\|^2$6,

$\alpha = \|a\|^2$7

and $\alpha = \|a\|^2$8 is block-diagonal. Log-uniform choice $\alpha = \|a\|^2$9 exactly recovers the RoPE spectrum.

4. Extensions: Learned Commuting Subspaces and Non-Commuting Mixtures

Learned Commuting Subspaces

Multiplicative GRAPE naturally extends to learned multi-plane actions:

• Let $\beta = \|b\|^2$0 (orthogonal basis), select planes $\beta = \|b\|^2$1 via $\beta = \|b\|^2$2
• $\beta = \|b\|^2$3 (mutually orthogonal, $\beta = \|b\|^2$4)
• Aggregate $\beta = \|b\|^2$5, $\beta = \|b\|^2$6

This allows learning per-head or per-plane spectra while maintaining $\beta = \|b\|^2$7 computational and memory cost per head. Each token's features are rotated independently in $\beta = \|b\|^2$8 planes.

Compact Non-Commuting Mixtures

To capture cross-subspace structure, GRAPE-M provides compact mixtures of $\beta = \|b\|^2$9 non-commuting rank-2 generators:

• $\gamma = a^\top b$0 (span of all $\gamma = a^\top b$1 has dimension $\gamma = a^\top b$2)
• Project features into the $\gamma = a^\top b$3-dimensional subspace using an orthonormal basis $\gamma = a^\top b$4
• Compress $\gamma = a^\top b$5 to $\gamma = a^\top b$6, compute real-Schur decomposition $\gamma = a^\top b$7
• Rotate in each $\gamma = a^\top b$8-plane by $\gamma = a^\top b$9

The operation requires $\Delta = \alpha\beta - \gamma^2 \geq 0$0 cost per head and enables non-commuting, coupled feature mixing beyond block-diagonal structure.

5. Pseudocode Overview

Two core algorithms underpin efficient implementation:

Algorithm	Description	Complexity
Algorithm 1	Commuting Multi-Subspace GRAPE-M: apply $\Delta = \alpha\beta - \gamma^2 \geq 0$1 planar rotations via orthogonal $\Delta = \alpha\beta - \gamma^2 \geq 0$2	$\Delta = \alpha\beta - \gamma^2 \geq 0$3 per head
Algorithm 2	Fast Non-Commuting via Schur: compress to $\Delta = \alpha\beta - \gamma^2 \geq 0$4-dim, $\Delta = \alpha\beta - \gamma^2 \geq 0$5, planar rotations in mixed subspaces	$\Delta = \alpha\beta - \gamma^2 \geq 0$6 per head

Algorithmic Steps:

1. For commuting case, rotate each coordinate pair by $\Delta = \alpha\beta - \gamma^2 \geq 0$7, $\Delta = \alpha\beta - \gamma^2 \geq 0$8, as parameterized by position and frequency.
2. For non-commuting case, project to a lower-dimensional subspace, apply multiple coupled planar rotations, and project back.

6. Initialization, Stability, and Computational Considerations

• Each plane generator is gauge-fixed with $\Delta = \alpha\beta - \gamma^2 \geq 0$9, $s = \sqrt{\Delta}$0; scale is absorbed into $s = \sqrt{\Delta}$1.
• Frequencies $s = \sqrt{\Delta}$2 or $s = \sqrt{\Delta}$3 can be fixed (e.g., log-uniform) or learned per head/plane.
• The closed-form uses $s = \sqrt{\Delta}$4, $s = \sqrt{\Delta}$5 with Taylor expansions guarding $s = \sqrt{\Delta}$6 for numerical stability.
• Differentiation with respect to $s = \sqrt{\Delta}$7, $s = \sqrt{\Delta}$8, $s = \sqrt{\Delta}$9 yields analytic, stable gradients.
• Complexity: GRAPE-M achieves $L$0 time, $L$1 memory, and $L$2 parameters per head (contrasted with $L$3 and $L$4 for full-matrix exponentials).
• Compared to block-diagonal RoPE, GRAPE-M enables richer feature coupling without increasing leading-order cost.

7. Empirical Performance and Model Integration

In experiments on language modeling (nanoGPT/LLaMA base, FineWeb-Edu 100B), Multiplicative GRAPE shows:

• Stable training curves by contrast with oscillations observed in RoPE.
• Validation loss on par or improved relative to RoPE for both medium (355M) and large (770M) models.
• In downstream zero-shot evaluation tasks (ARC, BoolQ, Hellaswag, OBQA, PIQA, WinoGrande, SciQ), both GRAPE-A (additive) and GRAPE-M outperform RoPE and the Forgetting Transformer (FoX) in average accuracy (see Tables A.1–A.2 of (Zhang et al., 8 Dec 2025)).

A plausible implication is that the additional expressivity and strict group-theoretic structure of Multiplicative GRAPE can enhance both training stability and generalization in long-context models, while preserving exact relative positional laws and $L$5 efficiency.