Group Representational RoPE (GRAPE)

• Group Representational RoPE (GRAPE) is a unified group-theoretic framework for Transformer positional encoding that combines rotary (multiplicative) and bias (additive) mechanisms.
• It leverages matrix Lie group actions to enforce compositionality, invertibility, and exact relative-offset attention, ensuring efficient streaming cache updates.
• GRAPE extends established techniques like RoPE, ALiBi, and FoX by enabling learned, mixed geometric representations for long-context and high-dimensional attention models.

Group Representational Position Encoding (GRAPE) is a unified, group-theoretic framework for positional encoding in Transformer architectures, encompassing both multiplicative (rotary) and additive (bias) mechanisms under the formalism of matrix group actions. GRAPE generalizes and subsumes widely used methods such as Rotary Position Embedding (RoPE), ALiBi, and the Forgetting Transformer (FoX) by systematically interpreting positional maps as Lie group actions—enabling norm-preserving rotary, additive bias, and hybrid positional geometries with strict relative laws, compositionality, and streaming cacheability. This design space supports learned inter-subspace coupling and compact representations, facilitating principled architectural extensions for long-context and high-dimensional attention models (Zhang et al., 8 Dec 2025, Liu et al., 7 Apr 2025).

1. Group-Action Foundations of Positional Encoding

GRAPE formalizes positional encoding as a group action on token representations, with the position index mapped to a one-parameter subgroup $G(n)$ of a chosen matrix Lie group $G$ (e.g., $\mathrm{SO}(d)$ or $\mathrm{GL}(d)$). The key requirements are:

• Compositionality: $G(n+m)=G(n)G(m)$,
• Invertibility: $G(0)=I$, $G(-n)=G(n)^{-1}$,
• Relative law: For positions $s<t$, $$\langle G(s)q_s, G(t)k_t\rangle = q_s^\top G(s)^\top G(t)k_t = q_s^\top G(t-s)k_t$$.

These properties ensure exact relative-offset dependence in the attention kernel and facilitate streaming and efficient cache implementations. The choice of $G$ and its representation defines the specific positional geometry and associated invariance structure (Zhang et al., 8 Dec 2025, Liu et al., 7 Apr 2025).

2. Multiplicative GRAPE in $G$0: Rotary Encodings

Multiplicative GRAPE (GRAPE-M) realizes norm-preserving, rotary encodings by exponentiating skew-symmetric matrices (elements of $G$1):

• Rank-2 generator: Given $G$2, $G$3. Its spectrum is $G$4 with $G$5.
• Exponential map: $G$6, with closed-form—Rodrigues formula:

$G$7

This rotates in the plane $G$8; identity elsewhere.

• Commuting case (MS-GRAPE / RoPE): Standard RoPE corresponds to $G$9 canonical 2D planes, $\mathrm{SO}(d)$0 ($\mathrm{SO}(d)$1), yielding block-diagonal rotations. Log-uniform $\mathrm{SO}(d)$2 recovers the canonical spectrum.
• Learned subspaces: Replacing canonical planes with a learned orthogonal basis $\mathrm{SO}(d)$3 enables cross-subspace frequency coupling under the same computational cost $\mathrm{SO}(d)$4.
• Non-commuting mixtures: By summing $\mathrm{SO}(d)$5 arbitrary rank-2 generators $\mathrm{SO}(d)$6 ($\mathrm{SO}(d)$7), richer rotational dependencies can be captured, incurring $\mathrm{SO}(d)$8 complexity per token for head-specific coupling.

This construction captures and extends the maximal Abelian subalgebra (MASA) interpretation of RoPE, as established in (Liu et al., 7 Apr 2025), generalizing position encodings to learned or mixed geometries.

3. Additive GRAPE in $\mathrm{SO}(d)$9: Bias-based Encodings

Additive GRAPE (GRAPE-A) implements additive, offset-dependent logit biases via (rank-1) unipotent actions in the general linear group:

• Homogeneous lift and generator: Input $\mathrm{GL}(d)$0 is augmented via $\mathrm{GL}(d)$1. For $\mathrm{GL}(d)$2, form $\mathrm{GL}(d)$3, $\mathrm{GL}(d)$4.
• Unipotent action: $\mathrm{GL}(d)$5.
• Relative law and transformation: For queries/keys at positions $\mathrm{GL}(d)$6,

$\mathrm{GL}(d)$7

imparting an additive, offset-linear, key-gated bias. The "+1" is absorbed in softmax bias.

• Special cases:
  • ALiBi is recovered by embedding in $\mathrm{GL}(d)$8 dimensions with $\mathrm{GL}(d)$9, inducing additive logit slope $G(n+m)=G(n)G(m)$0.
  • FoX (Forgetting Transformer) yields position-dependent, endpoint-gated biases via path integrals: $G(n+m)=G(n)G(m)$1.

Streaming cache efficiency is preserved via the relative law and per-head biases.

4. Unified Group-Theoretic Design Space

GRAPE synthesizes both rotary and additive positional mechanisms as special cases of Lie group actions:

• GRAPE-M: $G(n+m)=G(n)G(m)$2 rotary encoding; encompasses RoPE and learned commuting/non-commuting rotary mixtures.
• GRAPE-A: $G(n+m)=G(n)G(m)$3 additive encoding; subsumes ALiBi, FoX, and their generalizations.
• Exact relative-offset attention: Both ensure that the attention kernel depends solely on position offset, supporting streaming cache updates.
• Hybrid and mixed actions: The homogeneous coordinate lift allows for composite transformations:

$G(n+m)=G(n)G(m)$4

with mixed rotary+additive effects.

This framework enables principled exploration across the positional geometry design spectrum, including context-dependent generative parameters, multi-dimensional spatial encodings, and cross-modal applications (Zhang et al., 8 Dec 2025).

5. Implementation, Complexity, and Practical Considerations

Efficient algorithms exploit the group structure and low-rank generator properties:

• Commuting GRAPE-M/RoPE: Fast $G(n+m)=G(n)G(m)$5 implementation via basis transformations and block-2D rotations, requiring no full $G(n+m)=G(n)G(m)$6 storage.
• Non-commuting GRAPE-M: $G(n+m)=G(n)G(m)$7 complexity; exponentiation is confined to a compact $G(n+m)=G(n)G(m)$8-dimensional subspace.
• GRAPE-A: $G(n+m)=G(n)G(m)$9 per-head flops (one saxpy and dot-product).
• Cache strategies: Both families maintain streaming-compatible caches via the relative law—important for long-context or streaming architectures.

Parameterization choices are critical:

• Spectrum selection: Log-uniform rotation frequencies (spectrum of $G(0)=I$0) remain effective; learning frequencies and subspaces per head offers flexibility.
• Plane count: RoPE typically employs $G(0)=I$1; compact mixtures (with $G(0)=I$2) suffice for non-commuting cases without loss of expressiveness.
• Additive rank: Rank-1 for ALiBi; higher rank enables content-dependent, key- or query-gated slopes.

Extensions include context-dependent group actions, multi-axes position functions for vision or structured data, and group actions beyond translations (e.g., rotations in $G(0)=I$3 for 3D data) (Zhang et al., 8 Dec 2025, Liu et al., 7 Apr 2025).

6. Theoretical Context and Relation to Prior Work

GRAPE emerges from a rigorous Lie group algebraic analysis of the relative and reversibility properties required for valid rotary position encodings (Liu et al., 7 Apr 2025). All RoPE variants correspond to Abelian subalgebras (MASA) within $G(0)=I$4, block-diagonalized as orthogonal 2D rotations, optionally after orthogonal basis change to capture inter-dimensional coupling.

Through the unification of rotary and additive paradigms—parametrized by general group representations and low-rank matrix actions—GRAPE provides a systematic, extensible framework for designing positional encodings aligned with Transformer attention invariance and compute requirements. This consolidation enables uniform treatment of established and novel position encoding schemes, offering a foundation for further advances in long-context, structure-aware, and domain-specific Transformer models, as well as seamless integration with learned representations in high-dimensional spaces (Zhang et al., 8 Dec 2025, Liu et al., 7 Apr 2025).