GRAPE: Group Representational Position Encoding

• GRAPE is a unified group-theoretic framework for positional encoding in Transformers that combines multiplicative rotations with additive logit biases.
• It enforces exact relative positional laws using Lie groups like SO(d) and GL(d+1), enabling efficient streaming and batch decoding without architectural modifications.
• GRAPE subsumes existing schemes such as RoPE, ALiBi, and FoX, offering improved performance in long-context language and speech models with minimal overhead.

Group Representational Position Encoding (GRAPE) is a unified group-theoretic framework for positional encoding in Transformers. It brings together two principal categories of mechanisms for encoding position: multiplicative rotations rooted in special orthogonal groups and additive logit biases derived from unipotent actions in the general linear group. GRAPE provides a generalized algebraic approach for specifying positional geometry in long-context neural models and subsumes prominent schemes such as Rotary Position Embeddings (RoPE), ALiBi, and the Forgetting Transformer (FoX), while supporting efficient streaming and batch inference without architectural modification (Zhang et al., 8 Dec 2025, Tong et al., 22 May 2025).

1. Group-Theoretic Foundations of GRAPE

GRAPE posits that token positions are formalized as actions of Lie group elements on the representations entering the attention mechanism. The core principle is that the group-valued function $G: \mathbb{Z} \rightarrow G$ (where $G$ is locally either $\mathrm{SO}(d)$ or $\mathrm{GL}(d+1)$) satisfies the one-parameter subgroup law:

$$G(n+m) = G(n)G(m), \quad G(0) = I, \quad G(-n) = G(n)^{-1}.$$

This exact relative law ensures the attention logits depend only on relative displacement $(j-i)$, guaranteeing that the model's operations maintain strict equivariance to translation along the sequence.

GRAPE comprises two foundational constructions:

• Multiplicative GRAPE (GRAPE-M): $G(n) \in \mathrm{SO}(d)$ with $G(n) = \exp(n\,\omega\,L)$ for skew-symmetric $L \in \mathfrak{so}(d)$.
• Additive GRAPE (GRAPE-A): $G(n) \in \mathrm{GL}(d+1)$, realized as $G(n) = \exp(n\,\omega\,A) = I + n\,\omega\,A$ for nilpotent $A$ with $A^2 = 0$.

These mechanisms strictly enforce relativity, provide computational efficiency, and can be homogeneously combined for richer positional structure.

2. Multiplicative Mechanism: Lie Rotations and Extensions

Multiplicative GRAPE implements position encodings as rotations within the feature space:

$G(n) = \exp(n\,\omega\,L) \in \mathrm{SO}(d),$

with $L^\top = -L$ (skew-symmetric). A minimal practical construction employs a rank-2 generator $L = a b^\top - b a^\top$ for $a,b \in \mathbb{R}^d$, producing rotations confined to the plane $U = \operatorname{span}\{a,b\}$. The matrix exponential can be expressed in closed form:

$$\exp(L) = I + \frac{\sin s}{s}L + \frac{1 - \cos s}{s^2}L^2,$$

where $s = \sqrt{\,\|a\|^2\|b\|^2 - (a^\top b)^2\,}$. This yields norm-preserving and relative encodings.

RoPE Recovery: When $d$ is even, $L$ is constructed as a sum over $d/2$ mutually orthogonal planes, replicating the block-diagonal structure of RoPE, with frequencies determined by a log-uniform spectral law (Zhang et al., 8 Dec 2025).

Extensions:

• Learned commuting subspaces allow the rotation planes to be chosen via an arbitrary orthogonal basis, preserving efficient $O(d)$ computation per head.
• Non-commuting mixtures compress the rotation action onto an $r$-dimensional subspace, parameterizing $L_r \in \mathfrak{so}(r)$ and using a lift $E \in \mathbb{R}^{d \times r}$, facilitating richer geometry and cross-subspace coupling at $O(r d)$ cost per head.

Relative Relativity Guarantee: For any queries $q_i$ and keys $k_j$, one computes

$\tilde{q}_i = G(i)q_i,\ \tilde{k}_j = G(j)k_j,$

and attention scores $q_i^\top G(j-i)k_j$ depend only on $j-i$.

3. Additive Logit Biases: Unipotent Group Actions

Additive GRAPE represents a positional bias as an affine transformation operating on homogeneous coordinates. For $A \in \mathbb{R}^{(d+1) \times (d+1)}$ nilpotent ($A^2 = 0$), the group action is

$G_{\text{add}}(n) = I + n\,\omega\,A.$

When acting on $[x;1] \in \mathbb{R}^{d+1}$, this yields $[x + n\,\omega\,u;\ 1]$. Applying this to augmented queries and keys with group-inverse transpose, additive scores become exact functions of offsets.

ALiBi Recovery: By parameterizing with special generators $A_h^\top = \beta_h e_{d+1}e_{d+2}^\top$ and dimension extension ($D = d+2$), the formulation yields the ALiBi linear bias:

$q^\top k - \beta_h(j-i),$

where $\beta_h$ is the head-specific slope (Zhang et al., 8 Dec 2025).

Forgetting Transformer (FoX): The mechanism generalizes to the path-integral specialization for forgetting via time-varying, rank-1 nilpotent elements, exactly recapitulating the FoX recency bias when group products are accumulated and streamed.

4. Unified Design Space, Special Cases, and Practical Framework

GRAPE establishes a unified framework where both rotational and additive encodings can be employed jointly or separately. The mechanisms obey the same relative law and can be realized as block upper-triangular operators in $\mathrm{GL}(d+1)$:

$$\hat{G}(m) = \begin{bmatrix} \exp(m\,L) & m\,\omega\,u \ 0^\top & 1 \ \end{bmatrix},$$

allowing both rotation and bias in a single group operator. These constructions enable the recovery of existing popular approaches:

• RoPE as commuting rank-2 exponentials in $\mathrm{SO}(d)$.
• ALiBi as rank-1 unipotent actions in $\mathrm{GL}(d+2)$.
• FoX via the additive, path-integral specialization (Zhang et al., 8 Dec 2025).

GRAPE simultaneously supports streaming and batch decoding, preserving linear scaling of KV caches and constant O(1) recomputation per new token during autoregressive inference.

5. Streaming Adaptation and Batch–Streaming Consistency

Pretrained LLMs are often batch-oriented and pose challenges when deployed in streaming (incremental) scenarios due to position and attention mismatches (Tong et al., 22 May 2025). GRAPE enables seamless streaming adaptation as follows:

• The token stream is segmented into groups (e.g., source segment, target segment), with each assigned contiguous, never-changing blockwise position indices.
• The same positional encoding (absolute or rotary) is applied; attention masking precludes invalid input-output attention.
• By fixing group offsets (e.g., $\phi_0$, $\phi_1$), absolute relative differences split into a learnable, per-group offset plus within-group relative order.

Empirical ablation demonstrates:

• Removing group positional encoding degrades BLEU by $>10$; removing a single side cuts BLEU by $4$–$6$ points.
• Re-encoding of positions offers $<0.3$ BLEU improvement compared to group-based block assignment, indicating that no systematic reassignment of positions or re-encoding is required, yielding up to $11\times$ throughput improvements over naïve full re-encoding (Tong et al., 22 May 2025).
• Varying group offsets across a wide range yields negligible change in WER and BLEU, provided offsets remain within the pre-training context window.

6. Computational, Architectural, and Applicability Considerations

GRAPE minimizes computational and implementation overhead:

• Commuting GRAPE-M (RoPE-style) and learned commuting bases incur $O(d)$ cost per head.
• Non-commuting (Schur-mode) subspaces with dimension $r$ offer $O(r d)$ complexity, $r \ll d$.
• Additive GRAPE and ALiBi-like mechanisms require $O(d)$ per head, with bias-only cases applying $O(1)$.

Parameter counts scale as $O(d)$ (multiplicative or additive, per head) or $O(r d)$ where applicable. GRAPE maintains cacheability and streaming compatibility equivalent to RoPE/ALiBi, supporting O(1) streaming updates.

Domain applicability includes long-context language modeling, vision transformers with multi-dimensional rotary maps, and multimodal or context-adaptive warping scenarios. GRAPE’s exact relative law and design generality make it suitable for scenarios demanding robust recency and forgetting mechanisms, cross-modal alignment, or rapid batch-to-stream transitions (Zhang et al., 8 Dec 2025, Tong et al., 22 May 2025).

7. Comparative Performance and Experimental Findings

Empirical results on streaming translation (IWSLT-17 En$\to$Fr, En$\to$De) and ASR (LibriSpeech) confirm group-based GRAPE outperforms specialized streaming architectures (SimulMask, DST for MT; CAAT, Wav2Vec-S for ASR) across accuracy-latency trade-offs. Removal of blockwise positional encoding incurs marked accuracy drops, while proper group assignment preserves near-batch performance. Performance is near invariant to the choice of group offset $\phi$ within the recommended range; absolute ordering across more than two segments or extreme group offsets outside pre-training context windows may require further tuning (Tong et al., 22 May 2025).

Mode	BLEU (MT, k=7)	WER (ASR)	Throughput Gain
Interleaved-streaming	30.9	3.3–6.0	1.0×
GRAPE (no re-encode)	32.1–33.1	3.3–6.0	up to 11×
Full re-encode	+0.3 BLEU	N/A	O(1×), slowest

This table summarizes representative ablation and accuracy results (see (Tong et al., 22 May 2025)).

GRAPE supplies a principled algebraic approach for both rotational and additive positional encodings, reconciling batch and streaming modes under a unified mathematical law with minimal computational and implementation complexity, while generalizing and subsuming prevailing schemes.