---
title: Hyperbolic Rotary Positional Encoding (HoPE)
url: https://www.emergentmind.com/topics/hyperbolic-rotary-positional-encoding-hope
type: topic
---

# Hyperbolic Rotary Positional Encoding (HoPE)

Hyperbolic Rotary Positional Encoding (HoPE) is a geometric generalization of rotary positional encodings designed for stable and efficient modeling of long-range dependencies in transformers. HoPE replaces the periodic Euclidean rotations of standard rotary positional encoding (RoPE) with Lorentz boosts parameterized by hyperbolic functions, inducing monotonic exponential decay of attention weights as token distances increase. This formulation resolves the oscillatory and resonance effects pervasive in traditional RoPE, achieving improved extrapolation and performance in extended-sequence transformers for language and representation learning tasks [2509.05218][2509.16629].

## 1. Theoretical Foundations

HoPE arises by reinterpreting the geometric mechanism underlying positional encodings in self-attention. In RoPE, token vectors are decomposed into 2D subspaces, each rotated by an angle proportional to token position using block-diagonal matrices $\rho_E(p\,g_i)$ with frequencies $g_i$. The resulting dot-product attention kernel for tokens at positions $m$ and $n$ is expressed as:
\[
\langle q_m, k_n \rangle_{\text{RoPE}} = \sum_{i=1}^{d/2} (q_m^{(i)})^\top \rho_E(g_i)^{n-m} k_n^{(i)}
\]
where the rotation is periodic in $n-m$, causing oscillatory attention weights for large token distances due to the trigonometric nature of $\cos$ and $\sin$ functions.

HoPE replaces these planar (Euclidean) rotations with Lorentz transformations in hyperbolic space. The canonical 2D Lorentz boost (the hyperbolic analog of a rotation) is given by:
\[
B_i(\Delta p) =
\begin{pmatrix}
\cosh(\Delta p \theta_i) & \sinh(\Delta p \theta_i) \\
\sinh(\Delta p \theta_i) & \cosh(\Delta p \theta_i)
\end{pmatrix}
\]
where $\theta_i$ parameterizes per-block curvature, and $\Delta p = n-m$ is the positional offset. The full HoPE operator over $d$ dimensions is:
\[
R^d_{\rm h}(\Delta p) = \gamma(\Delta p) \bigoplus_{i=1}^{d/2} B_i(\Delta p)
\]
where the gamma factor $\gamma(\Delta p) = e^{-|\Delta p|\,\theta'}$ provides additional exponential damping.

Crucially, as the curvature vanishes ($\theta_i,\,\theta' \rightarrow 0$), the hyperbolic formulation reverts to the Euclidean (RoPE) case, as shown by the limiting behavior of hyperbolic trigonometric functions:
\[
\cosh(\sqrt{K}\,x) \rightarrow 1 + \tfrac12 K x^2, \quad \sinh(\sqrt{K}\,x) \rightarrow K x
\]
for $K \to 0$. Thus, RoPE is a special case of HoPE with zero curvature [2509.05218].

## 2. Mathematical Properties 

The defining property of HoPE is the strict monotonicity of the attention decay. For a single 2D subspace, the unnormalized dot-product between the query at $m$ and key at $n$ is:
\[
(q_m^{(i)})^\top [e^{-m\theta'} B_i(\theta_i, m)]^\top [e^{+n\theta'} B'_i(\theta_i, n)] k_n^{(i)}
\]
Combining transformations yields:
\[
\langle f_q(m), f_k(n) \rangle \propto e^{-|m-n|(\theta' - \theta_i)}
\]
resulting in exponential decay in $\lvert m-n \rvert$ whenever $\theta' > \max_i \theta_i$. This structure eliminates the oscillatory “resonances” present in RoPE, in which attention periodically recurs for increasing token separations.

Additionally, the block-diagonal construction in each embedding plane allows HoPE to retain the efficient complexity scaling of standard rotary encodings without additional dependence on sequence length. The operator is implemented as a direct sum over $d/2$ two-dimensional Lorentz blocks, each with learnable or fixed $\theta_i$, with a global damping $\theta'$. For the multi-dimensional case, the HoPE operator is:
\[
R_{\rm h}^d(\Delta p) = e^{-|\Delta p| \theta'}\, \text{diag}(B_1(\Delta p), \ldots, B_{d/2}(\Delta p))
\]

## 3. Implementation Details

HoPE requires minimal changes to transformer workflows. Input token representations are split into 2D chunks; for each chunk, the Lorentz boost with parameter $\theta_i$ is computed at a position-dependent argument. Query and key chunks are rotated using $\cosh$/$\sinh$-parametrized matrices, with exponential damping applied elementwise. Pseudocode (see [2509.05218]) illustrates the algorithmic loop over $d/2$ subspaces:

- For queries (at position $p_q$): apply $B_i(\theta_i, p_q)$ with $e^{-p_q\theta'}$ multiplicative damping.
- For keys (at position $p_k$): apply $B'_i(\theta_i, p_k)$ with $e^{+p_k\theta'}$ (i.e., reciprocal) damping.
- The final dot-product incorporates both relative Lorentz boosts and overall distance-dependent decay.

Numerical stability for large $\phi$ is addressed using numerically stable library routines for hyperbolic functions or clamping input ranges.

HoPE introduces only two extra parameter vectors of length $d/2$: $\{\theta_i\}$ (curvatures) and $\theta'$ (damping), and does not require additional state per sequence or token [2509.05218].

## 4. Empirical Evaluation

HoPE has been evaluated extensively on standard long-context benchmarks for language modeling and downstream tasks [2509.05218]. 

### Zero-Shot Perplexity

On PG19 and arXiv, HoPE demonstrates significantly lower perplexity at long sequence lengths compared to RoPE and Alibi, especially as token distances increase, indicating improved robustness and generalization for untrained context sizes.

| Method    | 1024 | 2048 | 3072 | 4096 | 5120 | 6144 |
|-----------|------|------|------|------|------|------|
| RoPE      |12.82 |25.80 |56.28 |88.59 |116.63|144.13|
| HoPE      |13.35 |16.46 |35.07 |60.03 |85.94 |110.02|
| Alibi     |11.95 |25.11 |52.54 |79.04 |107.59|132.80|

### Downstream Tasks

On SCROLLS, covering QA, NLI, and summarization (sequence length up to 8192), HoPE outperforms or matches sinusoidal, RoPE, and Alibi encodings across all major metrics. Ablation studies confirm that the damping factor $\theta'$ is influential in controlling the rate of attention decay; over-damping may harm capacity, while under-damping reduces long-range selectivity.

## 5. Geometric and Causal Extensions

The hyperbolic formalism underlying HoPE extends naturally to other forms of representation, including causality-informed and non-sequential feature sets. In CAPE ("Causality-Induced Positional Encoding") [2509.16629], feature graphs inferred by structural equation modeling are embedded as points in hyperbolic space (using the hyperboloid/Lorentz model), with rotary angles derived from hyperbolic distances. This associates strength of causal relationships (inverse hyperbolic separation) and specificity (radial position) to attention weighting, further leveraging the monotonicity and flexibility of hyperbolic encodings.

In such settings, the self-attention computation takes the form:
\[
\mathrm{Attention}(Q, K, V) = \mathrm{softmax} \left( \frac{1}{\sqrt{d}} (Q \circ R(\varphi)) (K \circ R(\varphi))^\top \right) V
\]
where rotary matrices $R(\varphi_{v})$ are derived from hyperbolic embeddings of the underlying causal DAG. This generalizes HoPE beyond sequential data to arbitrary relational or graph-structured features.

## 6. Strengths, Limitations, and Future Directions

HoPE's strengths include:

- Monotonic, bias-free exponential decay of attention weights as a function of token/feature separation.
- Theoretical grounding in the Lorentz group, unifying and extending prior Euclidean methods.
- Efficient implementation matching RoPE’s computational cost.
- Robustness against resonance and oscillatory failure modes.

Limitations:

- Choice of curvature and damping parameters ($\theta_i$, $\theta'$) must be tuned for each application; improper calibration can under- or over-damp long-range attention.
- Empirical validation to date is limited to textual modeling, with cross-modal and multimodal generalization remaining to be systematically verified [2509.05218].

Potential extensions include learnable curvature per transformer head, incorporation into encoder-decoder and retrieval-augmented architectures, integration with alternative hyperbolic models (Poincaré disk or upper half-plane), and hybridization with sparse attention patterns for extreme sequence lengths.

## 7. Connections to Related Methods

HoPE generalizes the following prior methodologies:

- Rotary Positional Encoding (RoPE), recoverable as the curvature $\rightarrow 0$ limit of the Lorentz-boost formalism.
- Linear or absolute encodings (e.g., sinusoidal), improved by HoPE's geometric damping and stable extrapolation.
- Causality-aware hyperbolic embeddings, as in CAPE, which systematically encode graph relationships and feature hierarchies in hyperbolic space and inject them via rotary attention mechanisms into transformers [2509.16629].

Both theoretical and empirical analyses demonstrate that HoPE achieves a smoother, geometry-consistent decay of attention, supporting improved length extrapolation and modeling capacity for long-sequence and graph-structured data domains.

Source: https://www.emergentmind.com/topics/hyperbolic-rotary-positional-encoding-hope