---
title: 'HyperHELM: Hyperbolic mRNA Modeling'
url: https://www.emergentmind.com/topics/hyperhelm
type: topic
---

# HyperHELM: Hyperbolic mRNA Modeling

HyperHELM is a framework for masked language modeling of mRNA sequences that explicitly utilizes hyperbolic geometry to encode biological hierarchies, specifically the codon-to-amino-acid relationship. HyperHELM integrates a standard Euclidean transformer backbone with a lightweight hyperbolic head, enabling hierarchical structure induction through hyperbolic representation and direct masked language prediction in the Poincaré ball. This approach outperforms several Euclidean and hierarchy-agnostic baselines across property prediction and annotation tasks, demonstrating superior out-of-distribution generalization and improved downstream performance in settings where biological encoding hierarchies are prominent [2509.24655].

## 1. Mathematical Framework: The Poincaré Ball Model

HyperHELM operates in the $n$-dimensional Poincaré ball $\mathbb{D}_c^n$ of constant negative curvature $-c$, formalized as:
\[
\mathbb{D}_c^n = \{x \in \mathbb{R}^n : \|x\|^2 < 1/c\}, \quad \mathfrak{g}_c^n(x) = \lambda_x^c I_n, \quad \lambda_x^c = \frac{2}{1 - c\|x\|^2}
\]
Core operations include:
- **Hyperbolic Distance:**  
  \[
  d^c(x, y) = \frac{1}{\sqrt{c}} \cosh^{-1} \left(1 + 2c \frac{\|x-y\|^2}{(1 - c\|x\|^2)(1 - c\|y\|^2)}\right)
  \]
- **Möbius Addition:**  
  \[
  x \oplus_c y = \frac{(1 + 2c\langle x, y\rangle + c\|y\|^2)x + (1-c\|x\|^2)y}{1 + 2c\langle x, y\rangle + c^2\|x\|^2\|y\|^2}
  \]
- **Exponential and Logarithmic Maps at the Origin:**  
  \[
  \exp^c_{\mathbf{0}}(v) = \mathbf{0} \oplus_c \left(\tanh\left(\frac{\sqrt{c} \lambda^c_{\mathbf{0}}\|v\|}{2}\right) \frac{v}{\sqrt{c}\|v\|}\right); \quad \lambda^c_{\mathbf{0}}=2
  \]
  \[
  \log^c_{\mathbf{0}}(x) = \frac{2}{\sqrt{c} \lambda^c_{\mathbf{0}}} \tanh^{-1}(\sqrt{c}\|x\|) \frac{x}{\|x\|}
  \]
These tools enable lifting representations from Euclidean to hyperbolic space and facilitate hyperbolic analogues of neural operators.

## 2. Model Architecture and Hierarchy Encoding

HyperHELM employs a hybrid architecture:
- **Backbone:** A 10-layer transformer with standard Euclidean geometry, hidden size 640 (∼50 M parameters).
- **Hyperbolic Head:** The Euclidean outputs are projected into $\mathbb{D}_c^n$ using the exponential map, then transformed via a hyperbolic linear layer:
  \[
  \mathrm{HypLin}^c(x) = \exp^c_{\mathbf{0}}(W \log^c_{\mathbf{0}}(x) + b)
  \]
Prototypes $\phi(u)$ representing codon leaf nodes are precomputed by embedding the codon–amino-acid tree $T = (V, E)$ into $\mathbb{D}_c^n$ with the distortion-minimizing HS-DTE algorithm. This ensures that codons under the same amino acid map close together in hyperbolic space, while different branches are exponentially separated:
\[
d^c(\phi(u), \phi(v)) \approx d_T(u, v) \quad \forall u,v\in V
\]

## 3. Objective Function in Hyperbolic Space

Masked language modeling masks 15% of codon tokens, predicting them via a prototype classifier in $\mathbb{D}_c^n$. The masked positions $M$ and true tokens $t_i$ at position $i$ are predicted using a softmax over similarity scores $s(z_i, \phi(u))$:
\[
p(t_i = u \mid x) = \frac{\exp(\beta s(z_i, \phi(u)))}{\sum_v \exp(\beta s(z_i, \phi(v)))}
\]
Two choices of similarity function $s$ are evaluated:
- **Negative Hyperbolic Distance:** $s_{\mathrm{dist}}(x, y) = -d^c(x, y)$
- **Entailment-Cone Energy:**  
  \[
  E(x, y) = \max \left\{0, \ \Xi(x, y) - \eta \psi(x)\right\}
  \]
  with $\psi(x)$ as the cone half-aperture and $\Xi(x, y)$ the axis angle.

The pretraining loss is cross-entropy:
\[
\mathcal{L}_{\mathrm{MLM}} = -\frac{1}{|M|} \sum_{i\in M} \log p(t_i \mid x)
\]
An additional hierarchical cross-entropy (HXE) regularizer (weight $\alpha=0.2$) promotes consistency with ancestor hierarchy.

## 4. Training Strategy and Implementation

- **Corpus:** Observed Antibody Sequences (OAS), with codon-level tokenization (vocabulary: 64+special).
- **Hardware:** 1024 sequences/batch, 8×A100 GPUs, 40 epochs.
- **Architecture parameters:** Max input length 444 codons, up to 2048 positions.
- **Optimization:** AdamW, warmup to LR $1 \times 10^{-4}$, cosine decay to $1 \times 10^{-5}$, weight decay $1 \times 10^{-1}$.
- **Hyperbolic Head:** prototype dimension $n_p=128$, curvature $c=1.0$, temperature $\beta=10$.
- **Regularization:** HXE with $\alpha=0.2$.
- **Fine-tuning/Probing:** Fixed backbone with TextCNN (100 channels), varying LR and batch.

## 5. Empirical Evaluation and Quantitative Results

HyperHELM was assessed on ten property-prediction tasks and an antibody region annotation task, spanning datasets such as Ab1, Ab2, mRFP, COVID-19 Vaccine, Drosophila, S. cerevisiae, P. pastoris, Fungal, E. coli, iCodon, and a held-out antibody annotation set.

Comparison baselines included Transformer XE, RNA-FM, SpliceBERT, CodonBERT, and Euclidean HELM. Metrics were Spearman $\rho$ (regression) and accuracy (classification/annotation).

Selected results:
- On 9/10 property prediction tasks, HyperHELM surpassed HELM by ∼10% on average.
- Substantial improvement in D. melanogaster (ρ=0.450 vs. 0.341, +32%) and E. coli (accuracy=50.8% vs. 45.8%, +10.9%).
- On the antibody region annotation task, HyperHELM(dist) achieved 76.48% accuracy vs. HELM's 73.48% (+3%).

| Task / Dataset          | Metric & Value (HyperHELM) | Relative Gain vs. HELM      |
|------------------------|---------------------------|-----------------------------|
| D. melanogaster        | ρ = 0.450                 | +32%                        |
| E. coli                | acc = 50.8%               | +10.9%                      |
| iCodon                 | ρ = 0.539                 | +2.7%                       |
| Ab1                    | ρ = 0.751                 | +5.1%                       |
| Antibody Annotation    | acc = 76.48%              | +3%                         |

## 6. Out-of-Distribution Generalization

HyperHELM exhibited improved generalization under resource-scarce or compositionally shifted conditions:
- **Sequence Length (P. pastoris):** For long sequences (2000–3000 nt), accuracy rose to 0.70 (vs. HELM 0.46).
- **GC Content (COVID-19):** For high GC content (>55%), 0.62 vs. HELM's 0.56; for medium (47–55%), 0.73 vs. 0.64.
- **Codon Usage Bias:** Datasets with stronger bias (lower ENC) saw larger gains, indicating alignment between hyperbolic inductive bias and tree-like codon–amino-acid structure.

A plausible implication is that low-data and distributionally biased regimes particularly benefit from the model's ability to encode hierarchical structure in hyperbolic space.

## 7. Significance and Alignment with Biological Structure

The empirical results confirm that the hyperbolic head, via codon hierarchy–aligned prototypes, provides an effective inductive bias for both in-distribution and out-of-distribution scenarios. HyperHELM’s architecture is compatible with modern hardware (due to Euclidean backbone) while achieving a more faithful representation of the biological hierarchy compared to Euclidean models. This suggests broader relevance for modeling biological sequences whose underlying generating process is hierarchical [2509.24655].

Source: https://www.emergentmind.com/topics/hyperhelm