---
title: 'HyenaDNA: Efficient Genomic Foundation Model'
url: https://www.emergentmind.com/topics/hyenadna
type: topic
---

# HyenaDNA: Efficient Genomic Foundation Model

HyenaDNA is a genomic foundation model designed for long-range sequence modeling at single-nucleotide resolution, leveraging implicit-convolution architectures to enable order-of-magnitude scaling beyond attention-based transformers. It provides a universal, efficient, and highly parameter-efficient backbone for DNA and, via adapters, RNA modeling. HyenaDNA and related approaches underpin advances in downstream prediction, rare disease gene discovery, and general retrieval-augmented inference across genomic modalities.

## 1. Architectural Principles of HyenaDNA

HyenaDNA replaces the quadratic-cost self-attention mechanism in standard transformers with the Hyena "implicit convolution + gating" operator. Each Hyena block computes
$$
H(x) = D_{x_2}\,T_h\,D_{x_1}
$$
where $D_{x_1}, D_{x_2} \in \mathbb{R}^{L \times L}$ are diagonal matrices from learned projections of the input $x \in \mathbb{R}^L$, and $T_h$ is a Toeplitz matrix parameterizing a global 1D convolution. The convolution filter $h_k$ is generated via a small MLP $\gamma_\theta(k)$ (as a neural field) rather than being directly learned, which decouples parameter count from window size. Each block includes pointwise nonlinearity (GELU), layer normalization, and a feed-forward network, with residual connections throughout.

A defining feature is the ability to process single-nucleotide tokens, eschewing fixed $k$-mer tokenization to preserve maximal nucleotide resolution—crucial for tasks involving SNPs or rare mutations. Architectural efficiency stems from the use of fast Fourier transform-based convolution ($O(N\log N)$ time/space per layer), making HyenaDNA orders of magnitude more efficient in both parameter count and runtime for long genomic inputs [2306.15794, 2508.04739].

## 2. Computational Complexity and Scaling

Traditional transformer models exhibit $O(N^2)$ time and memory complexity, limiting input lengths (typically $512$–$4\,000$ bases) to a small fraction of the human genome. In contrast, each HyenaDNA layer, via implicit convolution and diagonal gating, achieves $O(N\log N D)$ runtime and $O(ND)$ memory, where $N$ is sequence length and $D$ embedding dimension. Empirical benchmarks demonstrate up to 160$\times$ speedup relative to FlashAttention-Transformers at $10^6$ tokens, with practical context windows up to $10^6$ nucleotides and efficient scaling to tens of layers ($L=8$–$32$), allowing true whole-gene or ultra-long-range context [2306.15794, 2508.04757].

A comparison of key operational metrics is summarized below:

| Model            | Max Context | Params (M) | Time Complexity |
|------------------|-------------|------------|-----------------|
| HyenaDNA         | $10^6$      | 4–6.5      | $O(N\log N)$    |
| DNABERT/Enformer | $<4\,000$   | 20–80      | $O(N^2)$        |
| NucleotideTransf.| $<4\,000$   | 500–2\,500 | $O(N^2)$        |

## 3. Training Regime and Embedding Strategies

HyenaDNA is pre-trained as a masked language model on the full human reference genome (GRCh38/hg38), using contexts of up to $10^6$ bases. The objective is standard cross-entropy over masked positions:
$$
\mathcal{L}_{\mathrm{MLM}} = -\sum_{t=1}^{L} M_t \log p_\theta(x_t\mid x_{1:L}^{\setminus M})
$$
where $M_t$ flags masked positions. The model learns nucleotide representations at high resolution.

For downstream inference, HyenaDNA serves as a frozen feature extractor (retrieval-augmented pipeline). For input $x_i$ of length $L_i$, the output $h_{1},...,h_{L_i}$ from the last layer is mean-aggregated and L2-normalized to yield sequence embedding $e_i \in \mathbb{R}^D$, which is suitable for k-NN retrieval or as input to lightweight classifiers [2508.04757].

Enhancer classification with z-Curve features, which capture 3D sequence geometry, demonstrates that HyenaDNA embeddings combined with z-Curve systematically improve accuracy (from $0.65$ base to $0.68$ with z-Curve) and achieve inference >8$\times$ faster with 8$\times$ lower CO$_2$ emissions than fine-tuned models [2508.04757].

## 4. Adapter-Based Modalities: CodonMoE and RNA Analysis

HyenaDNA forms the basis for plug-in adapters enabling DNA-trained models to function on RNA-centric tasks. The CodonMoE adapter applies a codon-level mixture-of-experts (MoE) on HyenaDNA outputs:
- Every three nucleotides are averaged to form codon embeddings $z_i$.
- A gating network $G: \mathbb{R}^D \rightarrow \Delta^K$ assigns weights $g_k(z_i)$ for $K$ expert MLPs $E_k$.
- Output:
$$
f_{\mathrm{MoE}}(z_i) = \sum_{k=1}^K g_k(z_i) \cdot E_k(z_i)
$$
- Codon features are tiled back to nucleotide resolution, residual connections and normalization applied, and a lightweight head yields the property prediction.

The CodonMoE architecture is a universal approximator for codon-to-RNA mappings, with formal guarantees. Standard configurations ($K=4$–$8$ experts, $3$–$5$M params) or the "pro" version ($<12$M params) add 1D convolutions over codon neighborhoods, all maintaining sub-quadratic complexity, $O(L\log L + LK)$ [2508.04739].

Benchmarks in mRNA expression and stability demonstrate state-of-the-art rank correlations (Spearman's $\rho$ up to $0.88$), with HyenaDNA+CodonMoE matching or exceeding specialized RNA models (CodonBERT, SpliceBERT) at $<$10$\%$ of their parameter count, and delivering 5–10$\times$ faster inference [2508.04739].

## 5. Empirical Results and Downstream Applications

HyenaDNA sets new top-1 accuracy on regulatory element classification, enhancer detection, chromatin profile prediction, and species assignment, often outperforming baseline CNNs, DNABERT, GPT-style transformers, and Nucleotide Transformer models. On GenomicBenchmarks, it achieves $+10$–$20$ point improvements over prior state-of-the-art for multiple tasks [2306.15794].

Embedding-extraction pipelines using HyenaDNA maintain strong predictive performance across data splits with shifted distributions, indicating superior generalization to unseen genomic contexts compared to models reliant on full fine-tuning. Carbon calculations confirm $8\times$–$77\times$ lower emissions for retrieval-augmented approaches.

In rare disease genomics [2410.15367]:
- HyenaDNA is used to generate sample- and variant-aware gene embeddings by processing full gene sequences personalized by individual pathogenic variants.
- Embeddings at variant positions are averaged to yield a dynamic gene embedding $\mathbf{e}_{g,i}$, which is sensitive to deleterious changes.
- Embeddings across genes are used as features in a protein-protein interaction (PPI) graph neural network and further refined by a genetic algorithm to extract functionally-coherent diagnosis subnetworks.
- The workflow re-identifies known disease genes (e.g., IFIH1 with $F_1 \approx 0.87$, $p=0.009$) and pathways (e.g., interferon signaling), validating the capacity for interpretable target discovery.

## 6. Broader Implications and Future Directions

The HyenaDNA framework demonstrates that single nucleotide–resolution models with sub-quadratic scaling enable both parameter-efficient and computation-efficient solutions for broad genomic inference tasks. CodonMoE and similar adapters "RNA-ize" pretrained DNA models without full RNA pretraining, illustrating a unifying template for multi-modality in genomics. The plug-and-play MoE principle is extensible: analogous adapters could exploit amino-acid context for protein tasks or be designed for locus-level tasks in chromatin modeling [2508.04739].

Practically, HyenaDNA’s architecture and embedding strategies favor scalable, low-footprint, and robust solutions, making them suitable for resource-constrained high-throughput genomics and for systematic interrogation of genome function in rare or common diseases. The variant-sensitivity of embeddings highlights the model’s utility for personalized genomics, offering precise and explainable representations optimized for downstream machine learning integration.

A plausible implication is that continued refinement of such architectures and adapter-based multi-modal pipelines may further reduce the computational and carbon cost of universal genomic modeling while enabling rapid, interpretable hypothesis generation across increasingly complex biological tasks.

Source: https://www.emergentmind.com/topics/hyenadna