---
title: 'dnaHNet: Tokenizer-Free Genomic Model'
url: https://www.emergentmind.com/topics/dnahnet
type: topic
---

# dnaHNet: Tokenizer-Free Genomic Model

dnaHNet is a tokenizer-free, autoregressive foundation model for genomic sequences that addresses the central trade-off in large-scale genome modeling: reconciling biological fidelity with computational scalability. Standard fixed-vocabulary tokenizers fragment motifs such as codons and regulatory elements, while nucleotide-level models yield prohibitively high costs for long contexts. dnaHNet leverages a differentiable dynamic chunking mechanism to segment raw nucleotides into latent tokens end-to-end, enabling adaptive compression while maintaining predictive accuracy. The model is pretrained on prokaryotic genomic data and demonstrates superior scaling, efficiency, and emergent interpretability compared to leading architectures.

## 1. Model Architecture and Mechanisms

dnaHNet models a DNA sequence $X = (x_1,\dots,x_L)$, $x_t\in\{A,C,G,T\}$, through an autoregressive factorization:
\[
P(X)=\prod_{t=1}^L P(x_t \mid x_{<t})
\]

To scale efficiently to sequences of length $L\sim10^6$, dnaHNet abandons fixed tokenization in favor of a modular architecture composed of three interconnected components—Encoder ($\mathcal{E}$), Main Network ($\mathcal{M}$), and Decoder ($\mathcal{D}$)—each enabling recursive stacking for multi-stage compression.

### Differentiable Dynamic Chunking

The Encoder first embeds the raw sequence with four Mamba layers and one Transformer layer, producing hidden states $\mathbf{h}_{1:L}\in\mathbb{R}^{L\times D}$. For each position $t$, a boundary predictor computes a continuous boundary probability:
\[
p_t = \tfrac{1}{2}(1 - \mathrm{CosineSim}(W_q\mathbf{h}_t, W_k\mathbf{h}_{t-1}))
\]
where $W_q,W_k\in\mathbb{R}^{D\times D}$. High $p_t$ values indicate context shifts (e.g., codon boundaries or regulatory motifs). During inference, boundaries $b_t \in \{0,1\}$ are discretized, and the Chunking layer downsamples by retaining only $\mathbf{h}_t$ where $b_t = 1$, producing a compressed latent sequence $\mathbf{e}_{1:L'}$.

### Recursive Hierarchy

Latents $\mathbf{e}_{1:L'}$ are input to $\mathcal{M}$, which may itself be another Encoder–Main–Decoder block, supporting multi-stage compression. In practice, dnaHNet adopts a two-stage hierarchy with compression ratios $R_1=3$ (codon granularity) and $R_2=2$ (capturing codon-pair biases), for an overall expected compression factor $R=6$.

Each Main Network alternates Mamba layers (subquadratic SSM-based operators) and Transformer layers (quadratic attention), with parameter allocation by scale. In the XL configuration ($\approx 1$B parameters), 30% of parameters are assigned to encoders/decoders and 70% to Main Networks (Transformer depth $N=18$, $D=832$, $13$ attention heads).

### Upsampling and Decoder

The Main Network outputs $\hat{\mathbf{e}}_{1:L'}$, which are smoothed recursively:
\[
\bar{\mathbf{e}}_j = P_j\,\hat{\mathbf{e}}_j + (1-P_j)\,\bar{\mathbf{e}}_{j-1},\quad P_j=p_{\text{boundary}(j)}
\]
Each smoothed latent $\bar{\mathbf{e}}_j$ is uniformly copied to its constituent nucleotides, yielding $\tilde{X}\in\mathbb{R}^{L\times D}$. Four Mamba layers and one Transformer layer decode $\tilde{X}$ into logits over $\{A,C,G,T\}$.

### Compression-Rate Regularization

To avoid degenerate chunking, an auxiliary loss aligns the discrete boundary rate $F_s$ with continuous expectation $G_s$ per stage $s$. For target ratio $R_s$:
\[
\mathcal{L}_{\text{rate}}^{(s)} = \frac{R_s}{R_s-1}\left((R_s-1)F_sG_s + (1-F_s)(1-G_s)\right)
\]
The total loss is $\mathcal{L} = \mathcal{L}_{\mathrm{NLL}} + \alpha\sum_s \mathcal{L}_{\text{rate}}^{(s)}$, with $\alpha\approx 0.01$ and negative log-likelihood $\mathcal{L}_{\mathrm{NLL}}=-\sum_{t=1}^L\log P_\theta(x_t|x_{<t})$.

## 2. Computational Complexity and Efficiency

Standard Transformers on length $L$ incur a computational cost of $\Theta(L^2D)$ due to quadratic self-attention. dnaHNet restructures this cost:

\[
\mathrm{FLOPs}_{\text{total}}(L) = \mathrm{FLOPs}_{\mathcal{E}}(L) + \mathrm{FLOPs}_{\mathcal{M}}(L/R) + \mathrm{FLOPs}_{\mathcal{D}}(L)
\]
With quadratic attention in the Main Network operating on $L/R$, the dominant term becomes $\Theta((L/R)^2D)$. For $R=6$, this is a $36\times$ FLOP reduction. Encoder and decoder terms are linear: $\mathcal{O}(LD)$.

Empirical benchmarks on $L=10^6$ nucleotides indicate that dnaHNet (218M parameters) reduces inference FLOPs by $3.89\times$ compared to StripedHyena2 (166M parameters), and achieves more than $3\times$ end-to-end speedup over equivalently sized Transformers.

## 3. Pretraining and Zero-Shot Evaluation

### Pretraining Data and Objective

dnaHNet is pretrained on 17,648,721 prokaryotic genome sequences (from GTDB, 85,205 species), totaling 144 billion nucleotides, with each non-overlapping chunk up to 8192 bp. The end-to-end training objective combines autoregressive next-nucleotide prediction with two-stage compression-rate regularization ($\alpha=0.01$).

### Zero-Shot Task Performance

**Protein Variant Effect Prediction** (MaveDB; 21,250 data points): Model fitness predicted by log-likelihood differences.

| Model         | Training FLOPs | Spearman ρ |
|---------------|---------------|------------|
| dnaHNet       | $8.0\times 10^{18}$ | 0.2601   |
|               | $1.6\times 10^{19}$ | 0.2865   |
|               | $3.2\times 10^{19}$ | 0.3143   |
|               | $6.4\times 10^{19}$ | 0.3266   |
| StripedHyena2 | $1.0\times 10^{19}$ | 0.2639   |
|               | $2.0\times 10^{19}$ | 0.2921   |
|               | $4.0\times 10^{19}$ | 0.3084   |
|               | $7.1\times 10^{19}$ | 0.3110   |
| Transformer   | $8.0\times 10^{18}$ | 0.1348   |
|               | $2.0\times 10^{19}$ | 0.1355   |
|               | $4.0\times 10^{19}$ | 0.1497   |
|               | $8.0\times 10^{19}$ | 0.1555   |

dnaHNet outperforms StripedHyena2 by approximately 0.015 in Spearman correlation at the highest compute budgets.

**Gene Essentiality** (DEG; 185,226 genes): Predicts loss of essential genes after in silico premature stop codon insertion, measured by AUROC.

| Model                   | Training FLOPs        | AUROC   |
|-------------------------|----------------------|---------|
| dnaHNet (3,2 hierarchy) | $8.0\times 10^{18}$  | 0.5719  |
|                         | $1.6\times 10^{19}$  | 0.5840  |
|                         | $3.2\times 10^{19}$  | 0.5993  |
|                         | $6.4\times 10^{19}$  | 0.6050  |
| dnaHNet (2,2 hierarchy) | $8.0\times 10^{18}$  | 0.5538  |
|                         | $1.6\times 10^{19}$  | 0.5797  |
|                         | $3.2\times 10^{19}$  | 0.5854  |
|                         | $6.4\times 10^{19}$  | 0.5900  |
| StripedHyena2           | $1.0\times 10^{19}$  | 0.5375  |
|                         | $2.0\times 10^{19}$  | 0.5670  |
|                         | $4.0\times 10^{19}$  | 0.5720  |
|                         | $7.1\times 10^{19}$  | 0.5759  |

The (3,2) hierarchy, aligning with codon structure, yields the best scaling and AUROC.

## 4. Emergent Biological Interpretability

dnaHNet’s hierarchical chunking discovers biologically relevant structures without supervision. The learned boundaries, at each hierarchy stage, exhibit interpretable selection rates.

### Triplet Codon Awareness (Stage 1)

Boundary selection rates in coding regions show marked periodicity:

- Position 1: 6.5%
- Position 2: 42.6%
- Position 3: 58.4%

Outside coding regions, selection is comparatively uniform (∼35.8%), indicating chunking does not force a codon structure where none exists.

### Functional Region Awareness (Stage 2)

Selection rates differ by genomic region:

- Promoter: 71.5%
- Start codon: 81.3%
- Coding region: 48.4%
- Stop codon: 51.7%
- Intergenic: 74.6%

This suggests the upper hierarchy learns to identify functional elements, utilizing lower-layer codon-aware representations to delineate regulatory and genic structures.

**Hierarchical Chunking Statistics**

| Genomic Feature | Stage 1 | Stage 2 |
|-----------------|---------|---------|
| Global selection| 35.8%   | 51.3%   |
| Promoter        | 35.6%   | 71.5%   |
| Start codon     | 35.1%   | 81.3%   |
| Coding region   | 35.8%   | 48.4%   |
| Stop codon      | 34.5%   | 51.7%   |
| Intergenic      | 35.3%   | 74.6%   |
| Position 1      | 6.5%    | 51.8%   |
| Position 2      | 42.6%   | 38.0%   |
| Position 3      | 58.4%   | 55.9%   |

## 5. Context, Significance, and Applications

dnaHNet establishes a unified framework for modeling genomic sequences at scale, resolving the trade-off between long-context efficiency and biological motif coherence. Its recursive chunking enables theoretical $36\times$ FLOP savings, which translates to greater than $3\times$ empirical inference speedup. The model, pretrained on 144 billion prokaryotic bases, not only surpasses previous architectures (e.g., StripedHyena2, Transformers) on zero-shot protein and gene essentiality tasks, but also recapitulates biological syntax at multiple hierarchical levels.

A plausible implication is that the model architecture may generalize to other contexts in genomics where hierarchical motif discovery and computational tractability are essential. The adaptive chunking mechanism provides a transparent window into learned sequence grammar, conferring interpretability that is often lacking in large foundation models.

## 6. Limitations and Future Directions

While dnaHNet demonstrates robust scaling and interpretability on prokaryotic sequences, its current instantiation is based on unsupervised learning and fixed (albeit biologically motivated) compression ratios. Its generalizability to eukaryotic genomes, or to tasks requiring explicit supervision at intermediate levels, remains to be fully explored. Potential extensions may include dynamic adaptation of hierarchical depths, integration with downstream supervised predictors, and transferability to non-genomic biological sequence domains.

[2602.10603]

Source: https://www.emergentmind.com/topics/dnahnet