dnaHNet: Tokenizer-Free Genomic Model

• dnaHNet is a foundation model for genomic sequences that employs differentiable dynamic chunking to adaptively compress raw nucleotides while preserving motif integrity.
• Its modular, hierarchical design—with recursive Encoder, Main Network, and Decoder blocks—yields a theoretical 36× FLOP reduction and over 3× empirical speedup compared to standard Transformers.
• Pretraining on 144 billion prokaryotic bases demonstrates superior zero-shot performance in tasks like protein variant effect prediction and gene essentiality, outperforming comparable architectures.

dnaHNet is a tokenizer-free, autoregressive foundation model for genomic sequences that addresses the central trade-off in large-scale genome modeling: reconciling biological fidelity with computational scalability. Standard fixed-vocabulary tokenizers fragment motifs such as codons and regulatory elements, while nucleotide-level models yield prohibitively high costs for long contexts. dnaHNet leverages a differentiable dynamic chunking mechanism to segment raw nucleotides into latent tokens end-to-end, enabling adaptive compression while maintaining predictive accuracy. The model is pretrained on prokaryotic genomic data and demonstrates superior scaling, efficiency, and emergent interpretability compared to leading architectures.

1. Model Architecture and Mechanisms

dnaHNet models a DNA sequence $X = (x_1,\dots,x_L)$, $x_t\in\{A,C,G,T\}$, through an autoregressive factorization: $P(X)=\prod_{t=1}^L P(x_t \mid x_{<t})$

To scale efficiently to sequences of length $L\sim10^6$, dnaHNet abandons fixed tokenization in favor of a modular architecture composed of three interconnected components—Encoder ($\mathcal{E}$), Main Network ($\mathcal{M}$), and Decoder ($\mathcal{D}$)—each enabling recursive stacking for multi-stage compression.

Differentiable Dynamic Chunking

The Encoder first embeds the raw sequence with four Mamba layers and one Transformer layer, producing hidden states $\mathbf{h}_{1:L}\in\mathbb{R}^{L\times D}$. For each position $t$, a boundary predictor computes a continuous boundary probability: $$p_t = \tfrac{1}{2}(1 - \mathrm{CosineSim}(W_q\mathbf{h}_t, W_k\mathbf{h}_{t-1}))$$ where $x_t\in\{A,C,G,T\}$0. High $x_t\in\{A,C,G,T\}$1 values indicate context shifts (e.g., codon boundaries or regulatory motifs). During inference, boundaries $x_t\in\{A,C,G,T\}$2 are discretized, and the Chunking layer downsamples by retaining only $x_t\in\{A,C,G,T\}$3 where $x_t\in\{A,C,G,T\}$4, producing a compressed latent sequence $x_t\in\{A,C,G,T\}$5.

Recursive Hierarchy

Latents $x_t\in\{A,C,G,T\}$6 are input to $x_t\in\{A,C,G,T\}$7, which may itself be another Encoder–Main–Decoder block, supporting multi-stage compression. In practice, dnaHNet adopts a two-stage hierarchy with compression ratios $x_t\in\{A,C,G,T\}$8 (codon granularity) and $x_t\in\{A,C,G,T\}$9 (capturing codon-pair biases), for an overall expected compression factor $P(X)=\prod_{t=1}^L P(x_t \mid x_{<t})$0.

Each Main Network alternates Mamba layers (subquadratic SSM-based operators) and Transformer layers (quadratic attention), with parameter allocation by scale. In the XL configuration ($P(X)=\prod_{t=1}^L P(x_t \mid x_{<t})$1B parameters), 30% of parameters are assigned to encoders/decoders and 70% to Main Networks (Transformer depth $P(X)=\prod_{t=1}^L P(x_t \mid x_{<t})$2, $P(X)=\prod_{t=1}^L P(x_t \mid x_{<t})$3, $P(X)=\prod_{t=1}^L P(x_t \mid x_{<t})$4 attention heads).

Upsampling and Decoder

The Main Network outputs $P(X)=\prod_{t=1}^L P(x_t \mid x_{<t})$5, which are smoothed recursively: $P(X)=\prod_{t=1}^L P(x_t \mid x_{<t})$6 Each smoothed latent $P(X)=\prod_{t=1}^L P(x_t \mid x_{<t})$7 is uniformly copied to its constituent nucleotides, yielding $P(X)=\prod_{t=1}^L P(x_t \mid x_{<t})$8. Four Mamba layers and one Transformer layer decode $P(X)=\prod_{t=1}^L P(x_t \mid x_{<t})$9 into logits over $L\sim10^6$0.

Compression-Rate Regularization

To avoid degenerate chunking, an auxiliary loss aligns the discrete boundary rate $L\sim10^6$1 with continuous expectation $L\sim10^6$2 per stage $L\sim10^6$3. For target ratio $L\sim10^6$4: $L\sim10^6$5 The total loss is $L\sim10^6$6, with $L\sim10^6$7 and negative log-likelihood $L\sim10^6$8.

2. Computational Complexity and Efficiency

Standard Transformers on length $L\sim10^6$9 incur a computational cost of $\mathcal{E}$0 due to quadratic self-attention. dnaHNet restructures this cost:

$\mathcal{E}$1

With quadratic attention in the Main Network operating on $\mathcal{E}$2, the dominant term becomes $\mathcal{E}$3. For $\mathcal{E}$4, this is a $\mathcal{E}$5 FLOP reduction. Encoder and decoder terms are linear: $\mathcal{E}$6.

Empirical benchmarks on $\mathcal{E}$7 nucleotides indicate that dnaHNet (218M parameters) reduces inference FLOPs by $\mathcal{E}$8 compared to StripedHyena2 (166M parameters), and achieves more than $\mathcal{E}$9 end-to-end speedup over equivalently sized Transformers.

3. Pretraining and Zero-Shot Evaluation

Pretraining Data and Objective

dnaHNet is pretrained on 17,648,721 prokaryotic genome sequences (from GTDB, 85,205 species), totaling 144 billion nucleotides, with each non-overlapping chunk up to 8192 bp. The end-to-end training objective combines autoregressive next-nucleotide prediction with two-stage compression-rate regularization ($\mathcal{M}$0).

Zero-Shot Task Performance

Protein Variant Effect Prediction (MaveDB; 21,250 data points): Model fitness predicted by log-likelihood differences.

Model	Training FLOPs	Spearman ρ
dnaHNet	$\mathcal{M}$1	0.2601
	$\mathcal{M}$2	0.2865
	$\mathcal{M}$3	0.3143
	$\mathcal{M}$4	0.3266
StripedHyena2	$\mathcal{M}$5	0.2639
	$\mathcal{M}$6	0.2921
	$\mathcal{M}$7	0.3084
	$\mathcal{M}$8	0.3110
Transformer	$\mathcal{M}$9	0.1348
	$\mathcal{D}$0	0.1355
	$\mathcal{D}$1	0.1497
	$\mathcal{D}$2	0.1555

dnaHNet outperforms StripedHyena2 by approximately 0.015 in Spearman correlation at the highest compute budgets.

Gene Essentiality (DEG; 185,226 genes): Predicts loss of essential genes after in silico premature stop codon insertion, measured by AUROC.

Model	Training FLOPs	AUROC
dnaHNet (3,2 hierarchy)	$\mathcal{D}$3	0.5719
	$\mathcal{D}$4	0.5840
	$\mathcal{D}$5	0.5993
	$\mathcal{D}$6	0.6050
dnaHNet (2,2 hierarchy)	$\mathcal{D}$7	0.5538
	$\mathcal{D}$8	0.5797
	$\mathcal{D}$9	0.5854
	$\mathbf{h}_{1:L}\in\mathbb{R}^{L\times D}$0	0.5900
StripedHyena2	$\mathbf{h}_{1:L}\in\mathbb{R}^{L\times D}$1	0.5375
	$\mathbf{h}_{1:L}\in\mathbb{R}^{L\times D}$2	0.5670
	$\mathbf{h}_{1:L}\in\mathbb{R}^{L\times D}$3	0.5720
	$\mathbf{h}_{1:L}\in\mathbb{R}^{L\times D}$4	0.5759

The (3,2) hierarchy, aligning with codon structure, yields the best scaling and AUROC.

4. Emergent Biological Interpretability

dnaHNet’s hierarchical chunking discovers biologically relevant structures without supervision. The learned boundaries, at each hierarchy stage, exhibit interpretable selection rates.

Triplet Codon Awareness (Stage 1)

Boundary selection rates in coding regions show marked periodicity:

• Position 1: 6.5%
• Position 2: 42.6%
• Position 3: 58.4%

Outside coding regions, selection is comparatively uniform (∼35.8%), indicating chunking does not force a codon structure where none exists.

Functional Region Awareness (Stage 2)

Selection rates differ by genomic region:

• Promoter: 71.5%
• Start codon: 81.3%
• Coding region: 48.4%
• Stop codon: 51.7%
• Intergenic: 74.6%

This suggests the upper hierarchy learns to identify functional elements, utilizing lower-layer codon-aware representations to delineate regulatory and genic structures.

Hierarchical Chunking Statistics

Genomic Feature	Stage 1	Stage 2
Global selection	35.8%	51.3%
Promoter	35.6%	71.5%
Start codon	35.1%	81.3%
Coding region	35.8%	48.4%
Stop codon	34.5%	51.7%
Intergenic	35.3%	74.6%
Position 1	6.5%	51.8%
Position 2	42.6%	38.0%
Position 3	58.4%	55.9%

5. Context, Significance, and Applications

dnaHNet establishes a unified framework for modeling genomic sequences at scale, resolving the trade-off between long-context efficiency and biological motif coherence. Its recursive chunking enables theoretical $\mathbf{h}_{1:L}\in\mathbb{R}^{L\times D}$5 FLOP savings, which translates to greater than $\mathbf{h}_{1:L}\in\mathbb{R}^{L\times D}$6 empirical inference speedup. The model, pretrained on 144 billion prokaryotic bases, not only surpasses previous architectures (e.g., StripedHyena2, Transformers) on zero-shot protein and gene essentiality tasks, but also recapitulates biological syntax at multiple hierarchical levels.

A plausible implication is that the model architecture may generalize to other contexts in genomics where hierarchical motif discovery and computational tractability are essential. The adaptive chunking mechanism provides a transparent window into learned sequence grammar, conferring interpretability that is often lacking in large foundation models.

6. Limitations and Future Directions

While dnaHNet demonstrates robust scaling and interpretability on prokaryotic sequences, its current instantiation is based on unsupervised learning and fixed (albeit biologically motivated) compression ratios. Its generalizability to eukaryotic genomes, or to tasks requiring explicit supervision at intermediate levels, remains to be fully explored. Potential extensions may include dynamic adaptation of hierarchical depths, integration with downstream supervised predictors, and transferability to non-genomic biological sequence domains.

(Shah et al., 11 Feb 2026)