CodonMoE: Adaptive Mixture Experts for mRNA

• CodonMoE is an adapter that converts DNA foundation models into robust mRNA analyzers by reorganizing nucleotide features into codon-level representations.
• It employs a dense mixture-of-experts layer with multiple small MLPs and a gating network to capture codon-specific patterns and enhance RNA prediction performance.
• Empirical results demonstrate significant improvements in tasks like mRFP expression and vaccine degradation, achieving strong performance with up to 80% fewer parameters than specialized RNA models.

CodonMoE, short for Adaptive Mixture of Codon Reformative Experts, is a lightweight adapter that converts pretrained DNA LLMs into effective analyzers of mRNA sequences without RNA-specific pretraining. It is introduced to address an efficiency dilemma in genomic language modeling: maintaining separate DNA and RNA models duplicates infrastructure and pretraining effort, while large multi-modal architectures incur high parameter counts, quadratic attention, and expensive cross-modality pretraining. CodonMoE operates by taking nucleotide-level hidden states from a DNA backbone, regrouping them into codon-level representations, transforming those representations through a dense Mixture-of-Experts layer, and reinjecting the codon-aware signal into nucleotide features for downstream prediction. In the reported experiments, DNA backbones augmented with CodonMoE achieve large gains on RNA prediction tasks spanning stability, expression, and regulation, with the HyenaDNA+CodonMoE series reaching state-of-the-art results while preserving the backbone’s sub-quadratic complexity and using substantially fewer parameters than specialized RNA models (Du et al., 6 Aug 2025).

1. Problem setting and design rationale

Genomic LLMs have diverged into two main families: DNA gLMs, such as HyenaDNA, Caduceus, and GPN variants, and RNA/mRNA gLMs, such as CodonBERT, RNA-FM, and HELM. The first family is typically trained on genomes and often emphasizes very long context and sub-quadratic architectures; the second is typically trained on transcriptome or mRNA data and emphasizes tasks such as expression, stability, and splicing. This separation creates an efficiency tradeoff. Separate modality-specific models require duplicated pretraining and maintenance despite the fact that mRNA derives from DNA, whereas single large multi-modal models require massive parameter counts and expensive training (Du et al., 6 Aug 2025).

The motivation for CodonMoE is grounded in two observations. First, DNA data is far more abundant and large DNA foundation models are already being trained in many laboratories. Second, many mRNA tasks are supervised and relatively small, often involving hundreds to thousands of sequences with measured expression, stability, or regulatory effects. The resulting objective is to reuse a pretrained DNA model, add a small adapter for mRNA tasks, avoid large-scale RNA pretraining, and preserve the efficient architecture of the original DNA backbone.

A central difficulty is that DNA models are trained on nucleotide sequences without explicit codon-level semantics or translation information, whereas many mRNA properties depend strongly on codon usage and codon context rather than raw nucleotide patterns alone. The reported diagnostic baselines illustrate this mismatch: regressors trained on frozen DNA embeddings achieve only moderate Spearman correlations on mRFP and degradation tasks, for example mRFP Spearman values of 0.33–0.51 for GPN-MSA and HyenaDNA, which remain far below RNA-specific models. The stated thesis is therefore that a codon-aware adapter can transform a DNA LLM into a strong mRNA analyzer, reducing the need for either separate large RNA models or large multimodal architectures.

2. Architecture and codon-level reformulation

CodonMoE is a post-backbone adapter placed on top of a pretrained DNA model. The paper evaluates it with several DNA backbones: HyenaDNA, Caduceus, GPN-SS, and GPN-MSA. These backbones differ in complexity class—HyenaDNA is $O(L \log L)$, Caduceus and GPN-SS are $O(L)$, and GPN-MSA is $O(L^2)$—but in all cases the backbone remains DNA-focused and pretrained on DNA tasks or genomes rather than mRNA tasks (Du et al., 6 Aug 2025).

The adapter consumes hidden states

$$H \in \mathbb{R}^{\text{batch\_size} \times \text{seq\_len} \times d_{\text{model}}}.$$

It drops the last token, forming $Y = H[:, 0:S-1, :]$, and then reshapes every three consecutive nucleotide embeddings into a codon representation:

$$\text{codons} \in \mathbb{R}^{B \times \lfloor(S-1)/3\rfloor \times 3d}.$$

Each codon is therefore represented by concatenating the three nucleotide embeddings into a $3d$-dimensional vector. This explicitly reorganizes the DNA backbone output into triplet units aligned with codon structure.

For each codon embedding $X$, CodonMoE defines $K$ experts, with default $K=4$. Each expert is a small MLP,

$O(L)$0

and the gating network is

$O(L)$1

The codon-level MoE output is then

$O(L)$2

The routing is dense rather than sparse: there is no explicit top-$O(L)$3 selection, and the softmax distributes weight across all experts.

After the codon outputs are computed, each codon embedding is repeated three times and expanded back to nucleotide positions. The expanded tensor is added residually to $O(L)$4,

$O(L)$5

which yields a codon-enriched nucleotide representation. The architecture then applies LayerNorm, GELU, and dropout, flattens across sequence length, projects to a global feature of dimension $O(L)$6, applies LayerNorm, GELU, and dropout again, and finally produces a scalar prediction through $O(L)$7. The output corresponds to a sequence-level property such as expression, stability, or regulatory activity.

The enhanced variant, CodonMoE-pro, adds a codon neighborhood convolution after the MoE transformation. This convolution operates on sequences of codon-level features rather than isolated codons, allowing the model to capture local codon motifs, codon $O(L)$8-grams, codon pair biases, and other short-range contextual effects. The paper states that CodonMoE-pro is consistently stronger than the basic CodonMoE at similar or lower parameter counts. This suggests that codon context, not merely codon identity, is central to the adapter’s advantage.

3. Theoretical formulation and expressivity

The paper provides a formal universal approximation theorem for CodonMoE as an adapter on top of a pretrained DNA backbone (Du et al., 6 Aug 2025). The alphabet is $O(L)$9, with RNA’s $O(L^2)$0 mapped to $O(L^2)$1 to unify DNA and RNA representation. The DNA sequence space is

$O(L^2)$2

the codon space is

$O(L^2)$3

and the target function class is

$O(L^2)$4

The backbone is written as

$O(L^2)$5

and codon embeddings are formed from nucleotide embeddings $O(L^2)$6 by

$O(L^2)$7

The CodonMoE adapter is written as

$O(L^2)$8

with dense MoE structure over codon embeddings.

Under the assumptions that each expert $O(L^2)$9 can uniformly approximate any continuous function on compact subsets of $$H \in \mathbb{R}^{\text{batch\_size} \times \text{seq\_len} \times d_{\text{model}}}.$$0, that the gating network outputs simplex-valued non-negative weights summing to one, and that DNA sequences are partitioned into aligned codons, the theorem states that for any $$H \in \mathbb{R}^{\text{batch\_size} \times \text{seq\_len} \times d_{\text{model}}}.$$1 and any $$H \in \mathbb{R}^{\text{batch\_size} \times \text{seq\_len} \times d_{\text{model}}}.$$2, there exist a number of experts $$H \in \mathbb{R}^{\text{batch\_size} \times \text{seq\_len} \times d_{\text{model}}}.$$3 and adapter parameters such that

$$H \in \mathbb{R}^{\text{batch\_size} \times \text{seq\_len} \times d_{\text{model}}}.$$4

The significance of this result is narrowly defined but important. It does not claim that any finite implementation will recover arbitrary biology; rather, it establishes that the architecture is, in principle, expressive enough to represent any continuous mapping from codon sequences to a scalar property, provided sufficient capacity. The paper’s interpretation is that CodonMoE is a universal approximator at the codon level. A plausible implication is that the adapter’s inductive bias is not restricted to a small predefined class of codon effects: with enough experts and sufficient embedding quality from the backbone, it can model arbitrary codon-wise dependencies relevant to expression, stability, or regulation.

4. Training protocol, datasets, and evaluation

The adaptation strategy is deliberately narrow. The base DNA model is reused as pretrained, input RNA sequences are treated as DNA by mapping $$H \in \mathbb{R}^{\text{batch\_size} \times \text{seq\_len} \times d_{\text{model}}}.$$5, the backbone produces hidden states, and only CodonMoE or CodonMoE-pro is trained on the downstream supervised dataset. The paper emphasizes this as adapter-only training and states that all experiments are tractable on a single NVIDIA A100 GPU (Du et al., 6 Aug 2025).

Four supervised RNA tasks are used. The mRFP expression dataset contains 1,459 codon-randomized mRFP coding sequences of 675 bp, with fluorescence as the outcome. The SARS-CoV-2 vaccine degradation dataset contains 2,400 mRNA vaccine variants with measured in-cell stability or degradation rates in HEK293T cells. The Tc-riboswitch dataset contains 355 tetracycline-responsive riboswitch dimer sequences upstream of GFP, with switching factor as the outcome. The MLOS dataset contains 164 influenza hemagglutinin mRNA candidates with fixed UTRs and variable coding sequence, with protein expression measured in HeLa cells. All tasks use $$H \in \mathbb{R}^{\text{batch\_size} \times \text{seq\_len} \times d_{\text{model}}}.$$6 train/validation/test splits, matching CodonBERT and HELM for fair comparison.

The tasks are treated as regression problems, and the main reported evaluation metric is Spearman’s rank correlation between predicted and measured values. The paper does not write the loss explicitly. It states only that training is standard regression and reports regularization components including LayerNorm, GELU activations, and dropout with rate 0.1 in the pseudocode. Example hyperparameters include learning rates of $$H \in \mathbb{R}^{\text{batch\_size} \times \text{seq\_len} \times d_{\text{model}}}.$$7 or $$H \in \mathbb{R}^{\text{batch\_size} \times \text{seq\_len} \times d_{\text{model}}}.$$8 and 100 epochs for CodonMean and CodonMoE variants.

The experimental baselines fall into three groups. The first consists of RNA foundation models and RNA-pretrained deep baselines, including RNA-FM, SpliceBERT, CodonBERT, and the HELM family. The second consists of classical or shallow codon-sequence models such as TF-IDF on codon features, Plain TextCNN, and Codon2vec+TextCNN. The third consists of DNA backbone baselines without CodonMoE, where simple regressors such as MLP or XGBoost are trained on frozen or minimally adapted DNA embeddings. This baseline design is important because it isolates the effect of codon-aware adaptation from the effect of merely using a DNA backbone.

5. Empirical performance and ablation structure

The reported results show large improvements when CodonMoE is added to DNA backbones. On vaccine degradation and mRFP expression, GPN-SS improves from 0.60 to 0.74 and from 0.56 to 0.82, respectively; GPN-MSA improves from 0.55 to 0.77 and from 0.33 to 0.79; Caduceus improves from 0.56 to 0.80 and from 0.49 to 0.80; and HyenaDNA improves from 0.69 to 0.81 and from 0.44 to 0.84. With CodonMoE-pro, GPN-MSA reaches 0.82 on vaccine degradation and 0.81 on mRFP, while HyenaDNA reaches 0.84 and 0.88, respectively (Du et al., 6 Aug 2025).

Model configuration	Vaccine	mRFP
HyenaDNA	0.69	0.44
HyenaDNA + CodonMoE	0.81	0.84
HyenaDNA + CodonMoE-pro	0.84	0.88

These gains are especially pronounced for models that initially underperform on RNA tasks. The paper reports, for example, that GPN-MSA on mRFP moves from 0.33 to 0.79 with CodonMoE and to 0.81 with CodonMoE-pro, while HyenaDNA moves from 0.44 to 0.88 with the enhanced variant. The performance-boost summary gives average absolute $$H \in \mathbb{R}^{\text{batch\_size} \times \text{seq\_len} \times d_{\text{model}}}.$$9Spearman values of +0.38 for GPN-MSA+CodonMoE-pro and +0.29 for HyenaDNA+CodonMoE-pro across vaccine and mRFP tasks.

Against specialized RNA models, the strongest comparison is HyenaDNA+CodonMoE-pro. CodonBERT reports 0.77 on vaccine, 0.85 on mRFP, 0.56 on Tc-ribo, and 0.54 on MLOS. RNA-FM reports 0.74 on vaccine, 0.80 on mRFP, and 0.58 on Tc-ribo. HyenaDNA+CodonMoE-pro reports 0.84 on vaccine, 0.88 on mRFP, 0.60 on Tc-ribo, and 0.63 on MLOS. The paper therefore states that it outperforms all RNA-specific models on three of the four tasks and remains competitive on Tc-riboswitch.

Ablation results identify where the improvement originates. CodonMean, which simply averages nucleotides into codons, already improves performance, but CodonMoE adds a further gain and CodonMoE-pro performs best. For HyenaDNA on mRFP, the progression is 0.765 $Y = H[:, 0:S-1, :]$0 0.837 $Y = H[:, 0:S-1, :]$1 0.878; on vaccine it is 0.789 $Y = H[:, 0:S-1, :]$2 0.812 $Y = H[:, 0:S-1, :]$3 0.844. A separate ablation compares a dense baseline to CodonMoE-pro at the same parameter count, showing 0.80 versus 0.84 on vaccine and 0.82 versus 0.88 on mRFP. This directly argues against the misconception that the improvement is explained only by extra parameters: the paper attributes it to MoE + codon structure rather than model size alone.

The frozen-embedding diagnostics reinforce the same point. GPN-MSA combined with MLP or XGBoost attains 0.33 or 0.479 on mRFP and 0.572 or 0.750 on vaccine; HyenaDNA combined with MLP or XGBoost attains 0.439 or 0.512 on mRFP and 0.695 or 0.711 on vaccine. These numbers show that DNA models encode some task-relevant information, especially for vaccine stability, but remain far below the CodonMoE-enhanced models.

6. Efficiency, biological scope, and field position

Efficiency is central to the method’s framing. The best reported configuration, HyenaDNA+CodonMoE-pro, has 7.5M parameters in total, with the HyenaDNA backbone at approximately 4.1M and the adapter adding approximately 3.4M. The paper states that this is about 9% of CodonBERT’s 81.7M parameters, about 15% of the 50M XE/HELM models, and about 7.5% of RNA-FM’s 100M parameters. It therefore supports the abstract’s claim that the method achieves strong results with 80% fewer parameters than specialized RNA models, and in several comparisons the reduction is even larger (Du et al., 6 Aug 2025).

The computational complexity remains dominated by the backbone. CodonMoE processes roughly $Y = H[:, 0:S-1, :]$4 codon positions with a small number of experts and small feedforward layers, so it adds negligible complexity relative to the backbone. Consequently, HyenaDNA+CodonMoE-pro preserves sub-quadratic $Y = H[:, 0:S-1, :]$5 complexity, while Caduceus+CodonMoE preserves linear $Y = H[:, 0:S-1, :]$6 complexity. This distinguishes the approach from most high-performing RNA models in the comparison set, which are transformer-based and therefore $Y = H[:, 0:S-1, :]$7.

Biologically, the results suggest that DNA backbones already contain useful sequence and structural priors for RNA tasks, but not explicit codon semantics. CodonMoE bridges this gap by reorganizing nucleotide-level features into codon triplets, enabling expert specialization over codon classes and codon regimes, and, in the pro variant, modeling local codon context. The performance pattern is consistent with this interpretation: the strongest gains are on translation-related tasks such as mRFP expression and MLOS, substantial gains also appear on stability, and the relative advantage is weaker on Tc-riboswitch, which depends more on RNA folding and ligand responsiveness than on codon usage per se.

The method also has explicit limitations. It assumes codon-structured input, including a fixed reading frame and sequences organized in multiples of three, which makes it less naturally suited to noncoding RNAs or regulatory elements. The theorem relies on continuity, compactness, and sufficient expert capacity, all of which are idealized assumptions. The experiments cover four supervised RNA tasks with relatively small sample sizes, and cross-species generalization is not tested. The paper therefore presents CodonMoE not as a universal solution to RNA modeling, but as a modality-bridging adapter whose main demonstrated strength is converting DNA foundation models into strong mRNA analyzers.

Within the broader field, the distinctive contribution is threefold. First, CodonMoE is a modality-bridging adapter rather than merely a task-specific head. Second, it introduces a codon-aware MoE built directly from DNA backbone representations. Third, it provides both a codon-level universal approximation result and an empirical demonstration that DNA backbones, when properly adapted, can outperform dedicated RNA models while retaining favorable scaling. This suggests a unified genomic language-modeling strategy in which one or a few DNA foundation models are paired with lightweight adapters for RNA tasks, reducing duplicated pretraining and deployment overhead while preserving modality-specific performance.