NucEL: A Genomic Foundation Model for DNA Analysis

• NucEL is a genomic foundation model that leverages ELECTRA-style replaced-token detection and single-nucleotide tokenization to capture fine-grained DNA features.
• It achieves state-of-the-art results on regulatory tasks such as promoter detection and transcription factor binding prediction while using fewer parameters than traditional MLM methods.
• The model uses a ModernBERT-style transformer with hybrid local-global and flash attention to ensure computational efficiency and biologically interpretable outputs.

Searching arXiv for the NucEL paper and a few directly related genomic pretraining references. NucEL is a genomic foundation model that combines ELECTRA-style pre-training with single-nucleotide tokenization and an efficient ModernBERT architecture to learn rich, interpretable representations of DNA. It is specifically designed to address the limitations of masked language modeling (MLM) approaches like DNABERT and Nucleotide Transformer, while remaining computationally tractable and biologically meaningful. Pre-trained on the human genome, it uses a discriminator to identify tokens altered by a generator, thereby providing comprehensive token-level supervision across all sequence positions, and it reports state-of-the-art results on diverse downstream tasks including regulatory element identification, transcription factor binding prediction, open chromatin classification, and histone modification profiling (Ding et al., 15 Aug 2025).

1. Definition and problem setting

NucEL treats the genome as a “language” and pre-trains a transformer on raw DNA sequences from the human reference genome. The aim is to learn contextual embeddings that capture regulatory motifs, chromatin states, and other functional patterns, which can then be transferred to downstream tasks such as promoter detection, transcription factor binding prediction, and histone mark classification. In this formulation, DNA sequence modeling is not framed as a purely generative problem; rather, it is used to produce transferable sequence representations with fine-grained biological resolution (Ding et al., 15 Aug 2025).

The model is explicitly motivated by limitations attributed to MLM-based genomic LLMs. Those limitations are stated as partial token supervision, objective mismatch between masked pre-training and unmasked fine-tuning, and high computational cost. In the comparison presented, MLM supervises only the masked fraction of positions, whereas NucEL’s replaced-token detection supervises all sequence positions. This suggests that NucEL is positioned not simply as another genomic transformer, but as a reformulation of the pre-training objective itself.

A central design claim is that NucEL is the first ELECTRA-style pre-training framework for genomic foundation models. Its discriminator is the deployed encoder after pre-training, while the generator is retained only as a training scaffold. This aligns the pre-training signal more closely with token-level discrimination than with masked-token reconstruction, and the paper presents this as both an efficiency and transfer-learning advantage (Ding et al., 15 Aug 2025).

2. ELECTRA-style genomic pre-training

NucEL adopts the standard ELECTRA generator–discriminator framework, instantiated for genomic sequences. For a DNA sequence of length $N$,

$\mathcal{X} = \{x_1, x_2, \ldots, x_N\},$

a subset of positions is selected to be masked, producing a masked sequence $\mathcal{X}_{\text{masked}}$. A smaller generator predicts the identities of the masked tokens. Its loss is standard cross-entropy over masked positions: $$\mathcal{L}_{\text{generator}} = -\frac{1}{|\mathcal{M}|} \sum_{i \in \mathcal{M}} \log P_G(x_i \mid \mathcal{X}_{\text{masked}}).$$

The masked tokens are then replaced by the generator’s sampled predictions to create a corrupted sequence $\mathcal{X}'$. The discriminator receives $\mathcal{X}'$ and predicts, for every position, whether the token is original or replaced. With label

$$y_i = \begin{cases} 1 & \text{if } x'_i = x_i, \ 0 & \text{if } x'_i \neq x_i, \end{cases}$$

the replaced-token detection loss is

$$\mathcal{L}_{\text{discriminator}} = -\frac{1}{N} \sum_{i=1}^N \left[ y_i \log P_D(y_i = 1 \mid \mathcal{X}') + (1 - y_i)\log \big(1 - P_D(y_i = 1 \mid \mathcal{X}')\big) \right].$$

The total loss combines generator and discriminator terms: $$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{generator}} + \lambda \,\mathcal{L}_{\text{discriminator}},$$ with $\lambda = 50.0$. The paper emphasizes that, for a masking ratio around $\mathcal{X} = \{x_1, x_2, \ldots, x_N\},$0, the discriminator receives about $\mathcal{X} = \{x_1, x_2, \ldots, x_N\},$1–$\mathcal{X} = \{x_1, x_2, \ldots, x_N\},$2 denser supervision per sequence than MLM. This is the core methodological distinction: NucEL replaces sparse masked-token supervision with comprehensive token-level supervision over the full sequence (Ding et al., 15 Aug 2025).

The model’s pre-training objective is also presented as closer to downstream classification than MLM. A plausible implication is that this reduces the pre-training/fine-tuning mismatch that has been treated as a structural weakness of MLM-based genomic models.

3. Tokenization, architecture, and training configuration

A major design choice is single-nucleotide tokenization rather than k-mer tokenization or BPE subwords. NucEL’s vocabulary contains 27 tokens: 4 nucleotides, 7 special tokens, and 16 reserved tokens. In contrast, a 6-mer vocabulary scales as $\mathcal{X} = \{x_1, x_2, \ldots, x_N\},$3, while BPE vocabularies are described as typically lying in the 5k–8k range. The justification given is threefold: fine-grained resolution at the level of individual bases, improved interpretability of attention and feature analyses, and robustness to variants because token boundaries do not change under single-base mutations (Ding et al., 15 Aug 2025).

The tradeoff is longer token sequences. NucEL addresses this by adopting a ModernBERT-style encoder with hybrid local-global attention and flash attention. The generator has 11 transformer layers, hidden size 256, and 8 attention heads. The discriminator has 22 transformer layers, hidden size 512, and 16 attention heads. Each layer uses local attention with window size 128, and every third layer additionally introduces global attention. Flash attention v2 is used for all attention computations. The local attention pattern is described by

$\mathcal{X} = \{x_1, x_2, \ldots, x_N\},$4

Pre-training uses the human reference genome GRCh38/hg38. Data are extracted as overlapping sliding windows of length 1224 bp with 100 bp overlap, from which 1024 bp segments are randomly sampled during training. Optimization uses AdamW with learning rate $\mathcal{X} = \{x_1, x_2, \ldots, x_N\},$5, $\mathcal{X} = \{x_1, x_2, \ldots, x_N\},$6, $\mathcal{X} = \{x_1, x_2, \ldots, x_N\},$7, a 1000-step warmup schedule, max gradient norm 1.0, global batch size 192, FP16 mixed precision, and 50 epochs on 8 NVIDIA A100 GPUs. For downstream tasks, the discriminator is fine-tuned end-to-end with a linear classification head on the final-layer [CLS] representation (Ding et al., 15 Aug 2025).

These choices collectively define what might be called an “ELECTRA-at-base-resolution” design. This suggests that NucEL’s efficiency claim does not derive from shorter sequences, but from denser supervision and optimized attention.

4. Empirical performance on genomic benchmarks

NucEL is evaluated on three benchmark suites: GUE, GB, and the revised Nucleotide Transformer benchmark. Across these evaluations, the paper reports that a 93M-parameter discriminator surpasses similarly sized MLM-based models and rivals models 25 times larger.

Benchmark	Scope	Headline result
GUE	promoter, splice, TF binding, epigenetic, viral tasks	Average 75.16
GB	seven classification tasks	Average accuracy 0.899
NT benchmark	histone marks, enhancers, promoters, splice tasks	Average MCC 0.664

On GUE, NucEL reports an overall average performance of 75.16, compared with 72.94 for NT-multi-2.5B and 72.11 for DNABERT-2. Selected task-level results include CPD at 75.13 MCC, SSP at 90.30 MCC, TF-M at 70.62 MCC, EMP at 65.01 MCC, and CVC at 70.29 F1. The paper notes that NucEL is trained only on the human genome, yet generalizes well to mouse, yeast, and viral tasks (Ding et al., 15 Aug 2025).

On Genomic Benchmarks, NucEL reports average accuracy 0.899, compared with 0.890 for NT2-100M, 0.882 for Caduceus-Ph, and 0.863 for HyenaDNA. Selected task results include Human vs Worm at $\mathcal{X} = \{x_1, x_2, \ldots, x_N\},$8, Human Enhancer Ensembl at $\mathcal{X} = \{x_1, x_2, \ldots, x_N\},$9, and Human Non-TATA Promoters at $\mathcal{X}_{\text{masked}}$0.

On the revised Nucleotide Transformer benchmark, NucEL reports average MCC 0.664, compared with 0.651 for NT2-Multi (100M), 0.634 for NT-HumanRef (500M), 0.628 for DNABERT2, and about 0.661 for NT-multi-2.5B. Selected results include Enhancer at 0.578, Enhancer types at 0.536, Promoter TATA at 0.922, and Splice donor at 0.970. The paper explicitly states that NucEL matches or slightly exceeds NT-multi-2.5B while having 27 times fewer parameters (Ding et al., 15 Aug 2025).

A recurrent theme is that performance gains are not limited to one benchmark family. This supports the interpretation that the combination of RTD, single-nucleotide tokenization, and ModernBERT-style efficiency is broadly effective rather than narrowly tuned to a single task class.

5. Interpretability and representational analysis

Beyond predictive accuracy, the paper emphasizes interpretability. One analysis uses embeddings for Ensembl gene biotypes, visualized with t-SNE and evaluated using an XGBoost classifier. NucEL’s representations are described as forming clearer clusters than those of HyenaDNA, DNABERT2, and NT2-100M. Macro, micro, and weighted F1 scores for biotype classification are reported as 0.7306, 0.7603, and 0.7490 for NucEL, compared with lower macro F1 values for the comparison models (Ding et al., 15 Aug 2025).

A second analysis uses a synthetic motif-order task in which sequences contain two motifs, A and B, and the label depends on whether A appears before B or vice versa. NucEL and NT2-100M both achieve near-perfect accuracy after fine-tuning, but their attention maps differ substantially. NucEL’s global-attention layers show sharper, more localized peaks at motif positions, while NT2-100M exhibits more diffuse attention with higher background. Quantitatively, NucEL’s maximum signal-to-noise ratio is reported as 65% higher for motif A and 152% higher for motif B.

The paper interprets this as evidence that NucEL captures biologically relevant motifs more cleanly. Because the model operates at single-nucleotide resolution, attention weights can be mapped directly to exact base positions rather than to k-mer or subword units. This is presented as a practical interpretability gain, especially for motif discovery and regulatory grammar analysis.

A common misconception in genomic language modeling is that interpretability necessarily worsens when pre-training objectives become more discriminative and less generative. The reported attention and biotype analyses do not support that view here. Instead, the discriminator-based setup is presented as yielding more localized and biologically coherent attention patterns (Ding et al., 15 Aug 2025).

6. Ablations, limitations, and research directions

The paper includes several ablation-style conclusions. Single-nucleotide tokenization consistently outperforms 6-mer and BPE tokenization under the NucEL framework, particularly on tasks involving fine regulatory detail such as promoters and splice sites. The stated reasons are that 6-mer tokenization can dilute single-base context and that BPE can break motifs across subword boundaries and tokenize closely related sequences inconsistently when mutations alter segmentation. This directly challenges the assumption that sequence compression is necessarily the best route to efficient genomic pre-training (Ding et al., 15 Aug 2025).

The efficiency–performance tradeoff is also treated explicitly. The paper states that NucEL-93M occupies a “sweet spot,” outperforming much larger MLM-based models while using much less compute. This does not imply that scale is irrelevant; rather, it suggests that objective design and tokenization can materially shift the scaling frontier.

The authors also identify limitations. NucEL is pre-trained only on human data. Its effective context in this work is about 1 kbp, which may be insufficient for tasks requiring very long-range interactions such as enhancer–promoter contacts over tens of kb or 3D chromatin architecture. The model operates on linear DNA sequence alone and does not integrate structural variants, epigenetic marks, or expression data. The paper further notes that larger ELECTRA-style genomic models have not yet been systematically explored.

These limitations define the immediate research agenda. A plausible implication is that future extensions may involve multi-species pre-training, longer-context architectures, and multi-omic conditioning. Within the scope of the reported results, however, NucEL establishes ELECTRA-style replaced-token detection as an efficient and effective strategy for genomic representation learning, with particular strength in base-resolution modeling and interpretable regulatory sequence analysis (Ding et al., 15 Aug 2025).