GENEB: Genomic Benchmark for Models

• The paper introduces GENEB to standardize evaluation by using frozen representations and a unified diagnostic protocol across 100 classification tasks.
• GENEB aggregates tasks from 13 functional categories, offering clear insights into task-level trade-offs and enabling category-aware model selection.
• Key findings reveal that no model is universally best, with architecture, tokenization, and pretraining data significantly impacting performance.

Searching arXiv for the current usage of “GENEB” and related papers to ground the article. GENEB is a large-scale diagnostic benchmark for genomic foundation models that was introduced to address fragmented benchmarks, incompatible evaluation protocols, and task-specific reporting in genomic machine learning. It evaluates frozen representations from 40 genomic foundation models across 100 classification tasks spanning 13 functional categories under a unified probing-based protocol, including 1-shot, 10-shot, and full-data settings. Its central methodological claim is that many existing cross-paper assertions of superiority or generality are not directly comparable because models are typically evaluated under inconsistent task collections, preprocessing choices, splits, metrics, and downstream training setups. GENEB therefore functions as a shared reference framework for principled comparison and category-aware model selection rather than as a single aggregate leaderboard (Ledneva et al., 3 Jun 2026).

1. Definition and methodological role

GENEB was proposed for a specific comparability problem: genomic models are difficult to compare because the field relies on fragmented benchmarks, incompatible evaluation protocols, task-specific reporting, and sparse baseline overlap. Under such conditions, a model can appear state-of-the-art in one study and weak in another without any genuine contradiction, because the evaluation settings differ. GENEB addresses this by enforcing one common representation-evaluation pipeline across a broad set of tasks and models (Ledneva et al., 3 Jun 2026).

The benchmark is explicitly diagnostic. Its purpose is not merely to declare an overall winner, but to expose task-level trade-offs and to show where aggregate narratives are unstable or misleading. This suggests that GENEB treats evaluation as an analysis problem as much as a ranking problem: the full model-by-task performance matrix is more informative than any single summary score.

A central design choice is the use of frozen representations. Each pretrained model is kept fixed, embeddings are extracted, and a lightweight downstream probe is trained on top of those embeddings. This isolates representation quality from task-specific fine-tuning strategies. The paper frames that choice as a control mechanism: GENEB is intended to compare what the pretrained model already encodes, not the best possible result obtainable after extensive downstream adaptation (Ledneva et al., 3 Jun 2026).

2. Scope of the benchmark and biological coverage

GENEB evaluates 40 genomic foundation models on 100 classification tasks grouped into 13 functional categories. The task collection is aggregated from nine prior benchmark or data sources: NT, NT-rev, GUE, Genomic Benchmarks, Plant Genomic Benchmark, iPro-WAEL, deep4mc, iDNA-ABF, and iDHS-EL. The aggregation is deliberate: it broadens coverage while preserving comparability to prior work (Ledneva et al., 3 Jun 2026).

The 40 models span several architectural families. The benchmark includes 15 Transformer-encoder models, 13 Transformer-decoder models, 4 Mamba-SSM models, 2 Hybrid-Mamba-MoE models, 2 Hyena models, 2 CNN-Transformer hybrids, 1 Graph-Transformer, and 1 StripedHyena. Parameter counts range from under 2M to 7B. Tokenization schemes include 13 k-mer models, 13 BPE models, 11 single-nucleotide models, and 3 custom schemes: BioFM’s BioToken, LucaOne’s mixed nucleotide/amino-acid vocabulary, and Genomics-FM’s BPE+k-mer ensemble. Pretraining corpora include 20 multi-species models, 12 human-only models, 2 eukaryotic-gene models, 2 plant-genome models, 2 human-mouse epigenomic-profile models, 1 prokaryotic model, and 1 multi-species microbial model (Ledneva et al., 3 Jun 2026).

The benchmark’s 13 categories are shown below.

Category	Tasks	Representative coverage
Histone Modifications	30	H3, H4, H3K4me1/2/3, H3K27ac, H3K27me3
Promoters	22	human TATA/no-TATA/all, bacterial, plant, cell-type-specific
Enhancers	8	NT enhancer classification, human, mouse, Drosophila
DNA Methylation	8	4mC across six species, 5mC, 6mA
Splice Sites	7	donor, acceptor, combined, reconstructed splice-site
lncRNA	6	soybean, cassava, sorghum, tomato, wheat, maize
Mouse Enhancers	5	tissue-specific GUE mouse enhancer tasks
TF Binding	5	human transcription factor binding
Species Classification	3	human vs worm, fungi-20, virus-40
Regulatory	2	Ensembl regulatory region, open chromatin region
Virus/Phage	2	phage fragment identification, COVID variant classification
Coding/Non-coding	1	coding vs intergenic/non-coding sequence
Chromatin Accessibility	1	DNase-I hypersensitive site prediction

These tasks span regulatory element prediction, epigenomic state prediction, RNA or transcript-related tasks, sequence-type classification, and cross-species or evolutionary classification. At the same time, the benchmark is heavily skewed toward eukaryotic genomics: 12 of 13 categories are eukaryotic, while virus/phage classification is the only explicitly non-eukaryotic category. That asymmetry becomes important when interpreting the performance of microbial- or prokaryote-oriented models (Ledneva et al., 3 Jun 2026).

3. Evaluation protocol, metrics, and aggregation

GENEB uses a unified probing-based protocol. The procedure is fixed: take a pretrained model, keep it frozen, extract sequence embeddings, use those embeddings as features, and train the same simple downstream probe for each task. The main probe is logistic regression with max_iter = 1000, evaluated in 1-shot, 10-shot, and full-data regimes, with results averaged across five fixed random seeds: $\{13, 17, 42, 123, 997\}$ (Ledneva et al., 3 Jun 2026).

The benchmark uses mean pooling throughout when extracting frozen representations. The paper notes that this interacts with tokenization, because single-nucleotide and k-mer tokenizations yield longer token sequences than BPE on the same input window. A plausible implication is that some apparent tokenization effects may partly reflect pooling-tokenization interactions rather than tokenization alone.

The primary evaluation metric is Matthews Correlation Coefficient (MCC), chosen because it is robust to class imbalance and standard in genomic evaluation. GENEB reports both macro-averaged MCC and micro-averaged MCC, but treats macro-MCC as the principal benchmark score. Macro-MCC is formed by averaging MCC within each of the 13 categories and then averaging those category scores equally:

$$\mathrm{MacroMCC}_m = \frac{1}{13} \sum_{c=1}^{13} s_{m,c}.$$

Micro-MCC is the average over all 100 tasks:

$$\mathrm{MicroMCC}_m = \frac{1}{100} \sum_{t=1}^{100} \mathrm{MCC}_{m,t}.$$

Macro averaging is preferred because micro averaging would overweight Histone Modifications and Promoters, which together account for more than half of all tasks. Even so, the two rankings are very similar, with Spearman correlation $\rho = 0.988$ and mean absolute shift $|\Delta| = 0.009$ MCC, although out-of-domain and specialized models are more sensitive to the averaging choice (Ledneva et al., 3 Jun 2026).

GENEB also formalizes specialization. If $r_{m,t} \in \{1,\dots,40\}$ is the rank of model $m$ on task $t$, then for category $C$ with task set $T_C$,

$$\mathrm{MacroMCC}_m = \frac{1}{13} \sum_{c=1}^{13} s_{m,c}.$$0

$$\mathrm{MacroMCC}_m = \frac{1}{13} \sum_{c=1}^{13} s_{m,c}.$$1

and the specialization score is

$$\mathrm{MacroMCC}_m = \frac{1}{13} \sum_{c=1}^{13} s_{m,c}.$$2

Positive $$\mathrm{MacroMCC}_m = \frac{1}{13} \sum_{c=1}^{13} s_{m,c}.$$3 indicates relative strength on category $$\mathrm{MacroMCC}_m = \frac{1}{13} \sum_{c=1}^{13} s_{m,c}.$$4, and values above about 5 indicate substantial relative strength. Reported examples include BioFM-265M on DNA methylation with $$\mathrm{MacroMCC}_m = \frac{1}{13} \sum_{c=1}^{13} s_{m,c}.$$5, PlantCaduceus on lncRNA with $$\mathrm{MacroMCC}_m = \frac{1}{13} \sum_{c=1}^{13} s_{m,c}.$$6, JanusDNA-72-W on chromatin accessibility with $$\mathrm{MacroMCC}_m = \frac{1}{13} \sum_{c=1}^{13} s_{m,c}.$$7, and eccDNAMamba on chromatin accessibility with $$\mathrm{MacroMCC}_m = \frac{1}{13} \sum_{c=1}^{13} s_{m,c}.$$8 (Ledneva et al., 3 Jun 2026).

4. Principal empirical findings

The benchmark’s headline result is that aggregate leaderboards are unstable. No single model dominates the benchmark; the top model wins only 20 of 100 tasks, and the remaining task wins are distributed across 15 additional models. Rankings change sharply across categories, and the full-shot winner is reranked under 10-shot in 8 of 13 categories. This directly challenges the common assumption that a single overall ranking adequately summarizes genomic model quality (Ledneva et al., 3 Jun 2026).

A second major result is that rankings vary sharply by biological category. Representative examples are category-specific rather than universal. Enformer is only moderate in overall rank, yet leads TF binding with 0.698, enhancers with 0.539, and regulatory with 0.604, while also being near-top on mouse enhancers and chromatin accessibility. The GenomeOcean family is especially strong on virus/phage, with GenomeOcean-4B at 0.697 and GenomeOcean-500M at 0.657. PlantCaduceus and Agro-NT-1B show relative strength on plant lncRNA, and MUTBERT is a strong compact model and the top sub-100M model in many categories (Ledneva et al., 3 Jun 2026).

Scale helps, but only modestly and inconsistently. The benchmark-wide correlation between log-parameter count and macro-MCC is $$\mathrm{MacroMCC}_m = \frac{1}{13} \sum_{c=1}^{13} s_{m,c}.$$9; excluding the out-of-domain prokaryotic Evo-1-131K, it rises to $$\mathrm{MicroMCC}_m = \frac{1}{100} \sum_{t=1}^{100} \mathrm{MCC}_{m,t}.$$0. Yet among the 36 in-domain models there are 31 cases where a model at least 5× smaller beats a larger one in aggregate macro-MCC. MutBERT, at 86M, exceeds eccDNAMamba, at 1B, by +0.110 macro-MCC. Models above 1B average +0.064 macro-MCC above models below 200M, or +0.087 if Evo-1-131K is excluded. Category-level scaling is significant in 11 of 13 categories but non-significant in species classification and chromatin accessibility, indicating that size is neither uniformly beneficial nor sufficient as an explanatory variable (Ledneva et al., 3 Jun 2026).

Architecture often matters more than scale under controlled comparisons. GENEB defines 30 controlled pairs: 9 architecture comparisons, 9 pretraining-data comparisons, and 12 tokenization comparisons. In matched settings, Transformers generally outperform the evaluated state-space alternative. Omni-DNA-1B exceeds eccDNAMamba by 0.568 versus 0.419, a gap of +0.149, and GenomeOcean-500M exceeds eccDNAMamba by +0.131. However, encoder versus decoder is not globally ordered; the direction depends on category and pairing. Chromatin accessibility is a notable exception to broad Transformer dominance: eccDNAMamba exceeds GenomeOcean-500M there by +0.124, and JanusDNA-72-W and eccDNAMamba show strong positive specialization on that category (Ledneva et al., 3 Jun 2026).

Tokenization has no universal winner. For multi-species Transformer-decoders, BPE beats k-mer on average by +0.020 macro-MCC across three pairs, but variance is large. For multi-species Transformer-encoders, BPE and k-mer are nearly tied at +0.006 across five pairs, with all pairwise gaps below 0.02 MCC. In two human-pretrained encoder comparisons, single-nucleotide tokenization beats BPE: MutBERT exceeds GENA-LM by +0.033 and GROVER by +0.038. Non-standard vocabularies also behave inconsistently: k-mer exceeds BioFM’s BioToken by +0.134 in one decoder comparison, while LucaOne’s mixed vocabulary exceeds k-mer by +0.017 in one encoder comparison. The paper therefore rejects any universal tokenization hierarchy (Ledneva et al., 3 Jun 2026).

Pretraining-data alignment is frequently more important than parameter count. Across six controlled human-versus-multi-species pairs, multi-species pretraining has an overall advantage of +0.012 macro-MCC, with especially clear gains on chromatin accessibility (+0.062, winning 6 of 6), splice sites (+0.038), species classification (+0.031), mouse enhancers (+0.023), and lncRNA (+0.022). Virus/phage is the main category favoring human-only pretraining at -0.034. The largest controlled pretraining effect is multi-species versus microbial-focused pretraining, where general multi-species pretraining leads by +0.084 macro-MCC on average, with large category gaps in splice sites (+0.222), species classification (+0.130), lncRNA (+0.116), and DNA methylation (+0.108). A single matched pair, GENERATOR-EUKARYOTE-3B versus DNA-GPT-3B-M, shows +0.063 overall for eukaryotic-gene pretraining, with particularly large gains in chromatin accessibility, lncRNA, and mouse enhancers (Ledneva et al., 3 Jun 2026).

5. Few-shot behavior, diagnostic tasks, and practical interpretation

Few-shot evaluation is integral to GENEB rather than supplementary. Average benchmark macro-MCC drops from 0.488 in full-shot to 0.253 in 10-shot and 0.106 in 1-shot, corresponding to relative reductions of 48.2% and 78.2%. Low-shot rankings can differ substantially from full-shot rankings, and the paper argues that few-shot leaderboards are especially easy to misread because weaker models may appear comparatively robust simply because they start from a lower ceiling (Ledneva et al., 3 Jun 2026).

The reranking effect is category-specific and often large. In coding/non-coding, the full-shot leader is Generator-Eukaryote-3B at 0.904, while the 10-shot leader is MutBERT at 0.748. In mouse enhancers, the full-shot leaders are Omni-DNA-1B at 0.675 and Enformer at 0.674, but SPACE leads at 10-shot with 0.379. In splice sites, NT-2.5B-MS at 0.652 and Generator-Eukaryote-3B at 0.648 lead in full-shot, whereas LucaOne leads at 10-shot with 0.298. In enhancers, Enformer leads in full-shot with 0.539, but GENA-LM-Large-T2T leads in 10-shot with 0.372. These examples reinforce the benchmark’s practical recommendation that model choice should be conditioned on the supervision regime as well as the biological task family (Ledneva et al., 3 Jun 2026).

GENEB also identifies hard tasks that remain unsolved. Twenty-eight of the 100 tasks have mean MCC below 0.35. Among the hardest are 4mC methylation tasks such as G. subterraneus at 0.061, E. coli at 0.103, and G. pickeringii at 0.107, as well as plant lncRNA tasks such as S. lycopersicum at 0.221, G. max at 0.228, and T. aestivum at 0.238. Even the best category-level scores remain limited, with DNA methylation topping out at 0.440 for Generator-Eukaryote-3B and plant lncRNA at 0.508 for LucaOne. The paper interprets this as evidence that progress on difficult genomic tasks will require more than scaling, including better pretraining corpora, inductive biases, and task-specific supervision (Ledneva et al., 3 Jun 2026).

High-variance tasks are especially diagnostic. GENEB identifies 13 tasks with cross-model standard deviation above 0.12. On these tasks, multi-species and eukaryotic-gene pretraining account for 32 of 39 top-3 placements, while human-only pretraining accounts for 29 of 39 bottom-3 placements; prokaryotic and microbial-focused models are concentrated near the bottom. Architecturally, Transformer-decoders and Transformer-encoders account for 33 of 39 top-3 placements, while Mamba-SSM, Hybrid-Mamba-MoE, and StripedHyena dominate bottom placements. This suggests that diagnostic tasks reveal decisive design differences more clearly than benchmark-wide averages do (Ledneva et al., 3 Jun 2026).

Representative category leaders further underscore the absence of a universal best model. Reported leaders include Generator-Eukaryote-3B on coding/non-coding (0.904), promoters (0.774), chromatin accessibility (0.728), and DNA methylation (0.440); GenomeOcean-4B on species classification (0.762), virus/phage (0.697), and histone modifications (0.545); Enformer on TF binding (0.698), regulatory (0.604), and enhancers (0.539); Omni-DNA-1B on mouse enhancers (0.675); NT-2.5B-MS on splice sites (0.652); and LucaOne on lncRNA (0.508). The benchmark’s intended use follows from these trade-offs: it should support category-aware model selection and principled comparison of frozen embeddings, especially when fine-tuning is undesirable or impossible (Ledneva et al., 3 Jun 2026).

6. Limitations, interpretive caveats, and acronym ambiguity

GENEB’s conclusions are conditioned by several explicit limitations. Long-range tasks requiring interactions beyond 10 kb are underrepresented, which disadvantages long-context models such as HyenaDNA-Large-1M, Caduceus-PS-131K, Evo-1-131K, and JanusDNA variants. The benchmark uses frozen linear probing, which improves control but may underestimate what models can achieve under task-specific fine-tuning. Mean pooling is used uniformly, so pooling-tokenization confounds are not fully disentangled. The task suite inherits biases from publicly available datasets, and some tasks, especially in DNA methylation and plant lncRNA, may be noisy or weakly defined. The benchmark is strongly eukaryotic, which makes aggregate rankings poor proxies for prokaryotic or viral genomics. Finally, not all genomic foundation models were included because of unavailable weights, missing extraction code, broken infrastructure, or excessive compute requirements (Ledneva et al., 3 Jun 2026).

Probe stability analyses partially mitigate one concern: the benchmark’s conclusions do not appear to be artifacts of using logistic regression. On a tested subset, replacing logistic regression with a one-hidden-layer MLP probe with 256 hidden units, ReLU, and early stopping yields Spearman correlation $$\mathrm{MicroMCC}_m = \frac{1}{100} \sum_{t=1}^{100} \mathrm{MCC}_{m,t}.$$1 across all 143 model-task pairs and $$\mathrm{MicroMCC}_m = \frac{1}{100} \sum_{t=1}^{100} \mathrm{MCC}_{m,t}.$$2 for per-model average MCC, with identical top-3 and top-5 models under both probes. Logistic-regression regularization sweeps using $$\mathrm{MicroMCC}_m = \frac{1}{100} \sum_{t=1}^{100} \mathrm{MCC}_{m,t}.$$3 show that 1-shot rankings are almost invariant, with mean pairwise $$\mathrm{MicroMCC}_m = \frac{1}{100} \sum_{t=1}^{100} \mathrm{MCC}_{m,t}.$$4 and minimum $$\mathrm{MicroMCC}_m = \frac{1}{100} \sum_{t=1}^{100} \mathrm{MCC}_{m,t}.$$5, while 10-shot and full-data rankings are somewhat more sensitive, at mean pairwise $$\mathrm{MicroMCC}_m = \frac{1}{100} \sum_{t=1}^{100} \mathrm{MCC}_{m,t}.$$6 and $$\mathrm{MicroMCC}_m = \frac{1}{100} \sum_{t=1}^{100} \mathrm{MCC}_{m,t}.$$7 respectively. These results support the protocol’s internal stability, although the paper explicitly leaves stability under full fine-tuning as an open question (Ledneva et al., 3 Jun 2026).

The acronym itself also requires disambiguation. In current arXiv usage, GENEB denotes the genomic benchmark described above. A different 2017 paper, “Geometric Enclosing Networks,” introduces GEN rather than GENEB and studies an enclosing-ball-based generative model in which real and generated data are mapped into the same minimal enclosing sphere in feature space. That work is conceptually related only if “GENEB” is informally intended to mean enclosing-ball-based generative modeling; the abbreviation GENEB is not used there (Le et al., 2017).