- The paper introduces a unified benchmark evaluating 40 genomic foundation models across 100 tasks and 13 functional categories.
- It demonstrates that model performance crucially depends on architecture, tokenization, and pretraining data, highlighting context-specific advantages.
- The study reveals that scaling alone is insufficient, as well-aligned, smaller models can outperform larger counterparts in specific genomic tasks.
GENEB: A Critical Analysis of Model Comparisons in Genomic Representation Learning
Introduction
The proliferation of foundation models in genomics has rendered the unbiased and replicable comparison of such models an open methodological problem. "GENEB: Why Genomic Models Are Hard to Compare" (2606.04525) addresses this fragmentation by introducing a rigorously designed, large-scale benchmark evaluating 40 genomic foundation models across 100 tasks and 13 distinct functional categories. By employing a unified probing protocol, GENEB isolates and systematically interrogates the influence of model architecture, parameterization, tokenization, and pretraining corpus on downstream performance, with particular emphasis on robustness and transfer learning in low-data regimes.
Motivations and Context
Despite rapid advances in architecture—from Transformer encoders and decoders to SSMs (e.g., Mamba, Hyena)—and in data representation, genomics lacks the infrastructure for comprehensive benchmarking analogous to MTEB in NLP. Current practices are dominated by single-paper leaderboards and incompatible evaluation sets, resulting in ambiguous and often non-comparable claims of generality or superiority. As shown in GENEB’s analysis, state-of-the-art status is often context-specific: a model heralded as a breakthrough on one label set may underperform elsewhere, and this discrepancy reflects evaluation protocol heterogeneity rather than model inconsistency.
Many prior benchmarks—Nucleotide Transformer [Dalla-Torre et al., 2023], GUE/GUE+ [Zhou et al., 2024a], BEND [Marin et al., 2024], DNALongBench [Cheng et al., 2025]—are limited in model coverage, domain, or task diversity, and none supports systematic, cross-model evaluation at the current scale or complexity of genomic modeling.
Unified Benchmark Design
GENEB’s core methodological innovation is a strictly controlled, category-aware comparative framework. All models are evaluated with frozen sequence embeddings under logistic regression probes—providing clarity in assessing learned representations—across the same task suite, preprocessing protocol, and evaluation metric (macro-averaged MCC). Robustness to probe nonlinearity and regularization is empirically verified, establishing that reported model rankings reflect underlying feature quality rather than protocol artifact.
Tasks are drawn from an amalgam of existing resources, covering histone modifications, regulatory region identification, epigenetic site classification, species discrimination, and more, with task sizes and class distributions systematically balanced. Tokenization regimes span single-nucleotide, k-mer, BPE, and biologically-informed vocabularies. Pretraining data encompasses taxonomically-restricted sets (human, plant, prokaryote), broad multi-species corpora, and domain-centric profiles (e.g., chromatin accessibility).
Numerical Results and Comparative Findings
Parameter Scaling and Architecture
Aggregate analysis confirms a moderate but inconsistent correlation between log-parameter count and full-shot macro-MCC (ρ = 0.565; rising to 0.685 when controlling for domain-mismatched outliers; both p < 0.001). However, size is a necessary but insufficient predictor of performance. Across 36 "in-domain" models, there are 31 cases in which a model at least 5x smaller outperforms a larger counterpart—a pattern robust to micro- or macro-aggregation.
Controlled pairwise analyses reveal that architecture and pretraining are often determinative:
- Transformer models consistently outperform SSMs (Mamba/Hyena) and hybrids under matched pretraining and tokenization, especially on cross-species regulatory and splicing tasks.
- Transformer encoders and decoders exhibit task-dependent trade-offs, with encoders favored on TF binding/species classification, and decoder parity or advantage elsewhere.
- SSMs’ only prominent advantage is on chromatin accessibility (e.g., ECCDNAMAMBA > GenomeOcean-500M by +0.124 MCC), suggesting that specific inductive biases may confer value in certain biological contexts.
Tokenization Regimes
No universal best tokenization emerges. BPE marginally outperforms k-mer within multi-species Transformer-decoders (+0.02 macro-MCC on average, but with high variance). Single-nucleotide is notably strong in human-pretrained encoders for chromatin accessibility and splice tasks. BioToken (structurally-annotated) and hybrid vocabularies provide isolated boosts within certain models, but gains are inconsistent outside domain-aligned contexts.
Pretraining Corpus
Pretraining corpus taxonomic scope is a decisive axis:
- Cross-domain negative transfer is severe. Models pretrained exclusively on prokaryotic genomes (Evo-1-131K, 7B params) are systematically outperformed by low-capacity, human-trained models (e.g., MutBERT, 86M params) by +0.231 macro-MCC, despite an 81-fold parameter disadvantage—underscoring that scaling cannot compensate for domain misalignment.
- Multi-species pretraining consistently outperforms human-only pretraining for eukaryotic tasks, with the largest relative advantage for chromatin accessibility (+0.062), splicing (+0.038), species classification (+0.031), and IncRNA classification (+0.022).
- Plant-specific and eukaryotic gene-focused pretraining yields strong category-specific or global gains, notably for plant lncRNA (LUCAONE: 0.508 MCC; next-best plant-specific: PLANTCADUCEUS: 0.357 MCC).
- Category-level analysis shows that microbial pretraining transfers poorly to eukaryotic regime tasks, with multi-species pretraining outperforming microbe-centric models by sizable margins on splicing, IncRNA, and methylation tasks.
Few-Shot and Robustness Analysis
Macro-MCC for all models collapses by 48% (10-shot) and 78% (1-shot) relative to full-data, but this degradation is not monotonic in model size. Smaller or underperforming models may show less absolute drop due to low full-shot ceiling—not superior generalizability. Importantly, the top-performing model under full supervision is not always optimal in few-shot regimes; eight of 13 categories rerank the top-1 model entirely under 10-shot supervision.
Task Difficulty Spectrum
Approximately 28 tasks have mean MCC < 0.35, even for top models, with 4mC methylation and plant IncRNA classification particularly challenging (maximum model MCCs 0.206–0.508). Scaling is weak or insignificant for these "hard frontiers", indicating that further progress likely requires advances in curation, inductive bias incorporation, or explicit task supervision.
Theoretical and Practical Implications
GENEB’s findings starkly demonstrate the limitations of aggregate leaderboards and naïve scaling in genomics. Optimal model selection is intrinsically category- and regime-dependent. No single architectural or tokenization regime dominates globally; rather, careful alignment of pretraining data domain, architecture, and representation is required. Failures of transfer across taxonomic boundaries highlight the risk of domain-agnostic scaling strategies.
For the community, GENEB provides a robust diagnostic reference point, operationalizing best practices for controlled benchmarking, facilitating principled ablation studies, and providing actionable recommendations for both practitioners and model designers.
- For large-scale or bespoke deployments, the category-aware guidance allows targeted model selection.
- For foundation model researchers, GENEB specifies which architectural, tokenization, and pretraining decisions yield genuine, not just parameter-driven, generalization or specialization.
Limitations
GENEB does not exercise very long-context models on appropriately long-range regulatory tasks (>10kb), due to limitations in fair context-length matching and available datasets. There is a marked skew towards eukaryotic tasks, and models trained exclusively on prokaryotic or viral genomes are at a structural disadvantage. Furthermore, frozen, linear probing may not capture the capacity revealed under full fine-tuning protocols for certain models. Finally, ~25% of surveyed models could not be included due to insufficient open code or weights, or insurmountable computational requirements.
Future Directions
GENEB paves the way for longitudinal evaluation across future modeling paradigms and extended biological tasks—including direct 3D chromatin conformation prediction and genome-scale enhancer-promoter interaction tasks. Expanding model inclusion (conditioned on open science best practices), refining task curation, and integrating downstream finetuning benchmarks will further strengthen the field’s empirical foundations.
Enhancing tokenization approaches to better capture regulatory grammar, and designing architectures optimized for the stochastic, context-rich nature of genomic sequences (as opposed to analogies with NLP), remain open and promising areas for future development.
Conclusion
GENEB constitutes a rigorous reference point for genomic foundation model evaluation, demonstrating that claims of universal superiority are fragile in the absence of controlled, multi-category benchmarking. It disambiguates the interplay of architecture, scale, pretraining, and tokenization, exposing both methodological pitfalls and avenues for rational model development. The resource sets a methodological standard for the field and is a crucial tool for advancing task-specific and domain-aligned genomic machine learning research (2606.04525).
