Genome Understanding Evaluation (GUE) Benchmark

• The Genome Understanding Evaluation (GUE) benchmark is a standardized suite combining 36 genomic datasets spanning various sequence lengths, species, and tasks.
• It enables direct comparisons among DNA language models by enforcing fixed training, validation, and test splits along with unified evaluation metrics such as MCC and F1 score.
• GUE drives progress in genomics by promoting efficient tokenization methods and parameter-efficient model designs, yielding improved performance and reduced computational costs.

The Genome Understanding Evaluation (GUE) benchmark is a comprehensive, standardized multi-dataset suite designed to enable rigorous, reproducible, and equitable evaluation of genome-scale machine learning models. GUE encompasses tens of datasets spanning a wide range of genomic classification tasks, several species, and diverse input sequence lengths, providing a basis for direct head-to-head comparison among DNA LLMs and related architectures (Zhou et al., 2023).

1. Formal Definition and Motivation

The motivation for GUE arises from the historical heterogeneity in benchmarking within genomic sequence modeling. Previous approaches aggregated datasets with inconsistent annotation, window sizes, preprocessing steps, or data splits, impeding fair comparison between models. GUE remedies these issues by introducing a unified benchmark—formally, a collection of 36 datasets drawn from consortia such as ENCODE, EPDnew, GenBank, and GISAID, partitioned into nine biologically- and computationally-salient tasks across four species. Each dataset is split into fixed training, validation, and test subsets. Input sequence lengths vary considerably (from 70 to 10,000 base pairs), mirroring the diversity in biological contexts (Zhou et al., 2023).

The GUE construction guarantees that, for each task $t$ and dataset $i$, the collection $$\mathcal{D}_{t,i}=\{(x_n^{(t,i)}, y_n^{(t,i)})\}_{n=1}^{|\mathcal{D}_{t,i}|}$$ is strictly partitioned:

$\mathcal{D}_{t,i} = \mathcal{D}^{\mathrm{train}}_{t,i} \cupdot \mathcal{D}^{\mathrm{valid}}_{t,i} \cupdot \mathcal{D}^{\mathrm{test}}_{t,i}$

This design enforces reproducibility and comparability across studies (Zhou et al., 2023).

2. Composition: Task and Dataset Breakdown

GUE’s nine tasks encompass both regulatory/functional genomics and broader evolutionary questions, structured as follows:

Task	Num. Datasets	Species	Classes	Input Length (bp)
Core Promoter Detection	3	Human	2	70
Proximal Promoter Detection	3	Human	2	300
TF Binding (ENCODE)	5	Human	2	100
TF Binding	5	Mouse	2	100
Splice Site Prediction	1	Human	3 (donor/acceptor/–)	400
Epigenetic Marks	10	Yeast	2	500
Covid Variant Classification	1	Virus	9	1000
Enhancer–Promoter Interaction	6	Human	2	5000
Species Classification	2	Fungi/Virus	25/20	5000/10000

Within each group, the training, validation, and test splits are fixed a priori. This structure precludes cross-validation or re-sampling, ensuring consistent evaluation. Sequence labeling is standardized: for example, all core promoter and enhancer-promoter datasets use promoter/non-promoter classes, while Covid variant classification employs the nine major lineages (Alpha–Zeta), and multi-class species identification is executed for fungal and viral genomes (Zhou et al., 2023).

3. Evaluation Protocol and Metrics

Each GUE dataset is treated as a supervised classification task, with metric choices tailored to task structure:

• Matthews Correlation Coefficient (MCC): Used for binary and three-class tasks (e.g., histone marks, promoters, splice sites):

$$\mathrm{MCC} = \frac{TP \times TN - FP \times FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}$$

• F1 Score: Applied in the multi-class Covid variant classification:

$$F_1 = 2 \cdot \frac{\mathrm{precision} \cdot \mathrm{recall}}{\mathrm{precision} + \mathrm{recall}}$$

• No "macro-averaged" metric is used except in explicitly multi-label settings (e.g., in Lingo's extension). Each model is trained using only the training split, validated on the fixed validation split, and final results are reported on the held-out test split after hyperparameter selection (Zhou et al., 2023); (Zhan et al., 2024).

Hyperparameter regimes are standardized to facilitate head-to-head comparisons (e.g., AdamW optimizer, batch size 32, learning rate $3 \times 10^{-5}$ for DNABERT/DNABERT-2, up to 10 epochs for certain datasets) (Zhou et al., 2023).

4. Tokenization and Input Representation

Tokenization strategy in GUE is a salient variable for model efficiency and accuracy. Three main approaches are benchmarked:

• Overlapping $k$-mers: Employed in DNABERT, resulting in nearly one token per nucleotide but suffering from masked language modeling leakage and substantial increase in input length.
• Non-overlapping $k$-mers: Used in Nucleotide Transformer; reduces computational cost but is less robust to positional shifts.
• Byte Pair Encoding (BPE): Used in DNABERT-2, which automatically learns variable-length motifs, reducing token count by a factor of $\sim5\times$, mitigating information leakage, and empirically improving downstream accuracy (Zhou et al., 2023).

Sequence representations are strictly upper-cased (A/C/G/T only), with non-canonical bases either masked or discarded. All input sequences are fixed length per dataset; BPE vocabulary is typically $i$0 (Zhou et al., 2023).

5. Model Baselines and Comparative Performance

GUE enables benchmarking of a range of model classes, particularly:

• DNABERT ($i$186–89M parameters, $i$2-mer input)
• Nucleotide Transformer (500M–2.5B parameters, $i$3-mer input)
• DNABERT-2 (117M parameters, BPE input)
• HyenaDNA, Caduceus, SpliceAI, DeepSTARR, Orca (various convolutional or hybrid architectures)

On the aggregate GUE test sets (28 representative datasets), DNABERT-2 achieves an average MCC/F1 of 66.80, nearly matching the largest Nucleotide Transformer at 66.93 with $i$4 fewer parameters and $i$5 less GPU time for pretraining. DNABERT-2 outperforms DNABERT on 23/28 datasets by an average of 6 MCC points while being 3$i$6 more efficient (Zhou et al., 2023).

In comparative studies, hyperbolic convolutional neural networks (HCNNs) further surpassed state-of-the-art performance on seven GUE datasets, consistently outperforming large DNA LLMs with orders of magnitude fewer parameters and without pretraining (Khan et al., 29 Jul 2025).

6. Extensions, Alternative Benchmarks, and GUE in Broader Context

Several subsequent works have adopted or extended the GUE benchmark:

• Lingo evaluates genome understanding using efficient parameter-efficient fine-tuning (PEFT) of LLMs on 14 GUE datasets. It leverages byte-level BPE tokenization and combines language prefixes with low-rank adapters, outperforming other PEFT schemes while tuning less than 2% of model parameters (Zhan et al., 2024).
• Hyperbolic Genome Embeddings employ the GUE datasets to demonstrate that hyperbolic inductive biases yield marked gains on tasks with underlying hierarchical or tree-like biological structure, such as transcription factor binding and histone marks, but offer less advantage for tasks with more combinatorial or shallow hierarchy (e.g., promoter detection, Covid classification) (Khan et al., 29 Jul 2025).
• GenBench is a complementary comprehensive benchmarking suite, but focuses on short- and long-range genomic modeling and supports a wider range (43) of real-world datasets, with similar emphasis on modular, standardized evaluation (Liu et al., 2024).

A plausible implication is that the GUE benchmark has become a de facto standard for foundation model evaluation in genomics, driving advances in model and tokenizer efficiency, and stimulating rigorous evaluation of new architectures.

7. Impact, Recommendations, and Future Directions

GUE standardization enables biologically meaningful, computationally efficient progress in genome NLP by:

• Providing a multi-task, multi-species yardstick for fair, interpretable comparison of genome foundation models.
• Driving research towards more efficient tokenization (BPE over $i$7-mers) and more parameter-efficient models.
• Allowing rigorous statistical comparison and meta-analytic integration across architectures, parameter scales, and species.

Recommendations arising from GUE-focused research include the adoption of BPE tokenization for sample and compute efficiency, carefully tuned vocabulary sizes, and the use of hyperbolic models for tasks that exploit evolutionary or hierarchical sequence structures (Zhou et al., 2023); (Khan et al., 29 Jul 2025). Future work may further extend the benchmark to broader taxonomic coverage, add sequence-to-sequence and generative tasks, and incorporate additional metrics and modalities reflecting the rapidly advancing genomics landscape.

---

References:

(Zhou et al., 2023) DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome (Zhan et al., 2024) Efficient and Scalable Fine-Tune of LLMs for Genome Understanding (Khan et al., 29 Jul 2025) Hyperbolic Genome Embeddings (Liu et al., 2024) GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Models