DGEB Operon Prediction Benchmark

• DGEB Benchmark is a standardized testbed that evaluates whether sequence embeddings can accurately capture operon structure in prokaryotic genomes.
• It employs both unsupervised cosine similarity and a supervised Siamese MLP framework on embeddings from models like ESM-2 and ProtBERT.
• Results demonstrate that embedding-based methods outperform traditional physicochemical baselines, highlighting improvements for genome annotation and regulatory network reconstruction.

The Diverse Genomic Embedding Benchmark (DGEB) is a benchmark for operonic pair classification, introduced to standardize the task across hundreds of microbial genomes by assessing how well sequence embeddings capture true operon structure (Gupta et al., 10 May 2026). In this setting, the prediction target is whether two neighboring genes form an operon. The benchmark is motivated by the central role of operons in prokaryotic gene regulation and by the practical limits of experimental approaches such as RT-PCR and RNA-seq, which provide precise evidence of operon structure but are laborious and largely limited to well-studied model organisms. DGEB therefore serves as a computational testbed for genome-wide operon identification, regulatory network reconstruction, and functional annotation in diverse bacteria and archaea.

1. Biological setting and benchmark task

Operons are clusters of adjacent genes transcribed together in prokaryotes. Predicting whether two neighboring genes form an operon—described as operonic pair classification—is critical for reconstructing regulatory networks, functionally annotating uncharacterized genes, and guiding drug discovery efforts in non-model microbes (Gupta et al., 10 May 2026).

DGEB formalizes this problem at the level of consecutive gene pairs. The benchmark evaluates whether sequence representations preserve the functional relationships needed to distinguish operonic from non-operonic neighbors. This suggests that DGEB is not merely a sequence-similarity benchmark; it is a task-specific probe of whether learned embedding geometry corresponds to transcriptional organization.

The benchmark’s framing is also notable because it places pre-trained protein LLM embeddings in direct comparison with traditional handcrafted descriptors. In the SCOPE study, this comparison is used to test whether modern protein encoders provide a scalable basis for operon prediction across diverse microbial genomes.

2. Dataset composition, labels, and split design

DGEB aggregates data from a large collection of fully sequenced bacterial and archaeal genomes—on the order of hundreds—covering broad phylogenetic diversity (Gupta et al., 10 May 2026). Positive and negative labels are defined from neighboring genes within genomes.

Positive pairs are defined as consecutive gene pairs experimentally validated or conserved as part of the same transcriptional unit in OperonDB (ODB) and carried over into DGEB. Negative pairs are neighboring genes in the same genome that are not annotated to the same operon in ODB. Because most consecutive genes do not share an operon, DGEB exhibits strong class imbalance, with positives forming roughly 20–30 % of all neighbor pairs and negatives comprising the remainder.

The split design is genome-level rather than pair-level. Training and validation pairs were drawn from ODB-derived operons in a subset of genomes, with a held-out set of genomes reserved for final testing on the DGEB benchmark. The SCOPE study employs an implicit single train/validation/test split (no cross-validation across DGEB test genomes), ensuring that test genomes have no overlap with those used for training. This split structure makes cross-genome generalization part of the evaluation problem.

A plausible implication is that DGEB rewards representations that transfer across phylogenetic diversity rather than memorizing within-genome local regularities.

3. Feature representations and benchmark baselines

The SCOPE evaluation on DGEB uses three baseline families: physicochemical feature baselines, protein LLM embeddings, and an unsupervised cosine-similarity baseline (Gupta et al., 10 May 2026).

For the physicochemical baselines, each protein sequence is converted into a 305-dimensional vector capturing amino-acid composition, biochemical group frequencies, hydrophobicity, net charge, molecular weight, and Shannon entropy. A Siamese fusion pattern—concatenation, signed difference, absolute difference, element-wise product—merges the pair’s two 305-D vectors into a 1220-D input. Two off-the-shelf classifiers are then trained in this setup: Logistic Regression (L2 penalty, C=1.0, balanced weights) and XGBoost (300 trees, max_depth=6, learning_rate=0.05, scale_pos_weight set to the class ratio).

For embedding-based baselines, the study uses two frozen encoders as feature extractors: ESM-2 3B (2560-D per-protein embeddings) and ProtBERT-BFD (1024-D per-protein embeddings). Sequence tokens are passed through the transformer, and per-token representations are mean-pooled over sequence length. The resulting per-protein embeddings are fused through the same Siamese interaction pattern to produce 5×D dimensional pair vectors, namely 12,800 for ESM-2 and 5,120 for ProtBERT.

The benchmark also includes an unsupervised baseline: each consecutive gene pair is scored by the cosine similarity of its pre-trained embedding vectors (ESM-2 features), with no supervised classifier. Pairs with higher cosine scores are predicted as operonic. In the SCOPE paper, this baseline is described as the DGEB benchmark evaluation method against which the learned Siamese head is compared.

4. SCOPE model and the meaning of “Siamese Contrastive”

The model evaluated against DGEB in SCOPE is a Siamese MLP operating on frozen protein LLM embeddings (Gupta et al., 10 May 2026). Two input sequences share a frozen transformer encoder, either ESM-2 or ProtBERT. Their mean-pooled embeddings are fused via concatenation, signed difference, absolute difference, and element-wise product, and the fused vector is passed to a multi-layer perceptron (MLP) head with a sigmoid output for binary operonic versus non-operonic classification.

For ESM-2 3B embeddings (12,800-D input), the MLP has four hidden layers [4096 → 1024 → 256 → 64], dropout 0.5, and batch size 256. For ProtBERT-BFD embeddings (5,120-D input), it has four hidden layers [2048 → 512 → 128 → 32], dropout 0.3, and batch size 128. Optimization uses AdamW (lr = 1e-4) with weight decay = 0.1 (ESM-2) or 1e-2 (ProtBERT) and a cosine-annealing scheduler. The loss is BCEWithLogitsLoss, with pos_weight set to negative-to-positive class ratio, label smoothing = 0.1, and gradient clipping (max_norm = 1.0).

A recurrent point of confusion is the phrase “Siamese Contrastive” in the title. The benchmark study explicitly states that, although titled “Siamese Contrastive,” the classification head is trained via supervised BCE loss rather than a pairwise contrastive loss. This is significant because the paper’s theoretical argument is not that contrastive training is responsible for the gains, but that a learned classifier over the fused embedding space is better motivated for binary classification than raw cosine similarity, since cosine scores can become meaningless depending on embedding regularization.

5. Evaluation metrics and quantitative results

The DGEB evaluation in SCOPE focuses on two threshold-independent metrics that are robust to class imbalance: ROC–AUC and Average Precision (AP) (Gupta et al., 10 May 2026). They are defined as

$$\mathrm{ROC\!-\!AUC}=\int_0^1 \mathrm{TPR}(f)\, d\,\mathrm{FPR}(f),$$

where $$\mathrm{TPR}(f)=\mathrm{TP}/(\mathrm{TP}+\mathrm{FN})$$ and $$\mathrm{FPR}(f)=\mathrm{FP}/(\mathrm{FP}+\mathrm{TN})$$ at threshold $f$,

and

$\mathrm{AP}=\sum_n (R_n-R_{n-1})\cdot P_n,$

where $P_n$ and $R_n$ are precision and recall at the $n$-th decision threshold.

ROC–AUC measures the model’s ability to rank true operonic pairs above non-operonic ones across all thresholds. AP focuses on performance in the high-precision regime, reflecting how well top-scoring predictions are enriched for true operonic pairs.

Method	ROC–AUC	AP
Logistic Regression	0.6252	0.4100
XGBoost	0.6160	0.4000
ESM-2 3B + MLP	0.7104	0.5172
ProtBERT-BFD + MLP	0.7064	0.5074
Cosine similarity on ESM-2 embeddings	not reported	0.5247

These results support several benchmark-level conclusions. Embedding-based models outperform physicochemical baselines by ~0.10 in ROC–AUC, confirming that pre-trained protein LLMs capture functional relationships critical for operon pairing. At the same time, the learned Siamese MLP head yields only marginal gains in AP and in fact falls slightly below the unsupervised cosine baseline, suggesting that the geometry of ESM-2’s embedding space already encodes operonic relationships effectively. The paper also reports that the ESM-2 + MLP model’s ROC–AUC of 0.7104 matches or exceeds that of recent state-of-the-art encoders (e.g., ESM-3) on the DGEB leaderboard.

6. Interpretation, applications, and acronym ambiguity

The principal interpretation drawn from DGEB in SCOPE is that protein LLM embeddings are a viable, scalable foundation for genome-wide operon prediction across diverse microbes (Gupta et al., 10 May 2026). The practical implications identified in the paper include automated genome annotation, regulatory network reconstruction, and characterization of organisms lacking experimental operon annotations.

The benchmark also motivates a more specific methodological conclusion. Although a supervised Siamese head is theoretically preferable to raw cosine similarity for binary classification, the observed AP results indicate that the minimal margin between supervised and unsupervised performance calls for future work on embedding regularization or specialized contrastive objectives to refine the functional geometry for fine-grained regulatory tasks. This suggests that the main bottleneck on DGEB may lie less in downstream classifier capacity than in how embedding space geometry is shaped upstream.

A separate source of confusion is nomenclature. In the arXiv literature, DGEB is also used for the Directed-Graph Electronic-design Benchmark, a benchmark for directed graph representation learning in hardware designs (Wang et al., 2024). That benchmark concerns five hardware design datasets and 13 surrogate-model tasks rather than microbial operon prediction. In the genomics context discussed here, however, DGEB refers to the Diverse Genomic Embedding Benchmark for operonic pair classification (Gupta et al., 10 May 2026).