SCOPE: Siamese Contrastive Operon Pair Embeddings for Functional Sequence Representation and Classification

Abstract: Identifying operons is a fundamental step in understanding prokaryotic gene regulation, as classifying genes into operons supports the reconstruction of regulatory networks, functional annotation of unannotated genes, and drug candidate development. Experimental approaches such as RT-PCR and RNA-seq provide precise evidence of operon structure, but are laborious and largely limited to well-studied model organisms, making scalable computational methods essential for genome-wide operon identification. Prior computational approaches have employed traditional classifiers such as logistic regression and decision trees, motivating our use of these as physicochemical baselines. The DGEB benchmark evaluates operonic pair classification by embedding each sequence independently with a pre-trained protein LLM and computing pairwise cosine similarity. In contrast, our Siamese MLP learns a classifier over the fused embedding space, which is theoretically better motivated for binary classification, as cosine similarity can yield meaningless scores depending on the regularization of the embedding model. While protein LLM embeddings substantially outperform physicochemical features in ROC-AUC, a learned Siamese MLP head does not significantly improve over unsupervised cosine similarity in Average Precision, suggesting that the geometry of the embedding space already captures the functional relationships needed for this task. Nonetheless, our Siamese MLP achieves a ROC-AUC of 0.71, competitive with state-of-the-art models on the DGEB leaderboard. These findings indicate that protein LLM embeddings are a viable, scalable foundation for operonic pair classification across diverse microbial genomes, with implications for automated genome annotation, regulatory network reconstruction, and characterization of organisms lacking experimental operon annotations.

• The paper introduces a Siamese encoder model that leverages protein language embeddings for operon pair classification, achieving a ROC-AUC improvement over traditional methods.
• The study systematically compares embedding-based models to physicochemical feature models, highlighting the limited incremental gain from supervised fusion.
• The results imply that unsupervised embedding similarity captures most functional signals, offering scalable solutions for automated genome annotation.

SCOPE: Siamese Contrastive Operon Pair Embeddings for Functional Sequence Representation and Classification

Introduction

Operonic organization is a foundational aspect of prokaryotic genomic regulation, with direct implications for functional annotation, regulatory network reconstruction, and downstream biomedical applications. SCOPE addresses the operon pair classification problem, i.e., determining whether consecutive gene pairs belong to the same transcription unit, by leveraging protein LLM embeddings and a discriminative neural architecture. The study systematically compares sequence embedding-based models to traditional machine learning approaches based on physicochemical features, providing a robust empirical assessment under the Diverse Genomic Embedding Benchmark (DGEB) framework.

Methodology

The core of SCOPE is a Siamese encoder architecture with a multilayer perceptron (MLP) classification head for operon pair classification. Pairwise protein sequence inputs are independently embedded using large, pre-trained encoders (ESM-2 3B and ProtBERT-BFD), then fused through concatenation, signed/absolute differences, and element-wise product, before classification.

Figure 1: Architecture of the Siamese encoder model for operon pair classification. Both sequences are processed by a shared-weight encoder, mean-pooled into fixed-size embeddings, fused via concatenation and interaction operations, and classified by an MLP.

The baseline comparison comprises logistic regression and XGBoost models applied to hand-crafted 305-dimensional physicochemical feature vectors. All architectures adopt a Siamese paradigm for fair and consistent input pair fusion. Encoder models (ESM-2 3B: 3B parameters, ProtBERT-BFD: 1024D) are frozen during fine-tuning to isolate the benefits of discriminative fusion. Hyperparameters and training protocols—such as learning rate scheduling, dropout, class balancing, label smoothing, and gradient clipping—are optimized specifically for each architecture to fairly benchmark their capacity under genome-scale class imbalance and varying feature complexity.

Results

Across all metrics, the embedding-based Siamese MLP architectures outperform traditional classifiers using physicochemical descriptors:

• The ESM-2 3B Siamese MLP achieves a ROC-AUC of 0.7104 and Average Precision of 0.5172.
• ProtBERT-BFD Siamese MLP achieves a ROC-AUC of 0.7064 and Average Precision of 0.5074.
• In contrast, logistic regression and XGBoost baselines attain ROC-AUCs of 0.6252 and 0.6160 (Average Precisions: 0.41 and 0.40).

A notable observation is the performance of the DGEB baseline, which utilizes cosine similarity over independent ESM-2 embeddings (Average Precision: 0.5247). The learned Siamese MLP does not decisively surpass this, particularly in Average Precision, despite the additional representational capacity:

Figure 2: ROC-AUC across models on the operon pair classification task. Blue bars represent physicochemical baselines and orange bars represent Siamese MLP models.

These results suggest that most of the discriminatory capacity for this task is derived from the underlying protein LLM, with the fusion and classification head offering only marginal incremental benefit under current methodologies. The 10 percentage point increase in ROC-AUC from baselines to embedding-based models underscores the limitations of physicochemical hand-crafted features in capturing operonic context.

Discussion and Implications

SCOPE demonstrates that Siamese encoder models, particularly those utilizing large transformer protein LLMs, provide a scalable and performant foundation for operon pair functional classification. The findings further clarify that the geometry of the embedding space dominates the achievable predictive performance; the representational power of encoders like ESM-2 3B and ProtBERT-BFD encapsulates much of the functional relationship signal, making additional supervised fusion only incrementally beneficial for this binary classification task.

This has important implications for automated genome annotation pipelines in non-model organisms, where experimental operon characterization is infeasible. Leveraging such embed-and-fuse frameworks can directly enable scalable, annotation-free identification of operonic structures across microbial genomes. From a theoretical standpoint, the results suggest the potential sufficiency of unsupervised embedding similarity for certain molecular function inference tasks, subject to the expressivity and training objectives of the embedding model.

For future AI development in functional genomics, several trajectories are suggested:

• Incorporation of larger, more recently pre-trained encoders may close the remaining gap to state-of-the-art and potentially uncover additional context not presently accessible.
• Revisiting pairwise fusion mechanisms—e.g., attention-based or graph-based interaction modules—may enable richer modeling of inter-sequence dependencies.
• Extending the framework to multi-sequence operon inference, beyond pairwise classification, or exploring contrastive or metric learning objectives.
• Systematic evaluation on even more taxonomically diverse datasets and explicit out-of-distribution generalization studies.

Conclusion

SCOPE rigorously benchmarks Siamese encoder MLP models for operon pair classification and demonstrates their superiority over traditional physicochemical baselines, achieving ROC-AUCs competitive with the best available embeddings-only benchmarks. The evidence suggests that the choice and architecture of the embedding model are the primary determinants of classification performance. Further advancements are likely contingent on improvements in encoder expressivity and innovative pairwise or higher-order fusion designs, with practical implications for scalable functional annotation in prokaryotic genomics.

Reference: "SCOPE: Siamese Contrastive Operon Pair Embeddings for Functional Sequence Representation and Classification" (2605.11022)