- The paper introduces PepBenchmark to standardize peptide ML by integrating 35 curated peptide datasets and a transparent preprocessing pipeline.
- It establishes a unified evaluation protocol, revealing performance variability across different model types and challenges in canonical versus non-canonical peptides.
- The framework paves the way for improved peptide drug discovery and future expansion into structure-based tasks and generative design.
PepBenchmark: A Standardized Benchmark for Peptide Machine Learning
Motivation and Problem Context
Machine learning (ML) methods have become central to peptide drug discovery, with applications ranging from activity prediction (e.g., antimicrobial, anticancer peptides) to de novo peptide design. However, progress is notoriously impeded by the absence of standardized datasets, inconsistent preprocessing, and disparate evaluation protocols. This lack of standardization mirrors the pre-MoleculeNet era in small-molecule ML, resulting in non-comparable, non-reproducible findings and underpowered methodological innovation. PepBenchmark directly addresses this bottleneck by providing an integrated framework for data curation, preprocessing, and evaluation in peptide ML, specifically engineered for reproducibility and broad applicability.
Benchmark Composition and Coverage
PepBenchmark is constructed around three primary pillars:
1. PepBenchData: This module encompasses 35 datasets, including 29 canonical peptide datasets and 6 non-canonical peptide datasets. These span seven key pharmacologically relevant groups: bioactivity, toxicity, hemolysis, membrane permeability, antihypertensive activity, peptide-protein interactions, and physicochemical/structural properties. Each dataset consolidates high-quality curation and harmonizes previously scattered resources across the peptide ML literature. The integration of both canonical and non-canonical peptide sets enables investigation into the distinct challenges associated with non-natural modifications, which are increasingly relevant in peptidomimetic design.
2. PepBenchPipeline: The standardized preprocessing pipeline provides end-to-end reproducibility guarantees, covering essential processes such as sequence deduplication, label harmonization, intelligent dataset splitting (with options for sequence identity or random splits), and feature transformation. The pipeline reduces variability inherent to ad hoc workflows and ensures that quality control is scriptable and transparent, which is particularly crucial given the idiosyncrasies and data sparsity often observed in peptide datasets.
3. PepBenchLeaderboard: The benchmark establishes a unified evaluation protocol for model assessment. It implements standardized metrics for classification and regression (e.g., ROC-AUC, PR-AUC, RMSE), dataset splits, and reporting. The leaderboard features representative baselines from four main modeling paradigms in peptide/sequence ML: Fingerprint-based methods, GNN-based approaches (e.g., GCN, GIN, GAT), PLM-based models (e.g., ProteinBERT, ESM-2), and SMILES-based models (e.g., ChemBERTa). This structure supports direct comparison across a heterogeneous landscape of algorithmic choices and molecular representation frameworks.
Numerical Results and Notable Findings
PepBenchmark's baseline experiments demonstrate substantial performance variability across tasks and model families. Notably, no single model type outperforms others consistently across all tasks. For example, PLM-based models tend to excel in standard activity classification but exhibit limitations on datasets with short, non-canonical, or highly modified sequences. GNN-based approaches display robustness for tasks requiring topology-sensitive property prediction such as membrane permeability. SMILES-based and fingerprint approaches remain competitive on highly structured or physicochemically-oriented tasks, underscoring the need for multi-representational hybridization.
For several classification and regression endpoints, the reported best ROC-AUCs/PR-AUCs for state-of-the-art baselines cluster between 0.8 and 0.95 depending on the dataset, with room for improvement on challenging non-canonical and peptide-protein interaction datasets. Importantly, PepBenchmark's evaluation reveals significant gaps in generalization when models trained on canonical peptides are transferred to non-canonical tasks, pointing to the limitations of existing PLMs and encoders that are typically pretrained on protein-centric data distributions.
Limitations and Open Challenges
A salient limitation is the focus on sequence-level benchmarks; structure-based tasks and datasets are only briefly discussed. This is a major open problem: experimental peptide structural data are extremely sparse due to the underrepresentation of peptides (especially non-canonical ones) in PDB. Only about $2,000$ peptide structures are available—fewer than $100$ for non-natural peptides—rendering structure-function benchmarking currently infeasible. However, the authors plan to address this by leveraging computational chemistry methods (QM/MM, advanced MD) compatible with peptides' modest molecular size, to populate future structural datasets and expand PepBenchmark to generative and structure-based benchmarks.
Implications and Future Directions
PepBenchmark provides the peptide ML community with a critical foundation for method benchmarking, reproducibility, and fair comparison, analogous to what MoleculeNet and TAPE have contributed in other molecular and protein ML domains. This resource is expected to accelerate methodological development in:
- Generalizable peptide/protein representation learning
- Activity and off-target property prediction, including for non-canonical and highly modified peptides
- Multi-modal and transfer learning approaches incorporating structural and physiochemical context
- De novo peptide generation with robust real-world generalization
The inclusion of both canonical and non-canonical benchmarks confronts the field's tendency to overfit to protein-derived sequence distributions and invites innovation in peptide-specialized architectures and training regimes. Future developments will likely integrate structure-based tasks and expand support for generative modeling and activity optimization in silico. Furthermore, the public availability of data and code will ensure that PepBenchmark can grow organically with community input as the field evolves.
Conclusion
PepBenchmark represents a comprehensive standardization effort for peptide machine learning. By unifying datasets, preprocessing, and evaluation, it enables reproducibility, comparability, and meaningful progress in computational peptide science. Its transparent framework and baseline coverage establish a new norm in rigor for the field, while its current limitations point toward exciting future expansions—particularly at the intersection of sequence, structure, and high-throughput peptide design (2604.10531).