PepBenchmark: Peptide ML Benchmark for Drug Discovery

• PepBenchmark is a standardized benchmark for peptide drug discovery ML, unifying datasets, preprocessing, and evaluation protocols across canonical and non-canonical peptides.
• It features three integrated components: PepBenchData with 35 curated datasets, PepBenchPipeline for standardized data processing, and PepBenchLeaderboard with unified evaluation across model families.
• Empirical findings indicate that while PLM-based models excel in canonical bioactivity tasks, structure-aware models like GNNs and SMILES-based approaches perform better on non-canonical peptide challenges.

PepBenchmark is a standardized benchmark for peptide machine learning in peptide drug discovery, introduced as "PepBenchmark: A Standardized Benchmark for Peptide Machine Learning" and organized around three components: PepBenchData, PepBenchPipeline, and PepBenchLeaderboard (Zhang et al., 12 Apr 2026). It was designed to unify datasets, preprocessing, and evaluation protocols across canonical and non-canonical peptides, thereby making method comparison more reproducible and scientifically meaningful in a domain otherwise characterized by fragmented datasets, ad hoc preprocessing, inconsistent evaluation, and heterogeneous molecular representations.

1. Terminological scope and disambiguation

The name PepBenchmark has multiple uses in nearby literature, and the distinction is consequential. In the strict sense, PepBenchmark is the peptide drug-discovery benchmark introduced in 2026 and centered on peptide ML tasks spanning bioactivity, toxicity, permeability, immunogenicity, binding, and non-canonical chemistry-aware prediction (Zhang et al., 12 Apr 2026).

At the same time, the term also appears as a search variant or informal label in tandem mass spectrometry. The paper "PepSpecBench: A Unified Evaluation Benchmark for Peptide Tandem Mass Spectrometry Prediction" explicitly states that if one encountered the name “PepBenchmark,” the paper does not use that term; rather, PepSpecBench is the unified peptide MS/MS benchmark one is likely looking for under that name. A separate work, "Pep2Prob Benchmark: Predicting Fragment Ion Probability for MS$^2$-based Proteomics," states that PepBenchmark refers to the Pep2Prob Benchmark, a resource for peptide-specific fragment ion probability prediction. By contrast, "PEFT-Bench: A Parameter-Efficient Fine-Tuning Methods Benchmark" explicitly notes that PepBenchmark is a misnomer or search variant in that context and is not a synonym used by the authors (Yang et al., 3 May 2026, Xu et al., 12 Aug 2025, Belanec et al., 26 Nov 2025).

This ambiguity reflects a broader partitioning of peptide-related benchmarking into at least two technically distinct areas: peptide drug-discovery ML and peptide mass-spectrometry modeling. PepBenchmark, in the title-bearing sense, belongs to the former.

2. Benchmark architecture and motivation

PepBenchmark addresses a specific methodological gap: progress in peptide ML has been hindered by fragmented datasets, ad hoc preprocessing, inconsistent evaluation, and difficulty comparing methods across canonical and non-canonical peptides. Its stated contribution is a first standardized, end-to-end benchmark for peptide ML, with three integrated layers (Zhang et al., 12 Apr 2026).

PepBenchData is a curated collection of 35 datasets—29 canonical-peptide datasets and 6 non-canonical-peptide datasets—organized into 7 task families. This coverage is intended to span “key aspects of peptide drug development” and to align sequence-centric and chemistry-centric peptide representations within a single resource.

PepBenchPipeline provides a unified preprocessing framework for cleaning, construction, splitting, and feature transformation. The pipeline is explicitly designed to mitigate leakage, harmonize heterogeneous assay formats, and standardize the treatment of canonical versus non-canonical peptides.

PepBenchLeaderboard supplies a unified evaluation protocol with strong baselines across four methodological families: Fingerprint-based, GNN-based, PLM-based, and SMILES-based models. The benchmark therefore functions not only as a dataset repository but also as an executable comparison framework.

Relative to prior resources, the benchmark is positioned against MoleculeNet and the Therapeutics Data Commons. The distinction drawn is that MoleculeNet focuses on small molecules, whereas TDC spans broad therapeutic tasks but lacks peptide-specialized coverage, unified handling of canonical and non-canonical peptides, and peptide-specific evaluation protocols. PepBenchmark’s design therefore targets peptide-specific assay heterogeneity, representation mismatch, and split strategies tailored to peptide generalization.

3. Data scope, peptide representations, and task families

PepBenchmark distinguishes between canonical peptides and non-canonical peptides at the representation level. Canonical peptides are defined as sequences built from the 20 standard amino acids without non-natural monomers or covalent backbones beyond standard peptide bonds, and they are typically represented as FASTA or sequence strings. Non-canonical peptides include D-residues, cyclizations, N-methylations, $\beta/\gamma$ amino acids, and other modifications requiring chemistry-aware formats such as HELM, BILN, SMILES, and atomistic graphs (Zhang et al., 12 Apr 2026).

The benchmark’s 35 datasets are grouped into 7 task families:

Task family	Representative datasets	Typical endpoint format
Antimicrobial, antiviral, antiparasitic, anticancer bioactivity	APD3, dbAMP, DRAMP, CAMP, CancerPPD, AntiCP 2.0, ACP-DL, ACPred-FL, ParaPep, PredAPP	Binary classification; sometimes multi-label
Toxicity	Hemolytik / Hemolytik2, HLPpred-Fuse, ATSE	Classification; regression for HC50
Permeability and transport	CPPsite / CPPsite 2.0, CPPpred, BrainPeps, BBPpred, CycPeptMPDB	Classification; regression/classification
Immunogenicity and epitopes	TANTIGEN / TANTIGEN 2.0	Classification; interaction/regression proxies
Enzymatic inhibition and functional peptides	iDPPIV-SCM, AHTPDB, mAHTPred, BioDADPep, NeuroPep 2.0, QuorumPeps	Classification, regression, or multi-label
Peptide–protein interaction and binding	PepNN datasets	Binary interaction or site-level labeling
Non-canonical peptide tasks	CycPeptMPDB, membrane diffusion datasets with CLMs, additional curated non-canonical sets	Chemistry-centric regression/classification

The benchmark emphasizes peptide-length regimes typical of therapeutics, often <50 aa, with CPPs and AMPs frequently in the 5–50 aa range and dipeptides appearing in DPP-IV tasks. The data summary further notes that short peptides are underrepresented in common protein corpora, citing the observation that only ~2.8% of UniRef sequences are <50 residues, which helps explain why peptide-specialized benchmarks and models can diverge from general protein-learning practice.

Several forms of heterogeneity are explicitly identified. Class imbalance is common in bioactivity datasets such as ACP and AMP prediction, motivating the use of AUPRC and MCC. Measurement heterogeneity appears in toxicity and permeability regression, where assay units must be normalized, and classification tasks require harmonization of label definitions across sources. The benchmark therefore treats data curation as part of the modeling problem rather than as a purely preparatory step.

4. PepBenchPipeline: preprocessing, splitting, and leakage control

PepBenchPipeline standardizes preprocessing across heterogeneous peptide ML tasks. For canonical sequences, it enforces uppercase one-letter codes for the 20 canonical residues, removes invalid or ambiguous tokens, harmonizes X/U/ambiguous residues via filtering or mapping rules, and deduplicates exact sequences and near-duplicates post-normalization. For non-canonical peptides, it uses HELM, BILN, or SMILES to encode D-residues, N-methylation, cyclizations, PTMs, and related chemistry-aware structures; the benchmark notes that pyPept can generate atomistic graphs for modified peptides (Zhang et al., 12 Apr 2026).

Chemical standardization is also included. The pipeline standardizes peptide forms to neutral or dominant charge at physiological pH where the assay requires it, removes salts and counter-ions in SMILES, and documents consistent handling of termini such as amidation and acetylation. Missing values are either dropped or imputed according to task policy, while multi-assay conflicts are resolved through explicit rules such as median aggregation or best-quality assay selection.

Split construction is benchmarked rather than assumed. For classification tasks, random/stratified splits with class-ratio preservation are supported; 80/10/10 is described as typical for many datasets, and k-fold CV is available on smaller sets. To reduce homologous leakage, PepBenchmark uses sequence-identity splits, clustering sequences by identity via MMseqs2 or similar and assigning train/validation/test at the cluster level. For non-canonical peptides, scaffold-based or macrocycle-aware splits are used to prevent chemotype leakage.

Feature transformation is similarly standardized. The pipeline supports RDKit Morgan/circular fingerprints, physicochemical descriptors such as charge, hydrophobicity, HBD/HBA, and TPSA, PLM embeddings pooled by mean, max, or CLS, and GNN features over atomistic or residue-level graphs. For SMILES-based models, canonical SMILES tokenization is standardized and can be character-level or BPE-based. The benchmark therefore couples leakage mitigation with representation harmonization, which is central for comparing sequence models against chemistry-aware models on partially overlapping task families.

5. PepBenchLeaderboard: model families, metrics, and evaluation protocol

PepBenchLeaderboard evaluates four major methodological families under a unified protocol. The Fingerprint-based family includes Random Forest, XGBoost, Support Vector Machine, and Multi-Layer Perceptron models over Morgan fingerprints and RDKit descriptors. The GNN-based family includes graph convolutional and graph attention architectures such as GCN, GAT, and GIN, operating either on residue graphs for canonical peptides or atomistic graphs for non-canonical peptides. The PLM-based family uses pretrained protein or peptide LLMs such as ProtBERT/ProtTrans, ESM-2, and peptide-focused PLMs including PepBERT and Pepdora. The SMILES-based family uses tokenized chemical strings with Transformer or GRU/LSTM architectures, including ChemBERTa-like approaches where appropriate (Zhang et al., 12 Apr 2026).

For GNNs, message passing is formalized as

$$h_i^{(\ell+1)} = \phi^{(\ell)}\!\left(h_i^{(\ell)}, \square_{j \in \mathcal{N}(i)} \psi^{(\ell)}(h_i^{(\ell)}, h_j^{(\ell)}, e_{ij})\right),$$

with global mean, sum, or attention pooling at readout. Classification tasks use cross-entropy loss; regression tasks use MSE or Huber loss.

Evaluation is standardized across repeated runs or CV folds with fixed seeds and early stopping on validation metrics. The classification metrics include Accuracy, Precision, Recall, F1, AUROC, AUPRC, and Matthews Correlation Coefficient. The benchmark gives

$$\mathrm{MCC}=\frac{TP\cdot TN-FP\cdot FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}.$$

Regression metrics include RMSE, MAE, and $R^2$, while Spearman’s $\rho$ is used for ranking if needed. Calibration can also be reported through Expected Calibration Error.

A central design choice is that hyperparameter search spaces are kept modest to make large-scale comparison tractable. This makes the leaderboard a reproducibility instrument as much as a performance ranking mechanism: it is intended to test whether differences persist under controlled preprocessing, split construction, and metric computation rather than under bespoke task-specific tuning.

6. Reported findings, relation to adjacent peptide benchmarks, and limitations

PepBenchmark reports several high-level empirical trends. PLM-based approaches achieve strong performance on canonical bioactivity classification tasks such as AMP, ACP, and toxicity prediction. Structure-aware models, especially GNN and SMILES-based models, are advantageous on non-canonical cyclic peptide permeability and diffusion tasks, where explicit atomistic information is critical. Across model families, identity-based and scaffold-based splits reduce performance relative to random splits, underscoring that realistic peptide generalization is materially harder than interpolation over close homologs or related chemotypes. At the same time, fingerprint baselines remain competitive on small, imbalanced datasets, often yielding robust MCC and AUPRC with modest computational cost (Zhang et al., 12 Apr 2026).

These findings place PepBenchmark in a different problem class from adjacent peptide benchmarks in mass spectrometry. PepSpecBench standardizes peptide MS/MS intensity prediction, using a backbone-disjoint split strategy, a canonical 234-dimensional b/y-ion space, cross-species evaluation, and perturbation probes for NCE and charge awareness. Pep2Prob instead formalizes peptide-specific fragment ion probability prediction, with 608,780 unique precursors, 183,263,674 HCD MS$^2$ spectra, graph-based leakage-resistant splits, and metrics defined on masked fragment-probability vectors. PepBenchmark differs from both by targeting peptide drug-discovery ML across canonical and non-canonical therapeutic tasks rather than tandem mass-spectrometry prediction (Yang et al., 3 May 2026, Xu et al., 12 Aug 2025).

The benchmark also states clear limitations. Structural data are scarce, with ~2,000 peptide entries in PDB and <100 non-natural entries, which limits structure-based tasks. Planned extensions include a peptide structural simulation pipeline based on QM/MM, enhanced MD, and physics-informed simulations, broader non-canonical coverage, additional ADMET endpoints such as solubility, stability, and immunogenicity with standardized assay normalization, curation of negative sets and gold-standard negatives, and expansion of peptide–protein interaction datasets toward affinity and kinetics. The repository provides data loaders, preprocessing scripts, training and evaluation scripts, and documentation, making PepBenchmark not only a benchmark specification but also a reproducible infrastructure layer for peptide ML research (Zhang et al., 12 Apr 2026).