PRING: Graph-Based PPI Prediction Benchmark

• PRING is a graph-based benchmark for PPI prediction that shifts evaluation from isolated binary tasks to holistic network reconstruction.
• It offers a rigorously filtered, multi-species dataset and employs topological and function-oriented metrics like Graph Similarity and Relative Density.
• PRING facilitates comparative analysis across diverse modeling approaches, highlighting limitations in traditional pairwise predictions and guiding future graph-aware improvements.

PRING (PRotein–protein INteraction prediction from a Graph perspective) is a comprehensive benchmark for machine learning-based protein–protein interaction (PPI) prediction, designed to address the limitations of traditional, pairwise-only evaluation frameworks. Unlike conventional benchmarks that assess models on isolated binary classification tasks, PRING emphasizes the reconstruction of biologically meaningful PPI networks, providing both a rigorously filtered, multi-species dataset and a suite of topological and functionally grounded evaluation protocols. By shifting the focus from pairwise accuracy to network-level fidelity, PRING enables systematic comparison and analysis of models’ capabilities to recover the higher-order structural and functional organization of interactomes (Zheng et al., 7 Jul 2025).

1. Motivation and Conceptual Overview

Protein–protein interactions orchestrate essential cellular processes, from signal transduction to metabolic regulation. Despite advances in computational methods, nearly all existing PPI prediction models and benchmarks evaluate candidate protein pairs independently, optimizing for binary accuracy (e.g., AUPR, F1) on held-out pairs. However, biologically relevant discovery tasks demand the ability to reconstruct entire interactomes, identifying not only individual interactions but also the correct organization of protein complexes, hubs, and pathways.

PRING was developed to address this fundamental limitation. Its central objectives are:

1. The curation of a high-quality, multi-species PPI dataset with stringent controls on sequence redundancy and data leakage.
2. The establishment of graph-level and function-oriented metrics and tasks that assess whether models can recover the structure and function of real PPI networks beyond pairwise predictions.

2. Dataset Construction and Quality Controls

The PRING dataset integrates experimentally validated PPIs from UniProt, Reactome, and IntAct, augmented with high-confidence interactions from STRING (combined score > 0.7). Four phylogenetically distant species are covered: Homo sapiens (Human), Arabidopsis thaliana (Arath), Saccharomyces cerevisiae (Yeast), and Escherichia coli (Ecoli). Only SwissProt-reviewed entries with sequence lengths in [50, 1,000] amino acids are retained.

To minimize redundancy and prevent knowledge leakage:

• Protein pairs are filtered via MMSeqs2 clustering to enforce ≤0.40 sequence identity.
• Proteins sharing SwissProt function IDs are eliminated through cross-species functional deduplication.
• Partitioning strictly separates training and test proteins, disallowing sequence overlap within each species.

The resulting gold-standard dataset contains 21,484 unique proteins and 186,818 interactions, distributed as follows:

Species	Proteins	Interactions
Human	10,090	129,861
Arath	5,025	23,584
Ecoli	3,196	17,452
Yeast	3,173	15,921

This protocol supports both intra- and cross-species generalization evaluation while preserving the integrity of biological context (Zheng et al., 7 Jul 2025).

3. Evaluation Paradigms and Metrics

PRING’s evaluation suite is bifurcated into topology-oriented and function-oriented paradigms:

Topology-Oriented Network Reconstruction

A PPI prediction model $f$ receives protein representations $x_i=(s_i, c_i)$ (where $s_i$ is sequence, $c_i$ is optional context), yielding predictions

$I_{i,j}^{\text{pred}} = f(x_i, x_j)\in\{0,1\}\,.$

The reconstructed predicted graph $G^{\text{pred}}$ is compared to the ground-truth $G^{\text{true}}$ via:

• Graph Similarity (GS):

$GS = 1 - \frac{\|\hat{A} - A\|_1}{|E| + |\hat{E}|}$

• Relative Density (RD):

$RD = \frac{|\hat{E}|}{|E|}$

• Degree Distribution (Deg.), Clustering Coefficient (Clus.), and Spectral Distance (Spectral): All computed as MMD-based ratios comparing distributions between predicted and true graphs.

Network reconstruction is tested in:

• Intra-species: Train/test on Human subgraphs (8,072/2,018 split), sampling 500 test subgraphs (sizes 20–200) via BFS, DFS, or random-walk (RW).
• Cross-species: Train on Human, test on sampled subgraphs from Arath, Yeast, and Ecoli (Zheng et al., 7 Jul 2025).

Function-Oriented Tasks

1. Protein Complex Pathway Prediction: For dense Reactome pathways (size 4–20), compare overlap and connectivity:
   • Precision (PP), Recall (PR), and fraction forming a single connected component (PC).
2. GO Enrichment Analysis: Apply Louvain clustering to predicted Human test graphs, then assess:
   • FA: Mean maximal Jaccard overlap with ground-truth clusters.
   • CR: Ratio of within-cluster GO-term Jaccard overlaps versus ground-truth.
3. Essential Protein Justification: Compute Precision@100 and Distribution Overlap (DO) for centrality scores of essential vs. nonessential proteins.

4. Benchmarked Model Categories

PRING implements baseline experiments using four methodological families, enumerating the conceptual developments in the field:

• 1. Sequence-Similarity–Based:
  • SPRINT: Spaced-seed hashing, motif-based, unsupervised.
• 2. Naive Sequence–Based:
  • PIPR: Siamese residual RCNN, combines CNN/BiLSTM and dense layers.
  • D-SCRIPT: Predicts contact maps from structure-aware embeddings.
  • Topsy-Turvy: Integrates sequence encodings and network-level supervision during training, discards graph input at test.
• 3. Protein LLM–Based:
  • PPITrans: ProtT5 embeddings, transformer encoder.
  • TUnA: ESM-2 embeddings, Gaussian process output.
  • PLM-interact: Joint ESM-2 processing of sequence pairs with inter-sequence attention.
• 4. Structure–Based:
  • Struct2Graph: Residue-level graphs, graph-attention network.
  • TAGPPI: Hybrid GNN+CNN with predicted contact maps.
  • RF2-Lite: Streamlined RoseTTAFold2, predicts interchain distances.
  • Chai-1: Multi-modal foundation model (case study only).

All use binary cross-entropy on pairwise edges as the loss:

$$\mathcal{L} = -\frac{1}{M}\sum_{(i,j)}\left[y_{i,j}\log \hat{p}_{i,j} + (1-y_{i,j})\log(1-\hat{p}_{i,j})\right]$$

5. Summary of Experimental Results

Empirical benchmarking reveals:

• Topology Reconstruction (Human Intra-Species):
  • PLM-interact (650M) achieves highest GS (≈0.396 BFS, ≈0.491 RW) and moderate RD (≈1.64 BFS, ≈0.491 RW), indicating superior, yet still imperfect, recovery of network structure.
  • Classification metrics (Pearson $x_i=(s_i, c_i)$0 between AUPR and GS) correlate only partially with graph similarity. High recall models tend to overpredict, resulting in excessive RD and substantial distortions in degree distributions.
• Cross-Species Generalization:
  • All methods experience degradation: GS decreases by 15.2%, 25.3%, 35.2% for Arath, Yeast, and Ecoli, respectively. PLM-based methods retain the highest GS (≈0.24), whereas sequence-based models perform worst (≈0.17).
  • Evolutionary divergence challenges representations that lack conserved signal capture.
• Function-Oriented Tasks:
  • Complex Pathways: PLM-based models dominate (e.g., PPITrans recall≈0.583, precision≈0.863, PC≈0.818; PLM-interact (650M) PC≈0.887). Naive and structure-based models often generate fragmented pathways.
  • GO Enrichment: Even top PLM models achieve only FA≈0.368 and CR≈0.844, indicating less than 40% GO-term overlap within predicted communities.
  • Essential Proteins: PLM-interact (35M) scores P@100≈0.80, DO≈0.423, but the separation of centrality distributions for essential vs. nonessential proteins remains poor.

Model Family	Topology (GS)	Pathway PC	GO FA	Essential P@100
PLM-interact (650M/35M)	0.396/0.491	0.887	0.368	0.80
PPITrans	—	0.818	—	—
Naive Sequence, Structure	≪	≪	≪	≪

Most models, especially naive sequence and structure-based, systematically overestimate network density (RD≫1) and fail to preserve modularity and centrality patterns (Zheng et al., 7 Jul 2025).

6. Implications and Limitations

Observed limitations are twofold:

• Most evaluated approaches generate networks that are too dense, lack biologically meaningful community structure, and scatter essential hubs.
• Binary pairwise objectives lack penalties for global structural deviations and are incapable of modeling context-specific or multi-type interactions.

These deficits pose obstacles to the application of current PPI models for downstream biological discovery tasks such as therapeutic target prioritization, module detection, and pathway analysis.

A plausible implication is that future models incorporating graph-level objectives (e.g., diffusion-based denoising losses, community-aware regularization), richer edge semantics, or multi-modal data may better recapitulate interactome architecture and function.

7. Prospects and Future Work

PRING provides both a rigorous mirror of current limitations and a scaffold for the field’s advancement. Proposed extensions include:

• Expansion to additional taxa (non-model organisms, host–pathogen PPIs).
• Incorporation of multi-type, conditional interactions (activation, inhibition, tissue/state dependence).
• Integration of complementary modalities (gene expression, subcellular localization).
• Development of graph-aware objectives that explicitly optimize for topological and functional network validity.

By framing PPI prediction as a network reconstruction problem and supplying a controlled, open-access resource (https://github.com/SophieSarceau/PRING), PRING redefines benchmark standards and directs model development toward real-world translational impact (Zheng et al., 7 Jul 2025).