Multi-State Protein Benchmark

• Multi-state protein benchmarks are diverse evaluation regimes that assess methods’ ability to design sequences satisfying multiple conformations across different functional states.
• They cover tasks like multi-conformational design, folding-state discrimination, and state-aware structure prediction using tailored metrics and normalized evaluations.
• These benchmarks integrate structural refolding metrics, kinetic analyses, and ensemble comparisons to drive advances in computational protein design and simulation.

Searching arXiv for the benchmark papers and closely related work on multi-state protein evaluation. A multi-state protein benchmark is a benchmark designed to evaluate computational methods on proteins whose behavior cannot be reduced to a single native structure. In current usage, the term covers at least three distinct but related settings: benchmarks for multi-conformational design, where one sequence must satisfy two or more prescribed backbones; benchmarks for folding-state discrimination, where proteins are classified as two-state or multi-state folders from their native structures; and state-aware structure or dynamics benchmarks, which assess apo/holo transitions, disorder-to-order transitions, multimerization-dependent structure, or metastable conformational landscapes (Abrudan et al., 29 Jul 2025, Menichetti et al., 2015, Zeng et al., 18 Jun 2025, Ye et al., 2024, Qiao et al., 2022, Aghili et al., 20 Oct 2025). The literature therefore uses “multi-state” both for discrete experimentally resolved conformations and for kinetic or ensemble-level state structure.

1. Scope and meanings of “multi-state” in protein benchmarking

The most explicit use of the term appears in inverse folding and protein design. "Multi-state Protein Design with DynamicMPNN" introduces the first explicitly multi-state inverse-folding benchmark for proteins, with the goal of testing whether sequence-design methods can generate amino-acid strings that refold into two or more target conformations rather than optimizing each conformation separately and aggregating predictions post hoc (Abrudan et al., 29 Jul 2025). In this setting, a benchmark entry consists of multiple backbone graphs for the same protein, and success is assessed by state-specific refolding.

A second usage concerns folding kinetics. "Network measures for protein folding state discrimination" defines a binary classification task in which proteins are labeled as TS or MS, with TS denoting a two-state “all-or-none” mechanism and MS denoting a multi-state mechanism involving one or more intermediates (Menichetti et al., 2015). Here, “multi-state” refers not to multiple experimentally resolved conformers used as targets, but to experimentally annotated kinetic behavior.

A third usage is broader and task-rich. DisProtBench evaluates statefulness arising from intrinsically disordered regions, multimeric complexes, and GPCR ligand pairs, but it explicitly does not attempt to sample alternative conformations of the same sequence beyond these functional states (Zeng et al., 18 Jun 2025). ProteinBench distinguishes “Multiple-State Prediction,” which aims to sample two or more experimentally observed conformations, from “Conformational Distribution Prediction,” which compares a generated ensemble with an MD-derived reference distribution (Ye et al., 2024). NeuralPLexer frames the problem as ligand-conditioned prediction of apo and holo ensembles (Qiao et al., 2022), whereas the weighted-ensemble MD benchmark defines states through metastable basins in TICA space and PCCA+ macrostates (Aghili et al., 20 Oct 2025).

Benchmark line	State definition	Core task
DynamicMPNN	Two or more prescribed conformations	Multi-state inverse folding/design
TS/MS folding benchmark	Two-state vs multi-state folding kinetics	Binary classification
DisProtBench	Ordered/disordered, monomer/multimer, apo/holo	State-aware structure prediction
ProteinBench	Multiple-state prediction and distribution prediction	General foundation-model evaluation
NeuralPLexer benchmarks	Ligand-free vs ligand-bound ensembles	State-conditioned structure generation
WE MD benchmark	Metastable basins in TICA/PCCA+ space	Conformational landscape sampling

This taxonomy suggests that a “multi-state protein benchmark” is not a single standardized object but a family of evaluation regimes that differ in what constitutes a state, what is predicted, and how success is measured.

2. Multi-conformational design benchmark in DynamicMPNN

The DynamicMPNN benchmark is motivated by proteins that interconvert among discrete, functionally relevant structures, including metamorphic folds, hinge-mediated domain movements, open/closed transporters, and apo/holo switches. The paper argues that standard benchmarks based on sequence recovery or single-structure refolding do not capture the joint constraints required to stabilize multiple folds, and that prior multi-state design strategies based on post-hoc aggregation of single-state predictions have yielded extremely low experimental hit rates, including approximately $0.002\%$ success in designing 2-state hinge proteins in Praetorius et al. 2023 (Abrudan et al., 29 Jul 2025).

Its dataset construction begins from CoDNaS (v2025) and the PDB. Experimentally observed chains are grouped into clusters at at least $95\%$ local sequence identity, while ensuring the same UniProt ID across cluster members. For each cluster, the two chains with maximal $C_\alpha$-RMSD are selected, producing 46,033 high-diversity conformer pairs that cover $75\%$ of CATH superfamilies, compared with approximately $21\%$ covered by NMR alone. To form benchmark splits emphasizing large-amplitude rearrangements, the authors compile a curated challenging set from prior studies: 92 metamorphs from Porter and Looger 2018, 91 apo/holo transitions from Saldaño et al. 2022, the OC23/OC85 open/closed datasets, and 20 transporters from Kalakoti and Wallner 2025. Pairs are ranked by inter-state RMSD, with the top 94 assigned to test and the next 100 to validation; training pairs with TM-score greater than 0.4 to any test or validation structure are removed. The final split contains 44,243 training, 100 validation, and 94 test pairs (Abrudan et al., 29 Jul 2025).

The formalization is explicitly joint. Where a single-state inverse-folding model learns $p(Y \mid X)$, DynamicMPNN models

$$p(Y \mid X^1,\ldots,X^m)=\prod_{i=1}^{n} p(y_i \mid y_{1:i-1}; X^1,\ldots,X^m).$$

Each $X^k \in \mathbb{R}^{n \times 3 \times 3}$ encodes the 3D frames of the $k^{\text{th}}$ conformation, and $Y=(y_1,\ldots,y_n)$ is the amino-acid sequence. The model encodes each backbone separately with an SE(3)-equivariant GNN using Geometric Vector Perceptron layers, aligns and pools embeddings with a DeepSet reduction that is order-invariant across conformers, and then decodes the sequence autoregressively. Training uses cross-entropy on the ground-truth sequence across all conformations simultaneously (Abrudan et al., 29 Jul 2025).

In this benchmark, the unit of evaluation is therefore not a single structure but a coupled state set. That design choice is central: the benchmark tests whether one sequence can satisfy multiple geometric constraints in a single forward pass rather than through inference-time averaging of independently trained single-state models.

3. Evaluation metrics and reported performance

DynamicMPNN evaluates multi-state design through “refoldability” rather than sequence recovery. For each designed sequence $95\%$0 and target conformation $95\%$1, AlphaFold2 is run under an AlphaFold Initial Guess (AFIG) protocol, with backbone frames initialized directly to the coordinates of $95\%$2. The primary metric is

$95\%$3

Two normalizations are then applied. Structure normalization divides AFIG-RMSD by the maximum inter-state RMSD of the target:

$95\%$4

Decoy normalization compares the AFIG-RMSD against structurally unrelated decoys $95\%$5 with TM-score $95\%$6:

$95\%$7

Per-residue confidence pLDDT from AFIG predictions serves as a secondary metric; high pLDDT combined with low RMSD indicates that the sequence specifically refolds into the target state. Aggregation is reported under three strategies: “Best Single,” “Best Paired,” and “All Avg.” In “Best Single,” the minimum RMSD or maximum pLDDT across the two states is taken for each sequence, then the best sequence among 16 samples is chosen. In “Best Paired,” the two AFIG-RMSDs are averaged for each sequence and the sequence with the lowest mean is selected among 16 samples. “All Avg.” averages over all sequences and states (Abrudan et al., 29 Jul 2025).

On the 94-pair challenging test set, DynamicMPNN improves over ProteinMPNN-MSD most clearly under “Best Paired,” which the paper identifies as especially relevant for wet-lab candidate selection. The reported values are summarized below.

Metric, “Best Paired”	DynamicMPNN	ProteinMPNN-MSD
Raw AFIG-RMSD	13.43 Å	14.76 Å
Structure-normalized RMSD	0.58	0.65
Decoy-normalized RMSD	12.76	13.99
Mean pLDDT	59.31	57.74

The paper reports that the reduction in raw AFIG-RMSD corresponds to approximately $95\%$8 with $95\%$9 under a Wilcoxon test, and that the structure-normalized RMSD improvement is approximately $C_\alpha$0 (Abrudan et al., 29 Jul 2025). DynamicMPNN also outperforms on “Best Single,” despite ProteinMPNN being trained for one-state design. Over all samples, both learned models trail native sequences slightly, which the paper notes is also observed in single-state studies and is likely related to evolutionary optimization for functions beyond pure foldability. A case study on the Switch Arc metamorphic protein, PDB $C_\alpha$1, shows that DynamicMPNN’s top design recapitulates the native $C_\alpha$2-sheet core in both states, whereas ProteinMPNN’s does not (Abrudan et al., 29 Jul 2025).

4. Folding-state discrimination as a benchmark of multi-state kinetics

A distinct benchmark tradition defines “multi-state” at the level of folding mechanism rather than conformational target sets. The TS/MS benchmark of "Network measures for protein folding state discrimination" consists of 63 proteins with literature-curated kinetic assignments: 38 TS and 25 MS, corresponding to class balance of $C_\alpha$3 TS and $C_\alpha$4 MS. Inclusion requires known TS vs. MS behavior, resolved structures in the PDB, and chain lengths of approximately 40–200 residues. Homologous proteins are grouped during cross-validation to reduce over-optimistic performance (Menichetti et al., 2015).

The benchmark represents each protein by a Protein Contact Network $C_\alpha$5, where residues are nodes and contacts are defined by a $C_\alpha$6 distance threshold of 8 Å:

$C_\alpha$7

Three physically interpretable observables are introduced. The network-entropy ratio $C_\alpha$8 compares the entropy of a weighted contact ensemble with only node strengths constrained against one with both strengths and degrees constrained; larger $C_\alpha$9 is interpreted as more configurational microstates and correlates with MS behavior. The Laplacian observables are the three largest eigenvalues of a filtered long-range contact graph, normalized by chain length $75\%$0: $75\%$1, $75\%$2, and $75\%$3. The inter-residue link density

$75\%$4

measures the fraction of sequence-separation diagonals with no contacts; larger $75\%$5 correlates with more localized interactions and with MS folding (Menichetti et al., 2015).

Methodologically, the benchmark uses quadratic Fisher’s Discriminant Analysis, 10-fold cross-validation with 10,000 random resamplings, and homolog grouping within folds. Features are tested singly and in all 2-, 3-, and 4-variable combinations, with evaluation by overall accuracy, per-class accuracy, and Matthews Correlation Coefficient. The best two-feature pair, $75\%$6, achieves $75\%$7 accuracy, $75\%$8 MS sensitivity, $75\%$9 TS specificity, and $21\%$0. Without cross-validation, accuracy is $21\%$1. The best single feature is $21\%$2, with $21\%$3 accuracy and $21\%$4. Baselines such as contact order and long-range contact order perform substantially worse (Menichetti et al., 2015).

This benchmark is therefore “multi-state” in a kinetic and classification sense. It does not ask whether a model can generate or recover multiple structures, but whether native-state structural observables suffice to predict the presence of folding intermediates.

5. State-aware benchmarks for structure prediction and ligand-conditioned modeling

Several recent benchmarks generalize the concept of state beyond discrete conformer pairs. DisProtBench organizes evaluation along three axes: intrinsically disordered regions, multimeric complexes, and GPCR ligand pairs. Its datasets comprise approximately 180 proteins with 450 annotated IDRs and approximately 6,000 disordered residues, approximately 220 binary complexes with around 8,800 interfacial residue pairs, and 75 Class A GPCR apo/bound receptor pairs. It defines nine tasks, including disorder classification, disorder regression, PPI interface prediction, docking ranking, GPCR state classification, activation-state geometry, and disorder-to-order transition context. The paper explicitly states that tasks 7–9 directly probe “multi-state” behavior, but also emphasizes that the benchmark does not provide true alternative conformational ensembles for the same sequence (Zeng et al., 18 Jun 2025).

Its evaluation pipeline integrates global structural metrics, local confidence calibration, classification metrics, regression metrics, and interface-specific scores. The authors report that standard global metrics correlate weakly with interface accuracy in PPI tasks, with $21\%$5, and fail entirely to distinguish bound versus unbound GPCR states, with $21\%$6. On GPCR ligand pairs, most models are near-random at activation-state classification, around $21\%$7, except AlphaFold2-Multimer at approximately $21\%$8. No model recovers induced folding of long IDRs at interfaces, with precision@10 for $21\%$9 (Zeng et al., 18 Jun 2025).

ProteinBench embeds multi-state evaluation within a broader foundation-model taxonomy. It defines “Multiple-State Prediction” as sampling two or more experimentally observed conformations and “Conformational Distribution Prediction” as generating an ensemble whose statistics match a reference MD ensemble. The benchmark uses Apo-Holo with 91 proteins, BPTI with five conformational clusters from 13 $p(Y \mid X)$0s MD, and ATLAS with 82 proteins for distribution prediction. It evaluates multi-state outputs with ensemble metrics such as

$p(Y \mid X)$1

and

$p(Y \mid X)$2

along with pairwise diversity, clash and bond-break rates, flexibility correlation, $p(Y \mid X)$3 distances, Jaccard similarity, and mutual information on contact and exposure observables. The reported results show that folding-based methods obtain the highest ensemble TM on Apo-Holo, but specialized multi-state strategies do not outperform the trivial baseline of always returning the apo structure; on BPTI, the best $p(Y \mid X)$4 is approximately 1.37 Å for ConfDiff-ESM with classifier-free guidance; and for ATLAS, MD-trained generative models reach Pearson $p(Y \mid X)$5 up to 0.67 in flexibility correlation (Ye et al., 2024).

NeuralPLexer provides a more explicitly state-conditioned blueprint. It models an apo ensemble when only protein sequence is supplied and a holo ensemble when both sequence and ligand molecular graph are provided. Its benchmark-relevant datasets include PocketMiner apo/holo pairs with 33 holo and 29 apo structures, a recent ligand-bound set with 118 systems selected so that another PDB entry of the same sequence has backbone RMSD $p(Y \mid X)$6 Å, and a PDBBind2020 blind docking split. In addition to TM-score and $p(Y \mid X)$7 RMSD, the paper introduces a state-change-sensitive Weighted Q-factor and reports that, on PocketMiner holo prediction, NeuralPLexer reaches mean TM-score 0.934 versus 0.929 for AlphaFold2 and Weighted Q-factor 0.608 versus 0.538. In binding-site inpainting, success defined by $p(Y \mid X)$8, ligand RMSD $p(Y \mid X)$9 Å, and clash rate $$p(Y \mid X^1,\ldots,X^m)=\prod_{i=1}^{n} p(y_i \mid y_{1:i-1}; X^1,\ldots,X^m).$$0 is $$p(Y \mid X^1,\ldots,X^m)=\prod_{i=1}^{n} p(y_i \mid y_{1:i-1}; X^1,\ldots,X^m).$$1 versus $$p(Y \mid X^1,\ldots,X^m)=\prod_{i=1}^{n} p(y_i \mid y_{1:i-1}; X^1,\ldots,X^m).$$2 for AlphaFold2 template alignment (Qiao et al., 2022).

Taken together, these benchmarks broaden the state concept from “two known backbones for one sequence” to a wider set of biologically conditioned alternatives: ordered versus disordered, monomer versus complex, ligand-free versus ligand-bound, and experimentally resolved versus ensemble-level conformational variability.

6. Multi-state benchmarks for conformational landscape sampling

A further line of work defines multi-state behavior in terms of metastable basin discovery and equilibrium population recovery. "A Standardized Benchmark for Machine-Learned Molecular Dynamics using Weighted Ensemble Sampling" introduces a framework for benchmarking MD engines on nine proteins ranging from 10 to 224 residues, including Chignolin, Trp-cage, BBA, a3D, Protein B, Protein G, $$p(Y \mid X^1,\ldots,X^m)=\prod_{i=1}^{n} p(y_i \mid y_{1:i-1}; X^1,\ldots,X^m).$$3-repressor, Homeodomain, and the WW domain. Each protein exhibits at least two distinct metastable basins, and some, including BBA, Protein G, and $$p(Y \mid X^1,\ldots,X^m)=\prod_{i=1}^{n} p(y_i \mid y_{1:i-1}; X^1,\ldots,X^m).$$4-repressor, display high free-energy barriers between topologically distinct folds (Aghili et al., 20 Oct 2025).

States are represented in two ways. Continuous state space is described by the first four TICA components, and discrete states are given by PCCA+ spectral clustering of 100 $$p(Y \mid X^1,\ldots,X^m)=\prod_{i=1}^{n} p(y_i \mid y_{1:i-1}; X^1,\ldots,X^m).$$5-means microstates in the first ten TICA dimensions, yielding five macrostates per protein. WE sampling is implemented with WESTPA 2.0 using a Minimal Adaptive Binning grid of $$p(Y \mid X^1,\ldots,X^m)=\prod_{i=1}^{n} p(y_i \mid y_{1:i-1}; X^1,\ldots,X^m).$$6 bins in $$p(Y \mid X^1,\ldots,X^m)=\prod_{i=1}^{n} p(y_i \mid y_{1:i-1}; X^1,\ldots,X^m).$$7, targeting three walkers per bin, with 1,000 MD steps per iteration and data saved every 100 steps. Metrics focus on state coverage, equilibrium population matching, basin identification, transition connectivity via MSMs, and pathway diversity. The framework reports weighted KDEs, $$p(Y \mid X^1,\ldots,X^m)=\prod_{i=1}^{n} p(y_i \mid y_{1:i-1}; X^1,\ldots,X^m).$$8, Wasserstein-1 distance, contact-map differences, and stationary distributions, with key formulas including

$$p(Y \mid X^1,\ldots,X^m)=\prod_{i=1}^{n} p(y_i \mid y_{1:i-1}; X^1,\ldots,X^m).$$9

$X^k \in \mathbb{R}^{n \times 3 \times 3}$0

and

$X^k \in \mathbb{R}^{n \times 3 \times 3}$1

The benchmark reports that implicit-solvent WE reaches $X^k \in \mathbb{R}^{n \times 3 \times 3}$2 coverage of ground-truth TICA space in 9–83 days of single-node-equivalent time, while fully trained CGSchNet attains comparable coverage with a 10–25$X^k \in \mathbb{R}^{n \times 3 \times 3}$3 speedup. Under-trained CGSchNet collapses and samples only a narrow region of TICA space. Larger systems with rugged landscapes, including a3D, $X^k \in \mathbb{R}^{n \times 3 \times 3}$4-repressor, and Protein G, plateau at $X^k \in \mathbb{R}^{n \times 3 \times 3}$5 coverage even with WE (Aghili et al., 20 Oct 2025).

This benchmark does not use experimentally named states such as “open” and “closed.” Instead, it operationalizes multi-state behavior through metastable basin structure in latent slow-coordinate space. A plausible implication is that it complements structure-pair benchmarks by testing whether a simulator can recover the topology of an entire conformational landscape rather than only a small number of discrete targets.

7. Recurring design principles, limitations, and unresolved issues

Across the literature, several benchmark design principles recur. DynamicMPNN emphasizes real multi-conformational data, evaluation by refolding into each state, and normalization by task difficulty through structure-normalized and decoy-normalized RMSD (Abrudan et al., 29 Jul 2025). DisProtBench emphasizes realism, function-aware metrics, context-specific stratification, a unified extensible pipeline, interpretability through confidence calibration and residue-level error maps, and open release of splits, predictions, and scripts (Zeng et al., 18 Jun 2025). PFMBench, while not itself state-aware, contributes broadly transferable principles: strict homology control at $X^k \in \mathbb{R}^{n \times 3 \times 3}$6 identity, standardized metrics and protocols, hierarchical reduction of correlated tasks, and a streamlined evaluation protocol that reduces compute while retaining representativeness (Gao et al., 1 Jun 2025). ProteinBench similarly treats benchmark construction as modular and explicitly identifies multi-state prediction and distribution prediction as extensible tasks within a living benchmark (Ye et al., 2024).

The main limitations are also consistent. DynamicMPNN’s benchmark is computational, depends on AlphaFold2, is biased toward two-state tasks, and may still permit subtle homologous leakage because clustering is at $X^k \in \mathbb{R}^{n \times 3 \times 3}$7 identity; the paper also notes AlphaFold2’s bias toward a dominant state and the exclusion of continuous landscapes such as intrinsically disordered regions (Abrudan et al., 29 Jul 2025). The TS/MS folding benchmark is limited by small sample size, moderate class imbalance, reliance on resolved structures, possible ambiguity in MS annotations, and the absence of strict sequence-identity thresholding beyond homolog grouping (Menichetti et al., 2015). DisProtBench explicitly lacks sampled conformational ensembles for IDPs, broader allosteric and post-translational states, and NMR-derived flexible ensembles (Zeng et al., 18 Jun 2025). ProteinBench acknowledges that its multi-state coverage is still limited to apo-holo pairs and BPTI and does not yet incorporate user-objective tasks such as allosteric design or heterogeneity-driven docking (Ye et al., 2024). The WE MD benchmark lacks explicit reporting of kinetic rates such as MFPTs, experimentally validated state identities, and standardized train/validation/test splits for new engines beyond the provided CGSchNet comparisons (Aghili et al., 20 Oct 2025).

These limitations clarify an unresolved issue in the field: “multi-state” is not yet standardized. Inverse-folding benchmarks privilege compatibility with several prescribed structures, folding-state benchmarks privilege kinetic mechanism labels, structure-prediction benchmarks privilege state-conditioned geometric recovery, and simulation benchmarks privilege basin discovery and equilibrium sampling. This suggests that a mature multi-state protein benchmark would need to integrate several of these components simultaneously: explicit state labels, homology-controlled splits, refolding or generation metrics, ensemble-level comparisons, and, where possible, experimentally grounded validation of state populations and transitions.