ProtDBench: Protein Binder Benchmark

• ProtDBench is a benchmark that standardizes protein binder design evaluation by defining unified tasks, fixed pipelines, and throughput-aware metrics.
• It assesses binders at both sequence and backbone levels, incorporating cluster-level diversity to ensure reproducible and fair method comparisons.
• The benchmark shows that verifier choice and filtering rules significantly impact success metrics, highlighting the trade-off between quality, speed, and structural diversity.

Searching arXiv for the ProtDBench paper and closely related benchmark context. ProtDBench is a standardized, throughput-aware benchmark for protein binder design that was introduced to make evaluation of de novo binder generators fair, reproducible, and comparable across methods. It targets a long-standing problem in the field: reported in silico results are difficult to interpret across studies because prior evaluations used different targets, hotspot definitions, verifier models, score thresholds, filtering rules, and computational budgets. ProtDBench addresses this by defining unified benchmark tasks, a fixed evaluation pipeline, explicit verifier and filter configurations, throughput-aware metrics under a fixed 24-hour budget, and cluster-level success criteria that account for structural diversity (Liu et al., 5 May 2026).

1. Motivation and problem formulation

ProtDBench was proposed in response to four issues identified in prior binder-design evaluation. First, success was usually judged by structure-prediction confidence metrics such as pLDDT, ipTM, ipAE, pTM, which are only proxies for binding success and had not been validated consistently across targets and design regimes. Second, prior evaluations lacked standardization in target selection, hotspot definitions, filtering thresholds, verifier choice, and computational budgets. Third, many reports emphasized per-sample success rate without throughput awareness, obscuring the trade-off between quality and speed. Fourth, filtering pipelines often contained implicit choices and were difficult to reproduce (Liu et al., 5 May 2026).

Within ProtDBench, binder design is formalized as a two-stage problem. A generative method first produces binder backbones, and a verifier or filter pipeline then determines which candidates count as successes. The paper writes the generative setup as

$$f_{\theta}(T_{\mathrm{seq}}, T_{\mathrm{str}}, c) \rightarrow (B_{\mathrm{seq}}, B_{\mathrm{str}})$$

where $T_{\mathrm{seq}}$ is the target sequence, $T_{\mathrm{str}}$ the target structure, $c$ the binding constraints such as hotspots, $B_{\mathrm{seq}}$ the binder sequence, and $B_{\mathrm{str}}$ the binder structure. A method generates a set of candidates

$$\mathcal{D}_{\text{seq}} = \{(B_{\mathrm{seq}}^{(i)}, B_{\mathrm{str}}^{(i)})\}_{i=1}^{N}.$$

ProtDBench then evaluates these candidates with sequence-level success rate, cluster-level success and diversity, structural consistency, and throughput under a fixed compute budget (Liu et al., 5 May 2026).

This framing makes the benchmark explicitly sensitive not only to whether a design passes a filter, but also to how the definition of success itself depends on evaluation design. A plausible implication is that ProtDBench treats evaluation protocol as part of the scientific object of study, rather than as an implementation detail.

2. Benchmark tasks and target specification

ProtDBench uses 10 targets, selected to span different structural classes and difficulty levels: BHRF1, SC2RBD, IL-7RA, PD-L1, TrkA, Insulin, H1, VEGF-A, IL-17A, and TNF$\alpha$ (Liu et al., 5 May 2026). The paper notes that multi-chain or more complex cases include H1, VEGF-A, IL-17A, and TNF$\alpha$.

For each target, ProtDBench specifies the crop region, hotspot residues, natural binder context, and allowed binder length range. The paper gives examples of these task definitions. BHRF1 is defined with crop A2–158, hotspots A65/A74/A77/A82/A85/A93, and binder length 80–120. SC2RBD uses crop E333–526, hotspots E485/E489/E494/E500/E505, and binder length 80–120. IL-7RA uses crop B17–209, hotspots B58/B80/B139, and binder length 50–120. TNF$\alpha$ uses crop A12–157/B12–157/C12–157, hotspots A113/C73, and binder length 50–120 (Liu et al., 5 May 2026).

The target set also includes two crop modifications relative to AlphaProteo conventions. The paper states that H1 residue 46 was excluded because it had an insertion code unsupported by many baselines, and IL-17A residues A30–A40 were omitted because they were unresolved in the crystal structure (Liu et al., 5 May 2026). These adjustments are minor in scope but important methodologically, because they show that the benchmark fixes target definitions with explicit compatibility and data-quality constraints rather than inheriting them uncritically.

3. Standardized evaluation pipeline

ProtDBench replaces heterogeneous, paper-specific assessment practices with a single reproducible pipeline. For each generated backbone, the benchmark samples $T_{\mathrm{seq}}$0 sequences with ProteinMPNN, evaluates each sequence with a fixed verifier or filter, marks the backbone successful if at least one of its eight sequences passes, clusters only successful backbones by structural similarity, and reports sequence-level success rate, cluster-level success or diversity, and throughput-adjusted success (Liu et al., 5 May 2026).

This standardization fixes the verifier, the sequence sampling procedure, the pass or fail rule, the clustering method, and the compute budget. The result is that benchmark outcomes can be compared under a common protocol rather than through method-specific reporting conventions. In practice, this matters because much of the apparent variation across binder-design papers may arise from evaluation design rather than model quality alone.

ProtDBench also defines explicit sequence-level and backbone-level success functions. A sequence is successful if

$T_{\mathrm{seq}}$1

where $T_{\mathrm{seq}}$2 is the verifier score for the sequence–backbone pair and $T_{\mathrm{seq}}$3 is the threshold. The per-sequence success rate is

$T_{\mathrm{seq}}$4

At the backbone level,

$T_{\mathrm{seq}}$5

This distinction between sequence-level and backbone-level success is central to the benchmark’s design, because the pipeline explicitly treats backbone generation and downstream sequence realization as separate sources of variability (Liu et al., 5 May 2026).

4. Verifier models, filtering rules, and bias

A central conclusion of ProtDBench is that evaluation verifiers are not interchangeable. The benchmark analyzes structure-prediction models and their confidence metrics on the Cao wet-lab annotated dataset, which contains experimentally labeled de novo binder candidates that were not prefiltered with AlphaFold-based methods. The verifiers considered include AF2-IG, ColabFold, ESMFold, Boltz-1, Boltz-2, Chai-1, Protenix, and Protenix-Mini; the extracted metrics include ipTM, ipAE, pLDDT, and pTM (Liu et al., 5 May 2026).

For single-metric filtering, the paper defines a top-$T_{\mathrm{seq}}$6 filter

$T_{\mathrm{seq}}$7

with $T_{\mathrm{seq}}$8. Given experimental labels $T_{\mathrm{seq}}$9, filtering accuracy is

$T_{\mathrm{str}}$0

The reported findings are that the same candidate set can appear substantially different under different verifiers, no single verifier dominates all targets, AF3-style predictors, especially Protenix and Protenix-Mini, generally achieve stronger top-1% enrichment than AF2-IG, and Protenix-Mini remains competitive while reducing inference cost (Liu et al., 5 May 2026).

ProtDBench also studies agreement and complementarity across verifiers. The paper reports that union recall under top-1% filtering improves when combining multiple verifiers, while recall drops rapidly as the number of verifiers required to agree increases. It explicitly states that only a small fraction of true binders are consistently identified by multiple verifiers. This is interpreted as evidence of inductive bias differences across structure predictors (Liu et al., 5 May 2026). A plausible implication is that evaluator choice can shape the apparent design landscape, not merely measure it.

The benchmark further examines thresholded combined filters. Examples include AF3 with minimum ipAE $T_{\mathrm{str}}$1, binder pTM $T_{\mathrm{str}}$2, and complex RMSD $T_{\mathrm{str}}$3; AF2-IG with ipAE $T_{\mathrm{str}}$4, pLDDT $T_{\mathrm{str}}$5, and binder RMSD $T_{\mathrm{str}}$6; AF2-IG-easy with ipAE $T_{\mathrm{str}}$7, ipTM $T_{\mathrm{str}}$8, pLDDT $T_{\mathrm{str}}$9, and binder bound or unbound RMSD $c$0; and Protenix / Protenix-Mini with binder ipTM $c$1, binder pTM $c$2, and complex RMSD $c$3 (Liu et al., 5 May 2026). Threshold combinations are selected through a two-stage search that ranks metrics by top-1% enrichment or enrichment factor and then grid-searches thresholds to find Pareto-optimal combinations.

The paper gives the formal dominance definition

$c$4

with a solution called Pareto optimal if no other solution dominates it. ProtDBench ultimately reports that a simpler filter using only binder pTM and binder ipTM can work as well as or better than more complex triplets, while remaining more interpretable and robust (Liu et al., 5 May 2026).

5. Throughput-aware and diversity-aware metrics

One of ProtDBench’s defining features is that it does not treat success rate as sufficient. The benchmark introduces throughput-aware evaluation under a fixed 24-hour single NVIDIA A100 GPU budget, and this budget includes backbone generation, inverse folding, and verifier evaluation (Liu et al., 5 May 2026).

Total runtime is decomposed as

$c$5

where $c$6 is backbone generation time, $c$7 is time for sequence design by ProteinMPNN, and $c$8 is time for filtering or verifier scoring. The benchmark then defines the number of successful backbones expected in 24 hours as

$c$9

where $B_{\mathrm{seq}}$0 is the number of passing backbones, meaning that at least one sampled sequence for that backbone passes the filter (Liu et al., 5 May 2026).

The practical effect is that per-sequence success and throughput are explicitly decoupled. The paper states that diffusion methods generally have much higher throughput than hallucination methods, and that some hallucination methods can have high success but lower throughput because they are slower or rely on iterative internal filtering (Liu et al., 5 May 2026). This suggests that method rankings can change substantially when the unit of analysis shifts from fraction passing to successful output under a fixed compute budget.

ProtDBench also incorporates cluster-level success criteria to account for structural diversity. Passing backbones are clustered using Foldseek / TM-score similarity. If $B_{\mathrm{seq}}$1 is the set of clusters formed from successful backbones, then the cluster count is

$B_{\mathrm{seq}}$2

and the diversity-adjusted success rate is

$B_{\mathrm{seq}}$3

The purpose is to distinguish many distinct solution families from many near-duplicate backbones. The paper explicitly argues that practical design benefits from structural diversity rather than repeated rediscovery of one successful scaffold (Liu et al., 5 May 2026).

ProtDBench additionally defines a structural consistency metric that tests whether a designed backbone can be reproduced by an independent structure predictor after sequence design. For a designed sequence $B_{\mathrm{seq}}$4 from backbone $B_{\mathrm{seq}}$5, with predicted structure $B_{\mathrm{seq}}$6 from Protenix-Mini, the consistency indicator is

$B_{\mathrm{seq}}$7

with $B_{\mathrm{seq}}$8 Å, and the structural consistency rate is

$B_{\mathrm{seq}}$9

This metric asks whether the backbone remains structurally realizable after sequence design rather than merely whether it passed a verifier threshold (Liu et al., 5 May 2026).

6. Benchmarked methods, empirical findings, and significance

ProtDBench benchmarks representative open-source hotspot-conditioned binder-design methods. The diffusion-based methods are RFdiffusion-3, BoltzGen, Protpardelle-1, ODesign, and PXDesign. The hallucination-based methods are BindCraft and BoltzDesign-1. The paper excludes Latent-X, AlphaProteo, Chai-2, and SeedProteo from head-to-head comparison because code or weights were not publicly available (Liu et al., 5 May 2026).

Several empirical conclusions are emphasized. First, filtering rules strongly affect measured success: different verifiers produce different top-1% enrichment, combined filters vary substantially, and a stricter verifier such as Protenix-Mini yields much lower absolute success than AF2-IG-Easy. Second, verifier disagreement is real: models recover different subsets of true binders, and requiring agreement across multiple verifiers sharply reduces recall. Third, throughput and success are decoupled: high per-sequence success does not necessarily imply high 24-hour yield. Fourth, diversity forms a separate axis: hallucination-based methods often produce more structurally diverse successful backbones, whereas diffusion-based methods can concentrate on fewer structural modes. Fifth, structural consistency differs by target and method, with PXDesign highlighted as especially consistent across targets, while SC2RBD is identified as challenging for all methods. Sixth, the appendix reports secondary-structure biases: all-alpha binders generally perform better than mainly-beta binders on both AF2- and Protenix-based metrics (Liu et al., 5 May 2026).

These findings define the benchmark’s broader significance. ProtDBench argues that fair protein-binder benchmarking requires more than a single success rate under a single verifier. It therefore recommends standardized targets, hotspots, and length constraints; explicit reporting of verifier choice; throughput-aware metrics; cluster-level diversity reporting; and fixed, reproducible evaluation pipelines (Liu et al., 5 May 2026). In this sense, ProtDBench is not merely a leaderboard framework. It is a methodological intervention into how computational protein binder design is assessed, exposing systematic differences induced by filtering rules, success definitions, and throughput-aware evaluation between computational efficiency, nominal success rate, and structural diversity.