---
title: 'ProtDBench: Protein Binder Benchmark'
url: https://www.emergentmind.com/topics/protdbench
type: topic
---

# ProtDBench: Protein Binder Benchmark

Searching arXiv for the ProtDBench paper and closely related benchmark context.
ProtDBench is a standardized, throughput-aware benchmark for protein binder design that was introduced to make evaluation of de novo binder generators fair, reproducible, and comparable across methods. It targets a long-standing problem in the field: reported in silico results are difficult to interpret across studies because prior evaluations used different targets, hotspot definitions, verifier models, score thresholds, filtering rules, and computational budgets. ProtDBench addresses this by defining unified benchmark tasks, a fixed evaluation pipeline, explicit verifier and filter configurations, throughput-aware metrics under a fixed 24-hour budget, and cluster-level success criteria that account for structural diversity [2605.04118].

## 1. Motivation and problem formulation

ProtDBench was proposed in response to four issues identified in prior binder-design evaluation. First, success was usually judged by structure-prediction confidence metrics such as **pLDDT, ipTM, ipAE, pTM**, which are only proxies for binding success and had not been validated consistently across targets and design regimes. Second, prior evaluations lacked standardization in target selection, hotspot definitions, filtering thresholds, verifier choice, and computational budgets. Third, many reports emphasized per-sample success rate without throughput awareness, obscuring the trade-off between quality and speed. Fourth, filtering pipelines often contained implicit choices and were difficult to reproduce [2605.04118].

Within ProtDBench, binder design is formalized as a two-stage problem. A generative method first produces binder backbones, and a verifier or filter pipeline then determines which candidates count as successes. The paper writes the generative setup as
$$
f_{\theta}(T_{\mathrm{seq}}, T_{\mathrm{str}}, c) \rightarrow (B_{\mathrm{seq}}, B_{\mathrm{str}})
$$
where \(T_{\mathrm{seq}}\) is the target sequence, \(T_{\mathrm{str}}\) the target structure, \(c\) the binding constraints such as hotspots, \(B_{\mathrm{seq}}\) the binder sequence, and \(B_{\mathrm{str}}\) the binder structure. A method generates a set of candidates
$$
\mathcal{D}_{\text{seq}} = \{(B_{\mathrm{seq}}^{(i)}, B_{\mathrm{str}}^{(i)})\}_{i=1}^{N}.
$$
ProtDBench then evaluates these candidates with sequence-level success rate, cluster-level success and diversity, structural consistency, and throughput under a fixed compute budget [2605.04118].

This framing makes the benchmark explicitly sensitive not only to whether a design passes a filter, but also to how the definition of success itself depends on evaluation design. A plausible implication is that ProtDBench treats evaluation protocol as part of the scientific object of study, rather than as an implementation detail.

## 2. Benchmark tasks and target specification

ProtDBench uses **10 targets**, selected to span different structural classes and difficulty levels: **BHRF1**, **SC2RBD**, **IL-7RA**, **PD-L1**, **TrkA**, **Insulin**, **H1**, **VEGF-A**, **IL-17A**, and **TNF\(\alpha\)** [2605.04118]. The paper notes that multi-chain or more complex cases include **H1**, **VEGF-A**, **IL-17A**, and **TNF\(\alpha\)**.

For each target, ProtDBench specifies the crop region, hotspot residues, natural binder context, and allowed binder length range. The paper gives examples of these task definitions. **BHRF1** is defined with crop **A2–158**, hotspots **A65/A74/A77/A82/A85/A93**, and binder length **80–120**. **SC2RBD** uses crop **E333–526**, hotspots **E485/E489/E494/E500/E505**, and binder length **80–120**. **IL-7RA** uses crop **B17–209**, hotspots **B58/B80/B139**, and binder length **50–120**. **TNF\(\alpha\)** uses crop **A12–157/B12–157/C12–157**, hotspots **A113/C73**, and binder length **50–120** [2605.04118].

The target set also includes two crop modifications relative to AlphaProteo conventions. The paper states that **H1 residue 46** was excluded because it had an insertion code unsupported by many baselines, and **IL-17A residues A30–A40** were omitted because they were unresolved in the crystal structure [2605.04118]. These adjustments are minor in scope but important methodologically, because they show that the benchmark fixes target definitions with explicit compatibility and data-quality constraints rather than inheriting them uncritically.

## 3. Standardized evaluation pipeline

ProtDBench replaces heterogeneous, paper-specific assessment practices with a single reproducible pipeline. For each generated backbone, the benchmark samples **\(m=8\)** sequences with **ProteinMPNN**, evaluates each sequence with a fixed verifier or filter, marks the backbone successful if at least one of its eight sequences passes, clusters only successful backbones by structural similarity, and reports sequence-level success rate, cluster-level success or diversity, and throughput-adjusted success [2605.04118].

This standardization fixes the verifier, the sequence sampling procedure, the pass or fail rule, the clustering method, and the compute budget. The result is that benchmark outcomes can be compared under a common protocol rather than through method-specific reporting conventions. In practice, this matters because much of the apparent variation across binder-design papers may arise from evaluation design rather than model quality alone.

ProtDBench also defines explicit sequence-level and backbone-level success functions. A sequence is successful if
$$
\mathcal{F}(s) = \mathbb{I}\bigl[\mathrm{Score}(s, B_{\mathrm{str}}) \ge \tau\bigr],
$$
where \(\mathrm{Score}(s, B_{\mathrm{str}})\) is the verifier score for the sequence–backbone pair and \(\tau\) is the threshold. The per-sequence success rate is
$$
\mathrm{SR} = \frac{ \sum_{i=1}^{N} \sum_{s \in \mathcal{S}_i} \mathcal{F}(s) }{ \sum_{i=1}^{N} |\mathcal{S}_i| }.
$$
At the backbone level,
$$
\mathcal{F}_{\mathrm{bb}(i)} = \mathbb{I}\bigl[\exists\, s\in \mathcal{S}_i : \mathcal{F}(s)=1\bigr].
$$
This distinction between sequence-level and backbone-level success is central to the benchmark’s design, because the pipeline explicitly treats backbone generation and downstream sequence realization as separate sources of variability [2605.04118].

## 4. Verifier models, filtering rules, and bias

A central conclusion of ProtDBench is that evaluation verifiers are not interchangeable. The benchmark analyzes structure-prediction models and their confidence metrics on the **Cao wet-lab annotated dataset**, which contains experimentally labeled de novo binder candidates that were not prefiltered with AlphaFold-based methods. The verifiers considered include **AF2-IG**, **ColabFold**, **ESMFold**, **Boltz-1**, **Boltz-2**, **Chai-1**, **Protenix**, and **Protenix-Mini**; the extracted metrics include **ipTM**, **ipAE**, **pLDDT**, and **pTM** [2605.04118].

For single-metric filtering, the paper defines a top-\(1\%\) filter
$$
\mathcal{F}_{\alpha}(B) = \mathbb{I} \left[ s(B) \ge \mathrm{Quantile}_{1-\alpha} \bigl(\{s(B') \mid B' \in \mathcal{D} \}\bigr) \right],
$$
with \(\alpha = 1\%\). Given experimental labels \(y(B)\in\{0,1\}\), filtering accuracy is
$$
\mathrm{SR} = \frac{ \sum_{B \in \mathcal{D}} \mathcal{F}_{\alpha}(B)\, y(B) }{ \sum_{B \in \mathcal{D}} \mathcal{F}_{\alpha}(B) }.
$$
The reported findings are that the same candidate set can appear substantially different under different verifiers, no single verifier dominates all targets, **AF3-style predictors**, especially **Protenix** and **Protenix-Mini**, generally achieve stronger top-1% enrichment than **AF2-IG**, and **Protenix-Mini** remains competitive while reducing inference cost [2605.04118].

ProtDBench also studies agreement and complementarity across verifiers. The paper reports that union recall under top-1% filtering improves when combining multiple verifiers, while recall drops rapidly as the number of verifiers required to agree increases. It explicitly states that only a small fraction of true binders are consistently identified by multiple verifiers. This is interpreted as evidence of inductive bias differences across structure predictors [2605.04118]. A plausible implication is that evaluator choice can shape the apparent design landscape, not merely measure it.

The benchmark further examines thresholded combined filters. Examples include **AF3** with minimum **ipAE \(< 1.5\)**, binder **pTM \(> 0.8\)**, and complex **RMSD \(< 2.5 \text{ Å}\)**; **AF2-IG** with **ipAE \(< 7.0\)**, **pLDDT \(> 0.9\)**, and binder **RMSD \(< 1.5 \text{ Å}\)**; **AF2-IG-easy** with **ipAE \(< 10.85\)**, **ipTM \(> 0.5\)**, **pLDDT \(> 0.8\)**, and binder bound or unbound **RMSD \(< 3.5 \text{ Å}\)**; and **Protenix / Protenix-Mini** with binder **ipTM \(> 0.85\)**, binder **pTM \(> 0.88\)**, and complex **RMSD \(< 2.5 \text{ Å}\)** [2605.04118]. Threshold combinations are selected through a two-stage search that ranks metrics by top-1% enrichment or enrichment factor and then grid-searches thresholds to find Pareto-optimal combinations.

The paper gives the formal dominance definition
$$
x_1 \text{ dominates } x_2 \iff \forall i,\ \mathrm{SR}_i(x_1)\ge \mathrm{SR}_i(x_2) \ \text{and}\ \exists i,\ \mathrm{SR}_i(x_1) > \mathrm{SR}_i(x_2),
$$
with a solution called Pareto optimal if no other solution dominates it. ProtDBench ultimately reports that a simpler filter using only **binder pTM** and **binder ipTM** can work as well as or better than more complex triplets, while remaining more interpretable and robust [2605.04118].

## 5. Throughput-aware and diversity-aware metrics

One of ProtDBench’s defining features is that it does not treat success rate as sufficient. The benchmark introduces throughput-aware evaluation under a fixed **24-hour single NVIDIA A100 GPU budget**, and this budget includes backbone generation, inverse folding, and verifier evaluation [2605.04118].

Total runtime is decomposed as
$$
T_{\mathrm{tot}} = T_{\mathrm{sample}} + T_{\mathrm{MPNN}} + T_{\mathrm{eval}},
$$
where \(T_{\mathrm{sample}}\) is backbone generation time, \(T_{\mathrm{MPNN}}\) is time for sequence design by ProteinMPNN, and \(T_{\mathrm{eval}}\) is time for filtering or verifier scoring. The benchmark then defines the number of successful backbones expected in 24 hours as
$$
N_{24h} = \frac{24\ \mathrm{hours}}{T_{\mathrm{tot}}} \cdot N_{\mathrm{pass}},
$$
where \(N_{\mathrm{pass}}\) is the number of passing backbones, meaning that at least one sampled sequence for that backbone passes the filter [2605.04118].

The practical effect is that per-sequence success and throughput are explicitly decoupled. The paper states that diffusion methods generally have much higher throughput than hallucination methods, and that some hallucination methods can have high success but lower throughput because they are slower or rely on iterative internal filtering [2605.04118]. This suggests that method rankings can change substantially when the unit of analysis shifts from fraction passing to successful output under a fixed compute budget.

ProtDBench also incorporates cluster-level success criteria to account for structural diversity. Passing backbones are clustered using **Foldseek / TM-score similarity**. If \(\mathcal{C}_{\mathrm{pass}}\) is the set of clusters formed from successful backbones, then the cluster count is
$$
\mathrm{S}_{\mathrm{cluster}} = \left| \mathcal{C}_{\mathrm{pass}} \right|
$$
and the diversity-adjusted success rate is
$$
\mathrm{R}_{\mathrm{cluster}} = \frac{\mathrm{S}_{\mathrm{cluster}}}{N_{\mathrm{total}}} \times 100\%.
$$
The purpose is to distinguish many distinct solution families from many near-duplicate backbones. The paper explicitly argues that practical design benefits from structural diversity rather than repeated rediscovery of one successful scaffold [2605.04118].

ProtDBench additionally defines a structural consistency metric that tests whether a designed backbone can be reproduced by an independent structure predictor after sequence design. For a designed sequence \(s\) from backbone \(B_{\mathrm{str}}^{\mathrm{backbone}}\), with predicted structure \(\hat{B}_{\mathrm{str}}^{\mathrm{backbone}}(s)\) from **Protenix-Mini**, the consistency indicator is
$$
\mathcal{C}(s) = \mathbb{I} \Bigl[ \operatorname{RMSD}\!\bigl( B_{\mathrm{str}}^{\mathrm{backbone}}, \hat{B}_{\mathrm{str}}^{\mathrm{backbone}}(s) \bigr) < \delta \Bigr]
$$
with \(\delta = 2.5\) Å, and the structural consistency rate is
$$
\mathrm{CR} = \frac{1}{\sum_i |\mathcal{S}_i|} \sum_i \sum_{s \in \mathcal{S}_i} \mathcal{C}(s).
$$
This metric asks whether the backbone remains structurally realizable after sequence design rather than merely whether it passed a verifier threshold [2605.04118].

## 6. Benchmarked methods, empirical findings, and significance

ProtDBench benchmarks representative **open-source** hotspot-conditioned binder-design methods. The diffusion-based methods are **RFdiffusion-3**, **BoltzGen**, **Protpardelle-1**, **ODesign**, and **PXDesign**. The hallucination-based methods are **BindCraft** and **BoltzDesign-1**. The paper excludes **Latent-X**, **AlphaProteo**, **Chai-2**, and **SeedProteo** from head-to-head comparison because code or weights were not publicly available [2605.04118].

Several empirical conclusions are emphasized. First, filtering rules strongly affect measured success: different verifiers produce different top-1% enrichment, combined filters vary substantially, and a stricter verifier such as **Protenix-Mini** yields much lower absolute success than **AF2-IG-Easy**. Second, verifier disagreement is real: models recover different subsets of true binders, and requiring agreement across multiple verifiers sharply reduces recall. Third, throughput and success are decoupled: high per-sequence success does not necessarily imply high 24-hour yield. Fourth, diversity forms a separate axis: hallucination-based methods often produce more structurally diverse successful backbones, whereas diffusion-based methods can concentrate on fewer structural modes. Fifth, structural consistency differs by target and method, with **PXDesign** highlighted as especially consistent across targets, while **SC2RBD** is identified as challenging for all methods. Sixth, the appendix reports secondary-structure biases: all-alpha binders generally perform better than mainly-beta binders on both AF2- and Protenix-based metrics [2605.04118].

These findings define the benchmark’s broader significance. ProtDBench argues that fair protein-binder benchmarking requires more than a single success rate under a single verifier. It therefore recommends standardized targets, hotspots, and length constraints; explicit reporting of verifier choice; throughput-aware metrics; cluster-level diversity reporting; and fixed, reproducible evaluation pipelines [2605.04118]. In this sense, ProtDBench is not merely a leaderboard framework. It is a methodological intervention into how computational protein binder design is assessed, exposing systematic differences induced by filtering rules, success definitions, and throughput-aware evaluation between computational efficiency, nominal success rate, and structural diversity.

Source: https://www.emergentmind.com/topics/protdbench