Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

PDB-Struct: A Comprehensive Benchmark for Structure-based Protein Design (2312.00080v1)

Published 30 Nov 2023 in q-bio.QM and cs.LG

Abstract: Structure-based protein design has attracted increasing interest, with numerous methods being introduced in recent years. However, a universally accepted method for evaluation has not been established, since the wet-lab validation can be overly time-consuming for the development of new algorithms, and the $\textit{in silico}$ validation with recovery and perplexity metrics is efficient but may not precisely reflect true foldability. To address this gap, we introduce two novel metrics: refoldability-based metric, which leverages high-accuracy protein structure prediction models as a proxy for wet lab experiments, and stability-based metric, which assesses whether models can assign high likelihoods to experimentally stable proteins. We curate datasets from high-quality CATH protein data, high-throughput $\textit{de novo}$ designed proteins, and mega-scale experimental mutagenesis experiments, and in doing so, present the $\textbf{PDB-Struct}$ benchmark that evaluates both recent and previously uncompared protein design methods. Experimental results indicate that ByProt, ProteinMPNN, and ESM-IF perform exceptionally well on our benchmark, while ESM-Design and AF-Design fall short on the refoldability metric. We also show that while some methods exhibit high sequence recovery, they do not perform as well on our new benchmark. Our proposed benchmark paves the way for a fair and comprehensive evaluation of protein design methods in the future. Code is available at https://github.com/WANG-CR/PDB-Struct.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (53)
  1. Protein generation with evolutionary diffusion: sequence is all you need. bioRxiv, pages 2023–09, 2023.
  2. Protein sequence design with a learned potential. Nature communications, 13(1):746, 2022.
  3. De novo protein design by deep network hallucination. Nature, 600(7889):547–552, 2021.
  4. How cryo-em is revolutionizing structural biology. Trends in biochemical sciences, 40(1):49–57, 2015.
  5. The protein data bank. Nucleic acids research, 28(1):235–242, 2000.
  6. Design of protein-binding proteins from the target structure alone. Nature, 605(7910):551–560, 2022.
  7. Therapeutic approaches to protein-misfolding diseases. Nature, 426(6968):905–909, 2003.
  8. Robust deep learning–based protein sequence design using proteinmpnn. Science, 378(6615):49–56, 2022.
  9. Knowledge-design: Pushing the limit of protein deign via knowledge refinement. arXiv preprint arXiv:2305.15151, 2023a.
  10. Pifold: Toward effective and efficient protein inverse folding. In International Conference on Learning Representations, 2023b. URL https://openreview.net/forum?id=oMsN9TYwJ0j.
  11. Proteininvbench: Benchmarking protein inverse folding on diverse tasks, models, and metrics. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023c.
  12. A high-level programming language for generative protein design. bioRxiv, pages 2022–12, 2022.
  13. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  14. Learning inverse folding from millions of predicted structures. ICML, 2022. doi: 10.1101/2022.04.10.487779. URL https://www.biorxiv.org/content/early/2022/04/10/2022.04.10.487779.
  15. The coming of age of de novo protein design. Nature, 537(7620):320–327, 2016.
  16. Generative models for graph-based protein design. Advances in neural information processing systems, 32, 2019.
  17. Illuminating protein space with a programmable generative model. BioRxiv, pages 2022–12, 2022.
  18. Learning from protein structure with geometric vector perceptrons. arXiv preprint arXiv:2009.01411, 2020.
  19. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583–589, 2021.
  20. Structure determination by X-ray crystallography, volume 233. Springer, 1977.
  21. Direct prediction of profiles of sequences compatible with a protein structure by neural networks with fragment-based local and energy-based nonlocal profiles. Proteins: Structure, Function, and Bioinformatics, 82(10):2565–2573, 2014.
  22. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.
  23. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637):1123–1130, 2023.
  24. Joint generation of protein sequence and structure with rosettafold sequence space diffusion. bioRxiv, pages 2023–05, 2023.
  25. Antigen-specific antibody design and optimization with diffusion-based generative models for protein structures. 2022. URL https://openreview.net/forum?id=jSorGn2Tjg.
  26. Modeling protein structure using geometric vector field networks. bioRxiv, pages 2023–05, 2023.
  27. Mmseqs2 desktop and local web server app for fast, interactive sequence searches. Bioinformatics, 35(16):2856–2858, 2019.
  28. Colabfold: making protein folding accessible to all. Nature methods, 19(6):679–682, 2022.
  29. Size exclusion chromatography. Springer Science & Business Media, 1999.
  30. Spin2: Predicting sequence profiles from protein structures using deep neural networks. Proteins: Structure, Function, and Bioinformatics, 86(6):629–633, 2018.
  31. Cath–a hierarchic classification of protein domain structures. Structure, 5(8):1093–1109, 1997.
  32. Using alphafold to predict the impact of single mutations on protein stability and function. Plos one, 18(3):e0282689, 2023.
  33. Densecpd: improving the accuracy of neural-network-based computational protein sequence design with densenet. Journal of chemical information and modeling, 60(3):1245–1252, 2020.
  34. Cystic fibrosis: a disease of altered protein folding. Journal of Bioenergetics and Biomembranes, 29:483–490, 1997.
  35. Global analysis of protein folding using massively parallel design, synthesis, and testing. Science, 357(6347):168–175, 2017.
  36. Protein sequence and structure co-design with equivariant translation. arXiv preprint arXiv:2210.08761, 2022.
  37. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology, 35(11):1026–1028, 2017.
  38. Uniref clusters: a comprehensive and scalable alternative for improving sequence similarity searches. Bioinformatics, 31(6):926–932, 2015.
  39. Generative de novo protein design with global context. arXiv preprint arXiv:2204.10673, 2022.
  40. Diffusion probabilistic modeling of protein backbones in 3d for the motif-scaffolding problem. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=6TxBxqNME1Y.
  41. Mega-scale experimental analysis of protein folding stability in biology and design. Nature, 620(7973):434–444, 2023.
  42. Highly accurate protein structure prediction for the human proteome. Nature, 596(7873):590–596, 2021.
  43. Language models generalize beyond natural proteins. bioRxiv, pages 2022–12, 2022.
  44. Scaffolding protein functional sites using deep learning. Science, 377(6604):387–394, 2022.
  45. Broadly applicable and accurate protein design by integrating structure prediction networks and diffusion generative models. BioRxiv, pages 2022–12, 2022.
  46. High-resolution de novo structure prediction from primary sequence. BioRxiv, pages 2022–07, 2022.
  47. How significant is a protein structure similarity with tm-score= 0.5? Bioinformatics, 26(7):889–895, 2010.
  48. Improved protein structure prediction using predicted interresidue orientations. Proceedings of the National Academy of Sciences, 117(3):1496–1503, 2020.
  49. De novo design of luciferases using deep learning. Nature, 614(7949):774–780, 2023.
  50. Graph denoising diffusion for inverse protein folding. arXiv preprint arXiv:2306.16819, 2023.
  51. Pre-training via denoising for molecular property prediction. arXiv preprint arXiv:2206.00133, 2022.
  52. Tm-align: a protein structure alignment algorithm based on the tm-score. Nucleic acids research, 33(7):2302–2309, 2005.
  53. Structure-informed language models are protein designers. bioRxiv, pages 2023–02, 2023.
Citations (4)
View on Semantic Scholar

Summary

  • The paper introduces PDB-Struct with novel refoldability and stability metrics to rigorously assess protein design methodologies.
  • It leverages advanced prediction models like AlphaFold2 and ESMFold to evaluate structural integrity using TM-scores and pLDDT scores.
  • The benchmark integrates diverse datasets, revealing performance variations among models to guide improvements in de novo protein design.

A Comprehensive Evaluation Framework for Structure-based Protein Design: Introducing PDB-Struct

Protein design, especially de novo design, is a crucial aspect of bioengineering with extensive applications in therapeutics, enzyme engineering, and antibody innovation. The integration of deep learning techniques into protein design has significantly enhanced the ability to generate novel protein sequences that possess specific structural properties. Despite these advances, there exists a notable gap in standardized benchmarks for evaluating various protein design methodologies. The paper "PDB-Struct: A Comprehensive Benchmark for Structure-based Protein Design" addresses this deficit by introducing two novel metrics to bolster evaluation rigor: refoldability-based metrics and stability-based metrics.

Introduction to Novel Metrics

Refoldability-Based Metrics: Existing in silico methods primarily rely on sequence recovery and perplexity to evaluate protein design models. However, these metrics can be misleading as they may not accurately reflect the true foldability of the designed proteins. Even a high sequence similarity might not guarantee correct folding, as single mutations can lead to misfolding and diseases such as Alzheimer's. The authors propose "refoldability" as a metric, which assesses whether designed sequences can fold into a stable structure and if that structure resembles the input. High-accuracy structure prediction models, such as AlphaFold2 and ESMFold, are utilized to predict and evaluate the structural stability and similarity, measured by TM-scores and pLDDT scores.

Stability-Based Metrics: The perplexity of a designed sequence, based on various assumptions by different models, introduces significant challenges for comparison. The authors advocate for a stability-based metric using datasets with experimental stability scores. This metric evaluates whether a design method can correctly assign likelihoods to sequences with high experimental stability, providing a more reliable estimate of a model's capacity to predict sequence landscapes.

Datasets and Benchmark Composition

The benchmark, PDB-Struct, comprises datasets curated from the CATH protein structure database, high-throughput de novo designed proteins, and mega-scale mutagenesis experiments. This comprehensive collection ensures that the proposed metrics can evaluate a wide array of design models fairly and thoroughly.

Experimental Evaluation and Findings

The experiments conducted using PDB-Struct reveal nuanced insights:

  • ProteinMPNN, ByProt, and ESM-IF demonstrated notable performance on refoldability metrics with AlphaFold2 prediction exceeding Ref-TM scores of 0.80, evidencing their robustness in maintaining structural integrity and stability.
  • AF-Design and ESM-Design, despite their innovative integration of structure prediction networks, showed limited efficacy in both metrics, indicating a need for enhanced efficiency and prediction accuracy.
  • There was discrete consistency observed across different structure prediction models (AlphaFold2, OmegaFold, ESMFold) when evaluating refoldability, affirming the robustness of the proposed metrics.
  • Stability metrics uncovered that ESM-IF performs exceptionally well, indicating accurate sequence density estimation. On the other hand, structure-prediction based models did not appear as sensitive to point mutations as expected.

Implications and Future Directions

The introduction of PDB-Struct sets a foundation for more rigorous and consistent benchmarking in protein design. By addressing current metric limitations and introducing novel evaluative strategies, this work significantly enhances the precision of model assessment. Practically, these metrics can improve the selection process of models for drug discovery, enzyme design, and synthetic biology.

Future research could extend PDB-Struct by integrating additional datasets and developing metrics to assess diversity and functional specificity of designed proteins. Furthermore, adapting these metrics for real-time application in industrial and clinical environments remains an exciting avenue for exploration.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets