Model-based reinforcement learning for protein backbone design (2405.01983v1)
Abstract: Designing protein nanomaterials of predefined shape and characteristics has the potential to dramatically impact the medical industry. Machine learning (ML) has proven successful in protein design, reducing the need for expensive wet lab experiment rounds. However, challenges persist in efficiently exploring the protein fitness landscapes to identify optimal protein designs. In response, we propose the use of AlphaZero to generate protein backbones, meeting shape and structural scoring requirements. We extend an existing Monte Carlo tree search (MCTS) framework by incorporating a novel threshold-based reward and secondary objectives to improve design precision. This innovation considerably outperforms existing approaches, leading to protein backbones that better respect structural scores. The application of AlphaZero is novel in the context of protein backbone design and demonstrates promising performance. AlphaZero consistently surpasses baseline MCTS by more than 100% in top-down protein design tasks. Additionally, our application of AlphaZero with secondary objectives uncovers further promising outcomes, indicating the potential of model-based reinforcement learning (RL) in navigating the intricate and nuanced aspects of protein design
- Model-based reinforcement learning for biological sequence design. In International conference on learning representations, 2019.
- Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26–38, 2017.
- Determination of protein structural ensembles using cryo-electron microscopy. Current Opinion in Structural Biology, 56:37–45, 2019.
- Robust deep learning–based protein sequence design using proteinmpnn. Science, 378(6615):49–56, 2022.
- Multifunctional materials through modular protein engineering. Advanced Materials, 24(29):3923–3940, 2012.
- Deep learning in protein structural modeling and design. Patterns, 1(9), 2020.
- Winner takes it all: Training performant rl populations for combinatorial optimization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
- Optimization of protein models. Wiley Interdisciplinary Reviews: Computational Molecular Science, 2(3):479–493, 2012.
- Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026–1034, 2015.
- Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583–589, 2021.
- Learning combinatorial optimization algorithms over graphs. Advances in neural information processing systems, 30, 2017.
- Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
- Recent advances in (therapeutic protein) drug development. F1000Research, 6, 2017.
- Ranked reward: Enabling self-play reinforcement learning for combinatorial optimization. arXiv preprint arXiv:1807.01672, 2018.
- Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016.
- Top-down design of protein architectures with reinforcement learning. Science, 380(6642):266–273, 2023.
- Protein 3d structure computed from evolutionary sequence variation. PloS one, 6(12):e28766, 2011.
- Reinforcement learning for combinatorial optimization: A survey. Computers & Operations Research, 134:105400, 2021.
- Curriculum learning for reinforcement learning domains: A framework and survey. The Journal of Machine Learning Research, 21(1):7382–7431, 2020.
- Mol* Viewer: modern web app for 3D visualization and analysis of large biomolecular structures. Nucleic Acids Research, 49(W1):W431–W437, 05 2021. ISSN 0305-1048. doi: 10.1093/nar/gkab314. URL https://doi.org/10.1093/nar/gkab314.
- Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016.
- Mastering the game of go without human knowledge. nature, 550(7676):354–359, 2017.
- A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140–1144, 2018.
- Reinforcement learning: An introduction. MIT press, 2018.
- Self-play reinforcement learning guides protein engineering. Nature Machine Intelligence, 5(8):845–860, 2023.
- Machine learning-assisted directed protein evolution with combinatorial libraries. Proceedings of the National Academy of Sciences, 116(18):8852–8858, 2019.
- Se (3) diffusion model with application to protein backbone generation. arXiv preprint arXiv:2302.02277, 2023.
- A review of recurrent neural networks: Lstm cells and network architectures. Neural computation, 31(7):1235–1270, 2019.
- Enzyme discovery and engineering for sustainable plastic recycling. Trends in biotechnology, 40(1):22–37, 2022.
- Frederic Renard (1 paper)
- Cyprien Courtot (1 paper)
- Alfredo Reichlin (8 papers)
- Oliver Bent (12 papers)