Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

FREED++: Improving RL Agents for Fragment-Based Molecule Generation by Thorough Reproduction (2401.09840v1)

Published 18 Jan 2024 in q-bio.BM, cs.LG, and stat.ML

Abstract: A rational design of new therapeutic drugs aims to find a molecular structure with desired biological functionality, e.g., an ability to activate or suppress a specific protein via binding to it. Molecular docking is a common technique for evaluating protein-molecule interactions. Recently, Reinforcement Learning (RL) has emerged as a promising approach to generating molecules with the docking score (DS) as a reward. In this work, we reproduce, scrutinize and improve the recent RL model for molecule generation called FREED (arXiv:2110.01219). Extensive evaluation of the proposed method reveals several limitations and challenges despite the outstanding results reported for three target proteins. Our contributions include fixing numerous implementation bugs and simplifying the model while increasing its quality, significantly extending experiments, and conducting an accurate comparison with current state-of-the-art methods for protein-conditioned molecule generation. We show that the resulting fixed model is capable of producing molecules with superior docking scores compared to alternative approaches.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (101)
  1. Fast, accurate, and reliable molecular docking with QuickVina 2. Bioinformatics, 31(13):2214–2216, 02 2015. ISSN 1367-4803. doi: 10.1093/bioinformatics/btv082. URL https://doi.org/10.1093/bioinformatics/btv082.
  2. New substructure filters for removal of pan assay interference compounds (pains) from screening libraries and for their exclusion in bioassays. Journal of Medicinal Chemistry, 53(7):2719–2740, 2010. doi: 10.1021/jm901137j. URL https://doi.org/10.1021/jm901137j. PMID: 20131845.
  3. A renaissance of neural networks in drug discovery. Expert Opin Drug Discov, 11(8):785–795, July 2016.
  4. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 34:27381–27394, 2021.
  5. The Protein Data Bank. Nucleic Acids Research, 28(1):235–242, 01 2000. ISSN 0305-1048. doi: 10.1093/nar/28.1.235. URL https://doi.org/10.1093/nar/28.1.235.
  6. Memory-assisted reinforcement learning for diverse molecular de novo design. Journal of cheminformatics, 12(1):1–17, 2020.
  7. Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling, 59(3):1096–1108, 2019.
  8. CodeBPE: Investigating subtokenization options for large language model pretraining on source code. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=htL4UZ344nF.
  9. Machine learning of accurate energy-conserving molecular force fields. Science advances, 3(5):e1603015, 2017.
  10. We should at least be able to design molecules that dock well. arXiv preprint arXiv:2006.16955, 2020.
  11. Structural biology contributions to the discovery of drugs to treat chronic myelogenous leukaemia. Acta Crystallographica Section D, 63(1):80–93, Jan 2007. doi: 10.1107/S0907444906047287. URL https://doi.org/10.1107/S0907444906047287.
  12. Multi-task neural networks for qsar predictions, 2014.
  13. MolGAN: An implicit generative model for small molecular graphs. ICML 2018 workshop on Theoretical Foundations and Applications of Deep Generative Models, 2018.
  14. On the art of compiling and using ’drug-like’ chemical fragment spaces. ChemMedChem, 3:1503–7, 10 2008. doi: 10.1002/cmdc.200800178.
  15. LEA3D: a computer-aided ligand design for structure-based drug design. J Med Chem, 48(7):2457–2468, April 2005.
  16. Structure-aware generation of drug-like molecules. arXiv preprint arXiv:2111.04107, 2021.
  17. Spice, a dataset of drug-like molecules and peptides for training machine learning potentials. Scientific Data, 10(1):11, 2023.
  18. Three-dimensional convolutional neural networks and a cross-docked data set for structure-based drug design. Journal of chemical information and modeling, 60(9):4200–4215, 2020.
  19. Chembl: a large-scale bioactivity database for drug discovery. Nucleic acids research, 40(D1):D1100–D1107, 2012.
  20. Deep learning in drug discovery. Mol Inform, 35(1):3–14, December 2015.
  21. Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules. Advances in neural information processing systems, 32, 2019.
  22. Searching for high-value molecules using reinforcement learning and transformers, 2023.
  23. Automatic chemical design using a data-driven continuous representation of molecules. ACS central science, 4(2):268–276, 2018.
  24. 3d equivariant diffusion for target-aware molecule generation and affinity prediction. arXiv preprint arXiv:2303.03543, 2023.
  25. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pp.  1352–1361. PMLR, 2017.
  26. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp.  1861–1870. PMLR, 2018a.
  27. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018b.
  28. Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning, pp.  8867–8887. PMLR, 2022.
  29. Binding moad (mother of all databases). Proteins: Structure, Function, and Bioinformatics, 60(3):333–340, 2005.
  30. Equivariant 3d-conditional diffusion models for molecular linker design. arXiv preprint arXiv:2210.05274, 2022.
  31. Zinc: A free tool to discover chemistry for biology. Journal of Chemical Information and Modeling, 52(7):1757–1768, 2012. doi: 10.1021/ci3001277. URL https://doi.org/10.1021/ci3001277. PMID: 22587354.
  32. Qmugs, quantum mechanical properties of drug-like molecules. Scientific Data, 9(1):273, 2022.
  33. Multi-objective GFlowNets. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp.  14631–14653. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/jain23a.html.
  34. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=rkE3y85ee.
  35. Multiplicative interactions and where to find them. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rylnK6VtDH.
  36. Autonomous molecule generation using reinforcement learning and docking to develop potential novel inhibitors. Scientific reports, 10(1):22104, 2020.
  37. Junction tree variational autoencoder for molecular graph generation. In International conference on machine learning, pp.  2323–2332. PMLR, 2018.
  38. Hiroshi Kajino. Molecular hypergraph grammar with its application to molecular optimization. In International Conference on Machine Learning, pp.  3183–3191. PMLR, 2019.
  39. nabladft: Large-scale conformational energy and hamiltonian prediction benchmark and dataset. Physical Chemistry Chemical Physics, 24(42):25853–25863, 2022.
  40. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=SJU4ayYgl.
  41. USP7 inhibitors in cancer immunotherapy: Current status and perspective. Cancers (Basel), 14(22), November 2022.
  42. Deep-learning: investigating deep neural networks hyper-parameters and comparison of performance to shallow methods for modeling bioactivity data. Journal of Cheminformatics, 9(1):42, Jun 2017. ISSN 1758-2946. doi: 10.1186/s13321-017-0226-y. URL https://doi.org/10.1186/s13321-017-0226-y.
  43. rdkit/rdkit: 2022_03_1 (q1 2022) release, March 2022. URL https://doi.org/10.5281/zenodo.6388425.
  44. Defining and maintaining a high quality screening collection: the GSK experience. Drug Discov Today, 11(5-6):267–272, March 2006.
  45. Discovery of potent, selective, and orally bioavailable inhibitors of usp7 with in vivo antitumor activity. Journal of Medicinal Chemistry, 63(10):5398–5420, May 2020a. ISSN 0022-2623. doi: 10.1021/acs.jmedchem.0c00245. URL https://doi.org/10.1021/acs.jmedchem.0c00245.
  46. Discovery of potent, selective, and orally bioavailable inhibitors of usp7 with in vivo antitumor activity. Journal of Medicinal Chemistry, 63(10):5398–5420, 2020b. doi: 10.1021/acs.jmedchem.0c00245. URL https://doi.org/10.1021/acs.jmedchem.0c00245. PMID: 32302140.
  47. Recent advances in the development of ubiquitin-specific-processing protease 7 (USP7) inhibitors. Eur J Med Chem, 191:112107, February 2020.
  48. Structure-based de novo drug design using 3d deep generative models. Chemical science, 12(41):13664–13675, 2021.
  49. Shu-Kun Lin. Pharmacophore perception, development and use in drug design. edited by osman f. güner. Molecules, 5(7):987–989, 2000. ISSN 1420-3049. doi: 10.3390/50700987. URL https://www.mdpi.com/1420-3049/5/7/987.
  50. Experimental and computational approaches to estimate solubility and permeability in drug discovery and development settings. Advanced Drug Delivery Reviews, 23(1):3–25, 1997. ISSN 0169-409X. doi: https://doi.org/10.1016/S0169-409X(96)00423-1. URL https://www.sciencedirect.com/science/article/pii/S0169409X96004231. In Vitro Models for Selection of Development Candidates.
  51. Generating 3d molecules for target protein binding. In International Conference on Machine Learning, 2022.
  52. Paul D Lyne. Structure-based virtual screening: an overview. Drug discovery today, 7(20):1047–1055, 2002.
  53. Deep neural nets as a method for quantitative structure–activity relationships. Journal of Chemical Information and Modeling, 55(2):263–274, Feb 2015. ISSN 1549-9596. doi: 10.1021/ci500747n. URL https://doi.org/10.1021/ci500747n.
  54. Deeptox: Toxicity prediction using deep learning. Frontiers in Environmental Science, 3, 2016. ISSN 2296-665X. doi: 10.3389/fenvs.2015.00080. URL https://www.frontiersin.org/articles/10.3389/fenvs.2015.00080.
  55. Molecule generation using transformers and policy gradient reinforcement learning. Scientific Reports, 13(1):8799, May 2023. ISSN 2045-2322. doi: 10.1038/s41598-023-35648-w. URL https://doi.org/10.1038/s41598-023-35648-w.
  56. Playing atari with deep reinforcement learning. ArXiv, abs/1312.5602, 2013.
  57. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, Feb 2015. ISSN 1476-4687. doi: 10.1038/nature14236. URL https://doi.org/10.1038/nature14236.
  58. H. L. Morgan. The generation of a unique machine description for chemical structures-a technique developed at chemical abstracts service. Journal of Chemical Documentation, 5(2):107–113, May 1965. ISSN 0021-9576. doi: 10.1021/c160017a018. URL https://doi.org/10.1021/c160017a018.
  59. Directory of useful decoys, enhanced (dud-e): Better ligands and decoys for better benchmarking. Journal of Medicinal Chemistry, 55(14):6582–6594, 2012. doi: 10.1021/jm300687e. URL https://doi.org/10.1021/jm300687e. PMID: 22716043.
  60. Open babel: An open chemical toolbox. Journal of Cheminformatics, 3(1):33, Oct 2011. ISSN 1758-2946. doi: 10.1186/1758-2946-3-33. URL https://doi.org/10.1186/1758-2946-3-33.
  61. Molecular de-novo design through deep reinforcement learning. Journal of Cheminformatics, 9(1):48, Sep 2017. ISSN 1758-2946. doi: 10.1186/s13321-017-0235-x. URL https://doi.org/10.1186/s13321-017-0235-x.
  62. Highlights in usp7 inhibitors for cancer treatment. Frontiers in Chemistry, 10, 2022. ISSN 2296-2646. doi: 10.3389/fchem.2022.1005727. URL https://www.frontiersin.org/articles/10.3389/fchem.2022.1005727.
  63. SureChEMBL: a large-scale, chemically annotated patent document database. Nucleic Acids Research, 44(D1):D1220–D1228, 11 2015. ISSN 0305-1048. doi: 10.1093/nar/gkv1253. URL https://doi.org/10.1093/nar/gkv1253.
  64. Time limits in reinforcement learning. In International Conference on Machine Learning, pp.  4045–4054. PMLR, 2018.
  65. An empirical study of tokenization strategies for various korean nlp tasks. CoRR, abs/2010.02534, 2020. URL https://arxiv.org/abs/2010.02534.
  66. Pocket2mol: Efficient molecular sampling based on 3d protein pockets. In International Conference on Machine Learning, pp.  17644–17655. PMLR, 2022.
  67. Optimization of phenyl-substituted benzimidazole carboxamide poly(adp-ribose) polymerase inhibitors: Identification of (s)-2-(2-fluoro-4-(pyrrolidin-2-yl)phenyl)-1h-benzimidazole-4-carboxamide (a-966492), a highly potent and efficacious inhibitor. Journal of Medicinal Chemistry, 53(8):3142–3153, Apr 2010. ISSN 0022-2623. doi: 10.1021/jm901775y. URL https://doi.org/10.1021/jm901775y.
  68. BREED: Generating novel inhibitors through hybridization of known ligands. application to CDK2, p38, and HIV protease. J Med Chem, 47(11):2768–2775, May 2004.
  69. Pavel Polishchuk. Crem: chemically reasonable mutations framework for structure generation. Journal of Cheminformatics, 12(1):28, Apr 2020. ISSN 1758-2946. doi: 10.1186/s13321-020-00431-w. URL https://doi.org/10.1186/s13321-020-00431-w.
  70. Molecular sets (moses): a benchmarking platform for molecular generation models. Frontiers in pharmacology, 11:565644, 2020.
  71. Deep reinforcement learning for de novo drug design. Science advances, 4(7):eaap7885, 2018.
  72. Massively multitask networks for drug discovery. arXiv preprint arXiv:1502.02072, 2015.
  73. Exploring chemical space for drug discovery using the chemical universe database. ACS chemical neuroscience, 3(9):649–657, 2012.
  74. Extended-connectivity fingerprints. Journal of chemical information and modeling, 50(5):742–754, 2010.
  75. GroupBuild: a fragment-based method for de novo drug design. J Med Chem, 36(12):1700–1710, June 1993.
  76. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015.
  77. Computer-based de novo design of drug-like molecules. Nat Rev Drug Discov, 4(8):649–663, August 2005.
  78. Structure-based drug design with equivariant diffusion models, 2023. URL https://openreview.net/forum?id=uKmuzIuVl8z.
  79. Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS Central Science, 4(1):120–131, 2018. doi: 10.1021/acscentsci.7b00512. URL https://doi.org/10.1021/acscentsci.7b00512. PMID: 29392184.
  80. Sema: Antigen b-cell conformational epitope prediction using deep transfer learning. Frontiers in Immunology, 13, 2022. ISSN 1664-3224. doi: 10.3389/fimmu.2022.960985. URL https://www.frontiersin.org/articles/10.3389/fimmu.2022.960985.
  81. Brian K Shoichet. Virtual screening of chemical libraries. Nature, 432(7019):862–865, 2004.
  82. Symmetry-aware actor-critic for 3d molecular design. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=jEYKjPE1xYN.
  83. Autogrow4: an open-source genetic algorithm for de novo drug design and lead optimization. Journal of Cheminformatics, 12(1):25, Apr 2020. ISSN 1758-2946. doi: 10.1186/s13321-020-00429-4. URL https://doi.org/10.1186/s13321-020-00429-4.
  84. Design and structure-based study of new potential FKBP12 inhibitors. Biophys J, 85(5):3194–3201, November 2003.
  85. Graph convolutional networks for computational drug development and discovery. Brief. Bioinform., 21(3):919–935, May 2020.
  86. Charformer: Fast character transformers via gradient-based subword tokenization. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=JtBRnrlOEFN.
  87. Comparison of structure-and ligand-based scoring functions for deep generative models: a gpcr case study. Journal of Cheminformatics, 13(1):1–20, 2021.
  88. Deep learning as an opportunity in virtual screening. In Proceedings of the deep learning workshop at NIPS, volume 27, pp.  1–9, 2014.
  89. Atomnet: A deep convolutional neural network for bioactivity prediction in structure-based drug discovery. CoRR, abs/1510.02855, 2015. URL http://dblp.uni-trier.de/db/journals/corr/corr1510.html#WallachDH15.
  90. Structural basis for molecular recognition at serotonin receptors. Science, 340(6132):610–614, 2013. doi: 10.1126/science.1232807. URL https://www.science.org/doi/abs/10.1126/science.1232807.
  91. Learning to perceive and act by trial and error. Machine Learning, 7(1):45–83, Jul 1991. ISSN 1573-0565. doi: 10.1023/A:1022619109594. URL https://doi.org/10.1023/A:1022619109594.
  92. A compact review of molecular property prediction with graph neural networks. Drug Discovery Today: Technologies, 37:1–12, 2020. ISSN 1740-6749. doi: https://doi.org/10.1016/j.ddtec.2020.11.009. URL https://www.sciencedirect.com/science/article/pii/S1740674920300305.
  93. Moleculenet: a benchmark for molecular machine learning. Chem. Sci., 9:513–530, 2018. doi: 10.1039/C7SC02664A. URL http://dx.doi.org/10.1039/C7SC02664A.
  94. Hit and lead discovery with explorative rl and fragment-based molecule generation. Advances in Neural Information Processing Systems, 34:7924–7936, 2021.
  95. Graph convolutional policy network for goal-directed molecular graph generation. Advances in neural information processing systems, 31, 2018.
  96. LigBuilder v3: A Multi-Target de novo drug design approach. Front Chem, 8:142, February 2020.
  97. Selective and orally bioavailable phenylglycine tissue factor/factor viia inhibitors. Bioorganic & Medicinal Chemistry Letters, 15(23):5344–5352, 2005. ISSN 0960-894X. doi: https://doi.org/10.1016/j.bmcl.2005.04.079. URL https://www.sciencedirect.com/science/article/pii/S0960894X05011169.
  98. {AMBERT}: A pre-trained language model with multi-grained tokenization, 2021. URL https://openreview.net/forum?id=DMxOBm06HUx.
  99. Deep learning enables rapid identification of potent ddr1 kinase inhibitors. Nature biotechnology, 37(9):1038–1040, 2019.
  100. Optimization of molecules via deep reinforcement learning. Scientific reports, 9(1):1–10, 2019.
  101. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 3, AAAI’08, pp.  1433–1438. AAAI Press, 2008. ISBN 9781577353683.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (11)
  1. Alexander Telepov (3 papers)
  2. Artem Tsypin (4 papers)
  3. Kuzma Khrabrov (3 papers)
  4. Sergey Yakukhnov (1 paper)
  5. Pavel Strashnov (2 papers)
  6. Petr Zhilyaev (6 papers)
  7. Egor Rumiantsev (4 papers)
  8. Daniel Ezhov (1 paper)
  9. Manvel Avetisian (7 papers)
  10. Olga Popova (2 papers)
  11. Artur Kadurin (6 papers)
Citations (1)
View on Semantic Scholar