Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Bidirectional Generation of Structure and Properties Through a Single Molecular Foundation Model (2211.10590v4)

Published 19 Nov 2022 in cs.LG and cs.AI

Abstract: The recent success of large foundation models in artificial intelligence has prompted the emergence of chemical pre-trained models. Despite the growing interest in large molecular pre-trained models that provide informative representations for downstream tasks, attempts for multimodal pre-training approaches on the molecule domain were limited. To address this, we present a novel multimodal molecular pre-trained model that incorporates the modalities of structure and biochemical properties, drawing inspiration from recent advances in multimodal learning techniques. Our proposed model pipeline of data handling and training objectives aligns the structure/property features in a common embedding space, which enables the model to regard bidirectional information between the molecules' structure and properties. These contributions emerge synergistic knowledge, allowing us to tackle both multimodal and unimodal downstream tasks through a single model. Through extensive experiments, we demonstrate that our model shows remarkable capabilities in solving various meaningful chemical challenges, including conditional molecule generation, property prediction, molecule classification, and reaction prediction.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (68)
  1. Accurate, reliable and interpretable solubility prediction of druglike molecules with attention pooling and bayesian learning (2022).
  2. polybert: A chemical language model to enable fully machine-driven ultrafast polymer informatics (2022).
  3. Pignet: A physics-informed deep learning model toward generalized drug–target interaction predictions (2022).
  4. Transpolymer: A transformer-based language model for polymer property predictions. arXiv.org (2022).
  5. Paul, D. et al. Artificial intelligence in drug discovery and development. Drug Discovery Today (2020).
  6. A generalized-template-based graph neural network for accurate organic reactivity prediction (2022).
  7. Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS Central Science 4, 120–131 (2017).
  8. Gómez-Bombarelli, R. et al. Automatic chemical design using a data-driven continuous representation of molecules. ACS Central Science 4, 268–276 (2018).
  9. Molecular generative model based on conditional variational autoencoder for de novo molecular design (2018).
  10. Scaffold-based molecular design with a graph generative model. Chem. Sci. 11, 1153–1164 (2020).
  11. A transformer-based generative model for de novo molecular design (2022).
  12. Molecular de-novo design through deep reinforcement learning. Journal of Cheminformatics 9 (2017).
  13. Optimization of molecules via deep reinforcement learning (2019).
  14. Chemberta: Large-scale self-supervised pretraining for molecular property prediction. arXiv preprint arXiv:2010.09885 (2020).
  15. Zhang, Z. et al. Can pre-trained models really learn better molecular representations for ai-aided drug discovery? arXiv:2209.07423 (2022).
  16. Melnyk, I. et al. Reprogramming large pretrained language models for antibody sequence infilling (2022).
  17. Bommasani, R. et al. On the opportunities and risks of foundation models (2021).
  18. Horawalavithana, S. et al. Foundation models of scientific knowledge for chemistry: Opportunities, challenges and lessons learned. In Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models, 160–172 (Association for Computational Linguistics, virtual+Dublin, 2022).
  19. N-gram graph: Simple unsupervised representation for graphs, with applications to molecules (2018).
  20. Rong, Y. et al. Self-supervised graph transformer on large-scale molecular data (2020).
  21. Chemberta-2: Towards chemical foundation models (2022).
  22. Chen, F. et al. Vlp: A survey on vision-language pre-training (2022).
  23. Vaswani, A. et al. Attention is all you need (2017).
  24. Chen, Y.-C. et al. Uniter: Universal image-text representation learning (2019).
  25. Li, X. et al. Oscar: Object-semantics aligned pre-training for vision-language tasks (2020).
  26. Radford, A. et al. Learning transferable visual models from natural language supervision (2021).
  27. Li, J. et al. Align before fuse: Vision and language representation learning with momentum distillation (2021).
  28. Yu, J. et al. Coca: Contrastive captioners are image-text foundation models (2022).
  29. Agrawal, A. et al. Vqa: Visual question answering. arXiv:1505.00468 [cs] (2016).
  30. Comparative analysis on cross-modal information retrieval: A review. Computer Science Review 39, 100336 (2021).
  31. Text-to-image diffusion models in generative ai: A survey. arXiv:2303.07909 [cs] (2023).
  32. Ming, Y. et al. Visuals to text: A comprehensive review on automatic image captioning. IEEE/CAA Journal of Automatica Sinica 9, 1339–1365 (2022).
  33. Learning continuous and data-driven molecular descriptors by translating equivalent chemical representations. Chem. Sci. 10, 1692–1701 (2019).
  34. Grill, J.-B. et al. Bootstrap your own latent: A new approach to self-supervised learning (2020).
  35. Zhu, J. et al. Dual-view molecule pre-training (2021).
  36. De novo molecular design and generative models. Drug Discovery Today (2021).
  37. Bort, W. et al. Inverse qsar: Reversing descriptor-driven prediction pipeline using attention-based conditional variational autoencoder. Journal of Chemical Information and Modeling 62, 5471–5484 (2022).
  38. Crawshaw, M. Multi-task learning with deep neural networks: A survey (2020).
  39. Self-supervised co-learning of uncurated images and reports enables oversight ai in radiology (2022).
  40. Kim, S. et al. Pubchem in 2021: new data content and improved web interfaces. Nucleic Acids Research 49, D1388–D1395 (2021).
  41. Zinc 15 – ligand discovery for everyone. Journal of Chemical Information and Modeling 55, 2324–2337 (2015).
  42. Wu, Z. et al. Moleculenet: a benchmark for molecular machine learning. Chem. Sci. 9, 513–530 (2018).
  43. Nonadditivity in public and inhouse data: implications for drug design 13 (2021).
  44. Kwapien, K. et al. Implications of additivity and nonadditivity for machine learning and deep learning models in drug design. ACS Omega 7, 26573–26581 (2022).
  45. Ai, H. et al. Predicting drug-induced liver injury using ensemble learning methods and molecular fingerprints. Toxicological Sciences 165, 100–107 (2018).
  46. Yang, K. et al. Analyzing learned molecular representations for property prediction. Journal of Chemical Information and Modeling 59, 3370–3388 (2019). PMID: 31361484.
  47. Hu, W. et al. Strategies for pre-training graph neural networks (2019).
  48. Fang, X. et al. Geometry-enhanced molecular representation learning for property prediction. Nature Machine Intelligence 4, 127–134 (2022).
  49. Ross, J. et al. Large-scale chemical language representations capture molecular structure and properties. Nature Machine Intelligence 4, 1256–1264 (2022).
  50. Schwaller, P. et al. Molecular transformer: A model for uncertainty-calibrated chemical reaction prediction. ACS Central Science 5, 1572–1583 (2019).
  51. State-of-the-art augmented nlp transformer models for direct and single-step retrosynthesis. Nature Communications 11 (2020).
  52. Chemformer: a pre-trained transformer for computational chemistry. Machine Learning: Science and Technology 3, 015022 (2022).
  53. Permutation invariant graph-to-sequence model for template-free retrosynthesis and reaction prediction. Journal of Chemical Information and Modeling 62, 3503–3513 (2022).
  54. Sacha, M. et al. Molecule edit graph attention network: Modeling chemical reactions as sequences of graph edits. Journal of Chemical Information and Modeling 61, 3273–3284 (2021).
  55. Predicting retrosynthetic reactions using self-corrected transformer neural networks 60, 47–55 (2020).
  56. Valid, plausible, and diverse retrosynthesis using tied two-way transformers with latent variables 61, 123–133 (2021).
  57. Drug-likeness scoring based on unsupervised learning (2021).
  58. Bert: Pre-training of deep bidirectional transformers for language understanding (2018).
  59. Caron, M. et al. Emerging properties in self-supervised vision transformers. arXiv:2104.14294 [cs] (2021).
  60. Dosovitskiy, A. et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv:2010.11929 [cs] (2020).
  61. Gage, P. A new algorithm for data compression (1994).
  62. Liu, Y. et al. Roberta: A robustly optimized bert pretraining approach (2019).
  63. Raffel, C. et al. Exploring the limits of transfer learning with a unified text-to-text transformer (2019).
  64. Landrum, G. Rdkit: Open-source cheminformatics software (2016).
  65. Momentum contrast for unsupervised visual representation learning (2019).
  66. Brown, T. B. et al. Language models are few-shot learners (2020).
  67. Bjerrum, E. Smiles enumeration as data augmentation for neural network modeling of molecules (2017).
  68. Ramsundar, B. et al. Deep Learning for the Life Sciences (O’Reilly Media, 2019).
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Jinho Chang (11 papers)
  2. Jong Chul Ye (210 papers)
Citations (15)
View on Semantic Scholar