Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Multi-Modal Representation Learning for Molecular Property Prediction: Sequence, Graph, Geometry (2401.03369v2)

Published 7 Jan 2024 in q-bio.MN, cs.LG, and q-bio.BM

Abstract: Molecular property prediction refers to the task of labeling molecules with some biochemical properties, playing a pivotal role in the drug discovery and design process. Recently, with the advancement of machine learning, deep learning-based molecular property prediction has emerged as a solution to the resource-intensive nature of traditional methods, garnering significant attention. Among them, molecular representation learning is the key factor for molecular property prediction performance. And there are lots of sequence-based, graph-based, and geometry-based methods that have been proposed. However, the majority of existing studies focus solely on one modality for learning molecular representations, failing to comprehensively capture molecular characteristics and information. In this paper, a novel multi-modal representation learning model, which integrates the sequence, graph, and geometry characteristics, is proposed for molecular property prediction, called SGGRL. Specifically, we design a fusion layer to fusion the representation of different modalities. Furthermore, to ensure consistency across modalities, SGGRL is trained to maximize the similarity of representations for the same molecule while minimizing similarity for different molecules. To verify the effectiveness of SGGRL, seven molecular datasets, and several baselines are used for evaluation and comparison. The experimental results demonstrate that SGGRL consistently outperforms the baselines in most cases. This further underscores the capability of SGGRL to comprehensively capture molecular information. Overall, the proposed SGGRL model showcases its potential to revolutionize molecular property prediction by leveraging multi-modal representation learning to extract diverse and comprehensive molecular insights. Our code is released at https://github.com/Vencent-Won/SGGRL.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. Rethinking drug design in the artificial intelligence era. Nature Reviews Drug Discovery, 19(5):353–364, 2020.
  2. Deep learning methods for molecular representation and property prediction. Drug Discovery Today, page 103373, 2022.
  3. Applications of deep learning in molecule generation and molecular property prediction. Accounts of chemical research, 54(2):263–270, 2020.
  4. Geometry-enhanced molecular representation learning for property prediction. Nature Machine Intelligence, 4(2):127–134, 2022.
  5. Seq3seq fingerprint: towards end-to-end semi-supervised deep drug discovery. In Proceedings of the 2018 ACM international conference on bioinformatics, computational biology, and health informatics, pages 404–413, 2018.
  6. A compact review of molecular property prediction with graph neural networks. Drug Discovery Today: Technologies, 37:1–12, 2020.
  7. David Weininger. Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of chemical information and computer sciences, 28(1):31–36, 1988.
  8. Smiles-bert: large scale unsupervised pre-training for molecular property prediction. In Proceedings of the 10th ACM international conference on bioinformatics, computational biology and health informatics, pages 429–436, 2019.
  9. Seq2seq fingerprint: An unsupervised deep molecular embedding for drug discovery. In Proceedings of the 8th ACM international conference on bioinformatics, computational biology, and health informatics, pages 285–294, 2017.
  10. Smiles transformer: Pre-trained molecular fingerprint for low data drug discovery. arXiv preprint arXiv:1911.04738, 2019.
  11. Communicative representation learning on attributed molecular graphs. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, pages 2831–2838. International Joint Conferences on Artificial Intelligence Organization, 7 2020. Main track.
  12. Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263–1272. PMLR, 2017.
  13. Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems, 34:6790–6802, 2021.
  14. Graphaf: a flow-based autoregressive model for molecular graph generation. In International Conference on Learning Representations, 2020.
  15. Asgn: An active semi-supervised graph neural network for molecular property prediction. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 731–752, 2020.
  16. 3d infomax improves gnns for molecular property prediction. In International Conference on Machine Learning, pages 20479–20502. PMLR, 2022.
  17. Automated 3d pre-training for molecular property prediction. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2419–2430, 2023.
  18. Graseq: graph and sequence fusion learning for molecular property prediction. In Proceedings of the 29th ACM international conference on information & knowledge management, pages 435–443, 2020.
  19. Geomgcl: Geometric graph contrastive learning for molecular property prediction. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 4541–4549, 2022.
  20. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9(2):513–530, 2018.
  21. Exploring chemical space using natural language processing methodologies for drug discovery. Drug Discovery Today, 25(4):689–705, 2020.
  22. Identifying structure–property relationships through smiles syntax analysis with self-attention mechanism. Journal of chemical information and modeling, 59(2):914–923, 2019.
  23. Molecular fingerprint similarity search in virtual screening. Methods, 71:58–63, 2015.
  24. Fast and accurate modeling of molecular atomization energies with machine learning. Physical review letters, 108(5):058301, 2012.
  25. Analyzing learned molecular representations for property prediction. Journal of chemical information and modeling, 59(8):3370–3388, 2019.
  26. Geom, energy-annotated molecular conformations for property prediction and molecular generation. Scientific Data, 9(1):185, 2022.
  27. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826, 2018.
  28. Communicative representation learning on attributed molecular graphs. In IJCAI, volume 2020, pages 2831–2838, 2020.
  29. Uni-mol: A universal 3d molecular representation learning framework. In The Eleventh International Conference on Learning Representations, 2023.
  30. Molecular graph enhanced transformer for retrosynthesis prediction. Neurocomputing, 457:193–202, 2021.
  31. Pre-training molecular graph representation with 3d geometry. In International Conference on Learning Representations, 2022.
  32. Pre-training molecular graph representation with 3d geometry. arXiv preprint arXiv:2110.07728, 2021.
  33. Strategies for pre-training graph neural networks. In International Conference on Learning Representations (ICLR), 2020.
  34. Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493, 2015.
  35. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  36. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Zeyu Wang (137 papers)
  2. Tianyi Jiang (5 papers)
  3. Jinhuan Wang (27 papers)
  4. Qi Xuan (113 papers)
Citations (5)
View on Semantic Scholar

Summary

Multi-Modal Representation Learning for Molecular Property Prediction: Sequence, Graph, Geometry

The paper "Multi-Modal Representation Learning for Molecular Property Prediction: Sequence, Graph, Geometry" introduces a novel approach to molecular property prediction. This approach leverages multi-modal representation learning techniques that utilize sequence, graph, and geometry modalities, aiming to enhance the precision of molecular property prediction tasks. The paper's proposal, SGGRL, is a next-step evolution in the landscape of molecular representation learning, addressing the limitations of single-modality models by capturing a more comprehensive set of molecular characteristics.

The research places its foundation on the premise that each modality—the molecular sequence (such as SMILES strings), molecular graph (depicting atom connectivity), and geometry (3D molecular structures)—captures distinct aspects of molecular data which are critical for accurate prediction of biochemical properties. This paper argues that models focusing on single modalities often miss out on integrating valuable complementary information available in other forms, which could enhance the predictive performance.

Methodological Approach

SGGRL introduces a robust architecture composed of a sequence encoder, graph encoder, and geometry encoder. Each module is specially tailored for a specific modality:

  • Sequence Encoder: Leveraging a Bi-LSTM to preprocess SMILES sequences ensures the model effectively captures bidirectional context which is essential in dealing with the non-directionality of SMILES.
  • Graph Encoder: Based on Communicative Message Passing Neural Networks (CMPNN), this component utilizes the atom connectivity information to extract meaningful representations, emphasizing the role of structural topology.
  • Geometry Encoder: Employing the GEMGNN architecture, this module taps into the 3D geometrical conformation of molecules, augmenting the representation with detailed spatial configurations.

A salient feature of the SGGRL is the utilization of a GlobalAttentionPool layer to amalgamate representations from different modalities, followed by a fusion layer that combines them into a joint representation. This enables the capturing of comprehensive information while addressing data redundancy issues.

Furthermore, SGGRL employs a contrastive learning mechanism, NT-Xent Loss, across different modalities to maintain consistency and compatibility of feature spaces. This design choice ensures that the model captures universal molecular features that are congruent across modalities, avoiding the pitfalls of modality-specific biases.

Experimental Results

Empirical evaluations conducted on seven benchmark datasets demonstrate that SGGRL holds significant advantages over existing state-of-the-art models, particularly in both classification and regression tasks associated with molecular property prediction. The comparative analysis shows that SGGRL consistently outperforms popular models like GraSeq, 3D Infomax, and GraphMVP across different datasets, including small-scale tasks where comprehensive representation becomes more crucial.

Implications and Future Directions

The implications of SGGRL are manifold, suggesting that multi-modal integration could be a pivotal factor in advancing molecular property prediction tasks. By providing a more holistic view of molecular structures and properties, SGGRL could significantly impact drug discovery and biochemical research, potentially streamlining processes and reducing costs.

Future work could focus on refining each of the modal encoders to incorporate more advanced machine learning models that can capture intricate modality-specific features more effectively. Additionally, further research could explore real-time applications within biochemical simulations or predictive modeling in medicinal chemistry, extending the utility of SGGRL beyond the datasets analyzed.

In summary, this paper presents a compelling case for multi-modal representation learning in molecular property prediction, highlighting the potential of integrating sequence, graph, and geometry data to achieve superior predictive accuracy. The approach could redefine approaches in bioinformatics, opening new avenues for research and application in artificial intelligence-driven drug discovery.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown