Cross-Modal Learning for Chemistry Property Prediction: Large Language Models Meet Graph Machine Learning (2408.14964v1)

Abstract: In the field of chemistry, the objective is to create novel molecules with desired properties, facilitating accurate property predictions for applications such as material design and drug screening. However, existing graph deep learning methods face limitations that curb their expressive power. To address this, we explore the integration of vast molecular domain knowledge from LLMs with the complementary strengths of Graph Neural Networks (GNNs) to enhance performance in property prediction tasks. We introduce a Multi-Modal Fusion (MMF) framework that synergistically harnesses the analytical prowess of GNNs and the linguistic generative and predictive abilities of LLMs, thereby improving accuracy and robustness in predicting molecular properties. Our framework combines the effectiveness of GNNs in modeling graph-structured data with the zero-shot and few-shot learning capabilities of LLMs, enabling improved predictions while reducing the risk of overfitting. Furthermore, our approach effectively addresses distributional shifts, a common challenge in real-world applications, and showcases the efficacy of learning cross-modal representations, surpassing state-of-the-art baselines on benchmark datasets for property prediction tasks.

Cross-Modal Learning for Chemistry Property Prediction: LLMs Meet Graph Machine Learning

The paper "Cross-Modal Learning for Chemistry Property Prediction: LLMs Meet Graph Machine Learning" addresses an important challenge in computational chemistry: predicting the properties of novel molecules, crucial for applications such as material design and drug discovery. The unique proposition of this paper lies in the integration of LLMs with Graph Neural Networks (GNNs) to enhance predictive accuracy and robustness.

Introduction

Significant advancements have been witnessed in the field of molecular property prediction with the use of GNNs. Representing molecules as graphs has become common, facilitating the exploration of structural and functional properties of molecules. Despite these advancements, existing GNN-based methods face limitations such as limited expressive power, over-squashing, and over-smoothing, which curtail their performance. Simultaneously, LLMs have revolutionized natural language processing tasks and demonstrated capacities in zero-shot and few-shot learning, promising avenues for comprehensive learning frameworks. The integration of LLMs with GNNs for molecular property prediction, however, is relatively unexplored, presenting a distinct opportunity for innovation.

Proposed Framework

The paper introduces a Multi-Modal Fusion (MMF) framework aimed at leveraging the strengths of both GNNs and LLMs. The framework is structured to enhance the scope of molecular property prediction through three primary methodologies:

1. Synergistic Cross-Modal Embedding Generation (SEG):
   • The SEG approach employs zero-shot chain-of-thought (CoT) prompting to guide LLMs in generating detailed textual descriptions of molecules represented in SMILES notation.
   • These descriptions are subsequently used to fine-tune smaller LLMs (LMs) to generate context-aware token embeddings applicable for property prediction.
   • Parallelly, GNNs utilize methodologies like Chebyshev Graph Convolution (CGC) to interpret complex molecular graphs, generating molecular graph-level embeddings.
   • The text-level and graph-level embeddings are integrated using a multi-head attention mechanism to form robust, semantically enriched cross-modal embeddings.
2. Predictive Embedding Generation (PEG):
   • Few-shot In-Context Learning (ICL) is employed wherein LLMs are directed to predict molecular properties using demonstrations consisting of input-output pairs from training data.
   • This method capitalizes on the pre-trained knowledge of LLMs to generate prediction embeddings for new molecules, minimizing the need for domain-specific fine-tuning.
3. Mixture-of-Experts (MOE) Dynamic Prediction:
   • At the output layer, the framework employs a MOE mechanism with a gating system that dynamically integrates cross-modal and prediction embeddings.
   • This system optimizes the weight distribution based on predictive performance, enabling high-precision property predictions and reducing the risk of overfitting.

Experimental Results

The effectiveness of the proposed MMF framework is demonstrated through comprehensive experiments on six publicly available molecular property prediction benchmark datasets, including QM8 and QM9. The results highlight:

• QM8 Dataset:
  • The MMF framework achieves a Test MAE of $7.45 \times 10^{-3}$, significantly outperforming state-of-the-art models like LanczosNet and AdaLanczosNet by approximately 25.35%.
• QM9 Dataset:
  • The framework sets new benchmarks across multiple molecular properties such as Dipole Moment, HOMO, LUMO, and others, demonstrating superior performance compared to leading algorithms like SchNet, PhysNet, and DimeNet.

Implications and Future Work

The integration of GNNs and LLMs allows for the creation of a unified predictive framework that harnesses the structural insights from graphs and the semantic richness from natural language descriptions. This dual-modal approach significantly enhances the framework's robustness and accuracy. The ability to effectively handle distributional shifts and reduce overfitting makes this framework particularly useful for real-world applications in drug discovery and material science.

Moving forward, the scope for future research is extensive. Opportunities include refining LLMs for better interpretation of molecular structure representations, exploring other LLM architectures, and optimizing prompts for even more efficient property predictions. Additionally, the application of this framework to other domains like environmental science and biotechnology could present further advancements.

Overall, the introduction of such a comprehensive and multifaceted approach marks a significant step in the continuous effort to enhance molecular property prediction methods, fostering advancements in various scientific and industrial domains.