LLaMo: Large Language Model-based Molecular Graph Assistant (2411.00871v1)

Abstract: LLMs have demonstrated remarkable generalization and instruction-following capabilities with instruction tuning. The advancements in LLMs and instruction tuning have led to the development of Large Vision-LLMs (LVLMs). However, the competency of the LLMs and instruction tuning have been less explored in the molecular domain. Thus, we propose LLaMo: LLM-based Molecular graph assistant, which is an end-to-end trained large molecular graph-LLM. To bridge the discrepancy between the language and graph modalities, we present the multi-level graph projector that transforms graph representations into graph tokens by abstracting the output representations of each GNN layer and motif representations with the cross-attention mechanism. We also introduce machine-generated molecular graph instruction data to instruction-tune the large molecular graph-LLM for general-purpose molecule and language understanding. Our extensive experiments demonstrate that LLaMo shows the best performance on diverse tasks, such as molecular description generation, property prediction, and IUPAC name prediction. The code of LLaMo is available at https://github.com/mlvlab/LLaMo.

Overview of LLaMo: A LLM-based Molecular Graph Assistant

The paper at hand introduces LLaMo: a LLM-based Molecular Graph Assistant. This novel tool effectively combines molecular graph encoders with advanced LLMs to enhance the instruction-following capabilities specifically within the molecular domain. Here, we witness a structured attempt to bridge the gap between language and molecular graph modalities, an area relatively understudied despite the success of LLMs in natural language processing and vision-language tasks.

Key Contributions and Methodology

LLaMo integrates a molecular graph encoder with a multi-level graph projector and a LLM. The molecular graph encoder utilizes Graph Neural Networks (GNNs) to process 2D molecular structures, translating these into node representations via message-passing frameworks. The multi-level graph projector is pivotal as it transforms these node representations into molecular graph tokens suitable for LLM processing. This transformation incorporates a cross-attention mechanism, ensuring multi-scale data — from atoms to entire functional motifs — is captured.

A significant contribution of this work is the generation of machine-instructed molecular graph instruction data. This addresses the scarcity of such data by converting molecular descriptions and IUPAC names into multi-turn conversation formats, enhancing the instruction-tuning process.

Experimental Evaluation

The efficacy of LLaMo is evidenced by its superior performance over existing models, including GPT-4, across multiple tasks such as molecular description generation, property prediction, and IUPAC name prediction. Notably, the multi-level graph projector shows an ability to effectively abstract detailed molecular information, outperforming simple linear or high-level projection methods.

The two-stage training strategy is notably effective. Initial training aligns the graph encoder with the LLM, while instruction-tuning further refines the model's ability to follow diverse molecular task instructions. The use of LoRA for fine-tuning is particularly noteworthy, mitigating the computational cost typically associated with training large models.

Implications and Future Directions

The research holds significant promise for advancing molecular machine learning. By effectively integrating molecular graph representations with LLM capabilities, LLaMo sets a benchmark for molecular tasks that require understanding textual and graphical data simultaneously.

Future work may focus on expanding the dataset to include more diverse molecular representations or improving scalability. Moreover, further exploration into minimizing LLM-associated issues, such as hallucination and implicit data leakage, will be crucial for practical applications.

LLaMo presents a promising step in augmenting the capabilities of LLMs with molecular graph understanding, potentially facilitating advancements in areas such as drug discovery and material science. The integration strategies and architectural innovations introduced here could inform future work on multi-modal AI systems beyond the molecular field.