Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Text (2308.06911v3)

Published 14 Aug 2023 in cs.LG, cs.CL, and q-bio.BM

Abstract: LLMs have made significant strides in natural language processing, enabling innovative applications in molecular science by processing textual representations of molecules. However, most existing LLMs cannot capture the rich information with complex molecular structures or images. In this paper, we introduce GIT-Mol, a multi-modal LLM that integrates the Graph, Image, and Text information. To facilitate the integration of multi-modal molecular data, we propose GIT-Former, a novel architecture that is capable of aligning all modalities into a unified latent space. We achieve a 5%-10% accuracy increase in properties prediction and a 20.2% boost in molecule generation validity compared to the baselines. With the any-to-language molecular translation strategy, our model has the potential to perform more downstream tasks, such as compound name recognition and chemical reaction prediction.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems 35, 23716–23736.
  2. One transformer fits all distributions in multi-modal diffusion at scale. arXiv preprint arXiv:2303.06555 .
  3. Vlmo: Unified vision-language pre-training with mixture-of-modality-experts. Advances in Neural Information Processing Systems 35, 32897–32912.
  4. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676 .
  5. An open source chemical structure curation pipeline using rdkit. Journal of Cheminformatics , 1–16.
  6. Generative models for molecular discovery: Recent advances and challenges. Wiley Interdisciplinary Reviews: Computational Molecular Science 12, e1608.
  7. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 .
  8. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 .
  9. Translation between molecules and natural language, in: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 375–413.
  10. Text2mol: Cross-modal molecule retrieval with natural language queries, in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 595–607.
  11. Optical structure recognition software to recover chemical information: Osra, an open source solution.
  12. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines 30, 681–694.
  13. Chebi in 2016: Improved services and an expanding collection of metabolites. Nucleic Acids Research 44, D1214–D1219.
  14. Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778.
  15. Graphmae: Self-supervised masked graph autoencoders, in: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 594–604.
  16. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302.14045 .
  17. Scaling up visual and vision-language representation learning with noisy text supervision, in: International Conference on Machine Learning, PMLR. pp. 4904–4916.
  18. Pubchem 2019 update: improved access to chemical data. Nucleic Acids Research 47, D1102–D1109.
  19. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 .
  20. Empowering molecule discovery for molecule-caption translation with large language models: A chatgpt perspective. arXiv preprint arXiv:2306.06615 .
  21. Druggpt: A gpt-based strategy for designing potential ligands targeting specific proteins. bioRxiv , 2023–06.
  22. Multi-modal molecule structure-text model for text-based retrieval and editing. arXiv preprint arXiv:2212.10789 .
  23. Pre-training molecular graph representation with 3d geometry. arXiv preprint arXiv:2110.07728 .
  24. Swin transformer: Hierarchical vision transformer using shifted windows, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022.
  25. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics 23, bbac409.
  26. OpenAI, 2023. GPT-4 technical report. CoRR abs/2303.08774.
  27. Molvec: Open source library for chemical structure recognition, in: Abstracts of Papers of the American Chemical Society, Amer Chemical Soc 1155 16TH ST, NW, WASHINGTON, DC 20036 USA.
  28. Learning transferable visual models from natural language supervision, in: International Conference on Machine Learning, PMLR. pp. 8748–8763.
  29. Language models are unsupervised multitask learners. OpenAI blog 1, 9.
  30. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21, 5485–5551.
  31. Decimer 1.0: deep learning for chemical image recognition using transformers. Journal of Cheminformatics 13, 1–16.
  32. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 .
  33. A generalist agent. arXiv preprint arXiv:2205.06175 .
  34. Counting on natural products for drug design. Nature Chemistry 8, 531–541.
  35. From show to tell: A survey on deep learning-based image captioning. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 539–559.
  36. A molecular multimodal foundation model associating molecule graphs with natural language. arXiv preprint arXiv:2209.05481 .
  37. Attention is all you need. Advances in neural information processing systems 30.
  38. Foundation transformers. arXiv preprint arXiv:2210.06423 .
  39. Multi-modal chemical information reconstruction from images and texts for exploring the near-drug space. Briefings in Bioinformatics 23, bbac461.
  40. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442 .
  41. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence 4, 279–287.
  42. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35, 24824–24837.
  43. Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of Chemical Information and Computer Sciences 28, 31–36.
  44. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671 .
  45. Moleculenet: a benchmark for molecular machine learning. Chemical Science 9, 513–530.
  46. Swinocsr: end-to-end optical chemical structure recognition using a swin transformer. Journal of Cheminformatics 14, 1–13.
  47. Diffusion models: A comprehensive survey of methods and applications. arXiv preprint arXiv:2209.00796 .
  48. A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals. Nature Communications 13, 862.
  49. Uni-mol: A universal 3d molecular representation learning framework, in: The Eleventh International Conference on Learning Representations.
  50. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 .
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Pengfei Liu (191 papers)
  2. Yiming Ren (22 papers)
  3. Zhixiang Ren (23 papers)
  4. Jun Tao (73 papers)
Citations (59)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets