Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

An Evaluation of Large Language Models in Bioinformatics Research (2402.13714v1)

Published 21 Feb 2024 in q-bio.QM, cs.AI, and cs.LG

Abstract: LLMs such as ChatGPT have gained considerable interest across diverse research communities. Their notable ability for text completion and generation has inaugurated a novel paradigm for language-interfaced problem solving. However, the potential and efficacy of these models in bioinformatics remain incompletely explored. In this work, we study the performance LLMs on a wide spectrum of crucial bioinformatics tasks. These tasks include the identification of potential coding regions, extraction of named entities for genes and proteins, detection of antimicrobial and anti-cancer peptides, molecular optimization, and resolution of educational bioinformatics problems. Our findings indicate that, given appropriate prompts, LLMs like GPT variants can successfully handle most of these tasks. In addition, we provide a thorough analysis of their limitations in the context of complicated bioinformatics tasks. In conclusion, we believe that this work can provide new perspectives and motivate future research in the field of LLMs applications, AI for Science and bioinformatics.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (85)
  1. A. Birhane, A. Kasirzadeh, D. Leslie, and S. Wachter, “Science in the age of large language models,” Nature Reviews Physics, pp. 1–4, 2023.
  2. U. Katz, M. Geva, and J. Berant, “Inferring implicit relations in complex questions with language models,” in Findings of the Association for Computational Linguistics: EMNLP 2022, 2022, pp. 2548–2566.
  3. X. L. Li, A. Kuncoro, J. Hoffmann, C. de Masson d’Autume, P. Blunsom, and A. Nematzadeh, “A systematic investigation of commonsense knowledge in large language models,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2022, pp. 11 838–11 855.
  4. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “Training language models to follow instructions with human feedback,” in Proceedings of the Conference on Neural Information Processing Systems, 2022, pp. 27 730–27 744.
  5. F. Xu, Q. Lin, J. Han, T. Zhao, J. Liu, and E. Cambria, “Are large language models really good logical reasoners? a comprehensive evaluation from deductive, inductive and abductive views,” arXiv preprint arXiv:2306.09841, 2023.
  6. B. Pang, L. Lee, and S. Vaithyanathan, “Thumbs up? sentiment classification using machine learning techniques,” arXiv preprint cs/0205070, 2002.
  7. M. Marrero, J. Urbano, S. Sánchez-Cuadrado, J. Morato, and J. M. Gómez-Berbís, “Named entity recognition: fallacies, challenges and opportunities,” Computer Standards & Interfaces, vol. 35, no. 5, pp. 482–489, 2013.
  8. K. Lee, D. Palsetia, R. Narayanan, M. M. A. Patwary, A. Agrawal, and A. Choudhary, “Twitter trending topic classification,” in Proceedings of the International Conference on Data Mining Workshops, 2011, pp. 251–258.
  9. Y. Liu, T. Han, S. Ma, J. Zhang, Y. Yang, J. Tian, H. He, A. Li, M. He, Z. Liu et al., “Summary of chatgpt/gpt-4 research and perspective towards the future of large language models,” arXiv preprint arXiv:2304.01852, 2023.
  10. T. Sun, Z. He, H. Qian, Y. Zhou, X.-J. Huang, and X. Qiu, “Bbtv2: towards a gradient-free future with large language models,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2022, pp. 3916–3930.
  11. T. Zhang, F. Ladhak, E. Durmus, P. Liang, K. McKeown, and T. B. Hashimoto, “Benchmarking large language models for news summarization,” arXiv preprint arXiv:2301.13848, 2023.
  12. I. Beltagy, A. Cohan, R. Logan IV, S. Min, and S. Singh, “Zero-and few-shot nlp with pretrained language models,” in Proceedings of the Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, 2022, pp. 32–37.
  13. Y. Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung et al., “A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity,” arXiv preprint arXiv:2302.04023, 2023.
  14. W. Jiao, W. Wang, J.-t. Huang, X. Wang, and Z. Tu, “Is chatgpt a good translator? a preliminary study,” arXiv preprint arXiv:2301.08745, 2023.
  15. Y. Tan, D. Min, Y. Li, W. Li, N. Hu, Y. Chen, and G. Qi, “Evaluation of chatgpt as a question answering system for answering complex questions,” arXiv preprint arXiv:2303.07992, 2023.
  16. T. H. Kung, M. Cheatham, A. Medenilla, C. Sillos, L. De Leon, C. Elepaño, M. Madriaga, R. Aggabao, G. Diaz-Candido, J. Maningo et al., “Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models,” PLoS Digital Health, vol. 2, no. 2, p. e0000198, 2023.
  17. S. B. Patel and K. Lam, “Chatgpt: the future of discharge summaries?” The Lancet Digital Health, vol. 5, no. 3, pp. e107–e108, 2023.
  18. Q. Lu, D. Dou, and T. Nguyen, “Clinicalt5: A generative language model for clinical text,” in Findings of the Association for Computational Linguistics: EMNLP 2022, 2022, pp. 5436–5443.
  19. J. Otmakhova, K. Verspoor, T. Baldwin, A. J. Yepes, and J. H. Lau, “M3: Multi-level dataset for multi-document summarisation of medical studies,” in Findings of the Association for Computational Linguistics: EMNLP 2022, 2022, pp. 3887–3901.
  20. Z. Lin, H. Akin, R. Rao, B. Hie, Z. Zhu, W. Lu, N. Smetanin, R. Verkuil, O. Kabeli, Y. Shmueli et al., “Evolutionary-scale prediction of atomic-level protein structure with a language model,” Science, vol. 379, no. 6637, pp. 1123–1130, 2023.
  21. A. Elnaggar, M. Heinzinger, C. Dallago, G. Rehawi, Y. Wang, L. Jones, T. Gibbs, T. Feher, C. Angerer, M. Steinegger et al., “Prottrans: Toward understanding the language of life through self-supervised learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 10, pp. 7112–7127, 2021.
  22. H. Lee, S. Lee, I. Lee, and H. Nam, “Amp-bert: Prediction of antimicrobial peptide function based on a bert model,” Protein Science, vol. 32, no. 1, p. e4529, 2023.
  23. I. Jahan, M. T. R. Laskar, C. Peng, and J. Huang, “Evaluation of chatgpt on biomedical tasks: A zero-shot comparison with fine-tuned generative transformers,” arXiv preprint arXiv:2306.04504, 2023.
  24. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023.
  25. Ö. AYDIN, “Google bard generated literature review: metaverse,” Journal of AI, vol. 7, no. 1, pp. 1–14, 2023.
  26. M. Wysocka, O. Wysocki, M. Delmas, V. Mutel, and A. Freitas, “Large language models, scientific knowledge and factuality: A systematic analysis in antibiotic discovery,” arXiv preprint arXiv:2305.17819, 2023.
  27. Y. Liu, S. Feng, D. Wang, Y. Zhang, and H. Schütze, “Evaluate what you can’t evaluate: Unassessable generated responses quality,” arXiv preprint arXiv:2305.14658, 2023.
  28. S. S. Biswas, “Role of chat gpt in public health,” Annals of Biomedical Engineering, vol. 51, no. 5, pp. 868–869, 2023.
  29. C. Shyr, Y. Hu, P. A. Harris, and H. Xu, “Identifying and extracting rare disease phenotypes with large language models,” arXiv preprint arXiv:2306.12656, 2023.
  30. X. Li, Y. Zhang, and E. C. Malthouse, “A preliminary study of chatgpt on news recommendation: Personalization, provider fairness, fake news,” arXiv preprint arXiv:2306.10702, 2023.
  31. D. Pu and V. Demberg, “Chatgpt vs human-authored text: Insights into controllable text summarization and sentence style transfer,” arXiv preprint arXiv:2306.07799, 2023.
  32. P. Cramer, “Alphafold2 and the future of structural biology,” Nature Structural & Molecular Biology, vol. 28, no. 9, pp. 704–705, 2021.
  33. J. Deng, Z. Yang, I. Ojima, D. Samaras, and F. Wang, “Artificial intelligence in drug discovery: applications and techniques,” Briefings in Bioinformatics, vol. 23, no. 1, 2022.
  34. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
  35. B. Fabian, T. Edlich, H. Gaspar, M. Segler, J. Meyers, M. Fiscato, and M. Ahmed, “Molecular representation learning with language models and domain-relevant auxiliary tasks,” arXiv preprint arXiv:2011.13230, 2020.
  36. E. Shue, L. Liu, B. Li, Z. Feng, X. Li, and G. Hu, “Empowering beginners in bioinformatics with chatgpt,” bioRxiv, pp. 2023–03, 2023.
  37. S. R. Piccolo, P. Denny, A. Luxton-Reilly, S. Payne, and P. G. Ridge, “Many bioinformatics programming tasks can be automated with chatgpt,” arXiv preprint arXiv:2303.13528, 2023.
  38. I. Jahan, M. T. R. Laskar, C. Peng, and J. Huang, “A comprehensive evaluation of large language models on benchmark biomedical text processing tasks,” 2023.
  39. R. Taylor, M. Kardas, G. Cucurull, T. Scialom, A. Hartshorn, E. Saravia, A. Poulton, V. Kerkez, and R. Stojnic, “Galactica: A large language model for science,” 2022.
  40. R. Luo, L. Sun, Y. Xia, T. Qin, S. Zhang, H. Poon, and T.-Y. Liu, “Biogpt: generative pre-trained transformer for biomedical text generation and mining,” Briefings in Bioinformatics, vol. 23, no. 6, Sep. 2022. [Online]. Available: http://dx.doi.org/10.1093/bib/bbac409
  41. D. B. Lubahn, T. R. Brown, J. A. Simental, H. N. Higgs, C. J. Migeon, E. M. Wilson, and F. S. French, “Sequence of the intron/exon junctions of the coding region of the human androgen receptor gene and identification of a point mutation in a family with complete androgen insensitivity.” Proceedings of the National Academy of Sciences, vol. 86, no. 23, pp. 9534–9538, 1989.
  42. W. Zhu, A. Lomsadze, and M. Borodovsky, “Ab initio gene identification in metagenomic sequences ,” Nucleic Acids Research, vol. 38, no. 12, pp. e132–e132, 2010.
  43. J. H. Badger and G. J. Olsen, “Critica: coding region identification tool invoking comparative analysis.” Molecular Biology and Evolution, vol. 16, no. 4, pp. 512–524, 1999.
  44. R. Wise, T. Hart, O. Cars, M. Streulens, R. Helmuth, P. Huovinen, and M. Sprenger, “Antimicrobial resistance,” pp. 609–610, 1998.
  45. A. A. Bahar and D. Ren, “Antimicrobial peptides,” Pharmaceuticals, vol. 6, no. 12, pp. 1543–1575, 2013.
  46. M. Zasloff, “Antimicrobial peptides of multicellular organisms,” Nature, vol. 415, no. 6870, pp. 389–395, 2002.
  47. B. Liu, L. Ezeogu, L. Zellmer, B. Yu, N. Xu, and D. Joshua Liao, “Protecting the normal in order to better kill the cancer,” Cancer Medicine, vol. 4, no. 9, pp. 1394–1403, 2015.
  48. J. Li, S. Tan, X. Chen, C.-Y. Zhang, and Y. Zhang, “Peptide aptamers with biological and therapeutic applications,” Current Medicinal Chemistry, vol. 18, no. 27, pp. 4215–4222, 2011.
  49. A. Tyagi, A. Tuknait, P. Anand, S. Gupta, M. Sharma, D. Mathur, A. Joshi, S. Singh, A. Gautam, and G. P. Raghava, “Cancerppd: a database of anticancer peptides and proteins,” Nucleic Acids Research, vol. 43, no. D1, pp. D837–D843, 2015.
  50. W. Chiangjong, S. Chutipongtanate, and S. Hongeng, “Anticancer peptide: Physicochemical property, functional aspect and trend in clinical application,” International Journal of Oncology, vol. 57, no. 3, pp. 678–696, 2020.
  51. Q. Li, W. Zhou, D. Wang, S. Wang, and Q. Li, “Prediction of anticancer peptides using a low-dimensional feature model,” Frontiers in Bioengineering and Biotechnology, vol. 8, p. 892, 2020.
  52. M. L. Verdonk and M. J. Hartshorn, “Structure-guided fragment screening for lead discovery.” Current Opinion in Drug Discovery & Development, vol. 7, no. 4, pp. 404–410, 2004.
  53. B. Ouyang, J. Wang, T. He, C. J. Bartel, H. Huo, Y. Wang, V. Lacivita, H. Kim, and G. Ceder, “Synthetic accessibility and stability rules of nasicons,” Nature Communications, vol. 12, no. 1, p. 5752, 2021.
  54. G. R. Bickerton, G. V. Paolini, J. Besnard, S. Muresan, and A. L. Hopkins, “Quantifying the chemical beauty of drugs,” Nature Chemistry, vol. 4, no. 2, pp. 90–98, 2012.
  55. E. A. Bruford, B. Braschi, P. Denny, T. E. Jones, R. L. Seal, and S. Tweedie, “Guidelines for human gene nomenclature,” Nature Genetics, vol. 52, no. 8, pp. 754–758, 2020.
  56. K. Fundel and R. Zimmer, “Gene and protein nomenclature in public databases,” BMC Bioinformatics, vol. 7, no. 1, pp. 1–13, 2006.
  57. T. C. Rindflesch, L. Tanabe, J. N. Weinstein, and L. Hunter, “Edgar: extraction of drugs, genes and relations from the biomedical literature,” in Biocomputing 2000.   World Scientific, 1999, pp. 517–528.
  58. C. Blaschke, L. Hirschman, and A. Valencia, “Information extraction in molecular biology,” Briefings in Bioinformatics, vol. 3, no. 2, pp. 154–165, 2002.
  59. A. Yeh, A. Morgan, M. Colosimo, and L. Hirschman, “Biocreative task 1a: gene mention finding evaluation,” BMC Bioinformatics, vol. 6, pp. 1–10, 2005.
  60. M. F. Wangler, S. Yamamoto, and H. J. Bellen, “Fruit flies in biomedical research,” Genetics, vol. 199, no. 3, pp. 639–653, 2015.
  61. S. J. Goebel, G. P. Johnson, M. E. Perkus, S. W. Davis, J. P. Winslow, and E. Paoletti, “The complete dna sequence of vaccinia virus,” Virology, vol. 179, no. 1, pp. 247–266, 1990.
  62. Z. Chen, M. R. Min, S. Parthasarathy, and X. Ning, “A deep generative model for molecule optimization via one fragment modification,” Nature Machine Intelligence, vol. 3, no. 12, pp. 1040–1049, 2021.
  63. T. Greenhalgh, “How to read a paper: the medline database,” Bmj, vol. 315, no. 7101, pp. 180–183, 1997.
  64. M. Furuno, T. Kasukawa, R. Saito, J. Adachi, H. Suzuki, R. Baldarelli, Y. Hayashizaki, and Y. Okazaki, “Cds annotation in full-length cdna sequence,” Genome Research, vol. 13, no. 6b, pp. 1478–1487, 2003.
  65. T. Chen, T. He, M. Benesty, V. Khotilovich, Y. Tang, H. Cho, K. Chen, R. Mitchell, I. Cano, T. Zhou et al., “Xgboost: extreme gradient boosting,” R package version 0.4-2, vol. 1, no. 4, pp. 1–4, 2015.
  66. A. M. Kibriya, E. Frank, B. Pfahringer, and G. Holmes, “Multinomial naive bayes for text categorization revisited,” in AI 2004: Advances in Artificial Intelligence: 17th Australian Joint Conference on Artificial Intelligence, Cairns, Australia, December 4-6, 2004. Proceedings 17, 2005, pp. 488–499.
  67. V. Jakkula, “Tutorial on support vector machine (svm),” School of EECS, Washington State University, vol. 37, no. 2.5, p. 3, 2006.
  68. G. Guo, H. Wang, D. Bell, Y. Bi, and K. Greer, “Knn model-based approach in classification,” in On The Move to Meaningful Internet Systems 2003: CoopIS, DOA, and ODBASE: OTM Confederated International Conferences, CoopIS, DOA, and ODBASE 2003, Catania, Sicily, Italy, November 3-7, 2003. Proceedings, 2003, pp. 986–996.
  69. M. Maalouf, “Logistic regression in data analysis: an overview,” International Journal of Data Analysis Techniques and Strategies, vol. 3, no. 3, pp. 281–299, 2011.
  70. A. Pinkus, “Approximation theory of the mlp model in neural networks,” Acta Numerica, vol. 8, pp. 143–195, 1999.
  71. G. Biau and E. Scornet, “A random forest guided tour,” TEST, vol. 25, pp. 197–227, 2016.
  72. H. Saigo, S. Nowozin, T. Kadowaki, T. Kudo, and K. Tsuda, “gboost: a mathematical programming approach to graph classification and regression,” Machine Learning, vol. 75, pp. 69–89, 2009.
  73. S. A. Wildman and G. M. Crippen, “Prediction of physicochemical parameters by atomic contributions,” Journal of Chemical Information and Computer Sciences, vol. 39, no. 5, pp. 868–873, 1999.
  74. P. Ertl and A. Schuffenhauer, “Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions,” Journal of Cheminformatics, vol. 1, pp. 1–11, 2009.
  75. W. Jin, K. Yang, R. Barzilay, and T. Jaakkola, “Learning multimodal graph-to-graph translation for molecular optimization,” arXiv preprint arXiv:1812.01070, 2018.
  76. T. Sterling and J. J. Irwin, “Zinc 15–ligand discovery for everyone,” Journal of Chemical Information and Modeling, vol. 55, no. 11, pp. 2324–2337, 2015.
  77. H. Cho and H. Lee, “Biomedical named entity recognition using deep neural networks with contextual information,” BMC bioinformatics, vol. 20, pp. 1–11, 2019.
  78. H. Zhao, Y. Yang, Q. Zhang, and L. Si, “Improve neural entity recognition via multi-task data selection and constrained decoding,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), 2018, pp. 346–351.
  79. J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang, “Biobert: a pre-trained biomedical language representation model for biomedical text mining,” Bioinformatics, vol. 36, no. 4, pp. 1234–1240, 2020.
  80. T. Bansal, R. Jha, and A. McCallum, “Learning to few-shot learn across diverse natural language classification tasks,” 2020.
  81. T. Zhang, C. Xia, P. S. Yu, Z. Liu, and S. Zhao, “Pdaln: Progressive domain adaptation over a pre-trained model for low-resource cross-domain named entity recognition,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2021.
  82. R. A. Baeza-Yates, “Algorithms for string searching,” in ACM SIGIR Forum, vol. 23, no. 3-4, 1989, pp. 34–58.
  83. L. Lovász and H. J. Prömel, “Combinatorics,” Oberwolfach Reports, vol. 1, no. 1, pp. 5–110, 2004.
  84. R. C. Edgar and S. Batzoglou, “Multiple sequence alignment,” Current Opinion in Structural Biology, vol. 16, no. 3, pp. 368–373, 2006.
  85. K. G. Field, G. J. Olsen, D. J. Lane, S. J. Giovannoni, M. T. Ghiselin, E. C. Raff, N. R. Pace, and R. A. Raff, “Molecular phylogeny of the animal kingdom,” Science, vol. 239, no. 4841, pp. 748–753, 1988.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (9)
  1. Zhonghui Gu (2 papers)
  2. Fanhao Wang (2 papers)
  3. Yiparemu Abuduhaibaier (1 paper)
  4. Yanqiao Zhu (45 papers)
  5. Xinming Tu (2 papers)
  6. Xian-Sheng Hua (85 papers)
  7. Xiao Luo (112 papers)
  8. Yizhou Sun (149 papers)
  9. HengChuang Yin (2 papers)
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets