Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
60 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
8 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Exploring the Comprehension of ChatGPT in Traditional Chinese Medicine Knowledge (2403.09164v1)

Published 14 Mar 2024 in cs.CL and stat.AP

Abstract: No previous work has studied the performance of LLMs in the context of Traditional Chinese Medicine (TCM), an essential and distinct branch of medical knowledge with a rich history. To bridge this gap, we present a TCM question dataset named TCM-QA, which comprises three question types: single choice, multiple choice, and true or false, to examine the LLM's capacity for knowledge recall and comprehensive reasoning within the TCM domain. In our study, we evaluate two settings of the LLM, zero-shot and few-shot settings, while concurrently discussing the differences between English and Chinese prompts. Our results indicate that ChatGPT performs best in true or false questions, achieving the highest precision of 0.688 while scoring the lowest precision is 0.241 in multiple-choice questions. Furthermore, we observed that Chinese prompts outperformed English prompts in our evaluations. Additionally, we assess the quality of explanations generated by ChatGPT and their potential contribution to TCM knowledge comprehension. This paper offers valuable insights into the applicability of LLMs in specialized domains and paves the way for future research in leveraging these powerful models to advance TCM.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (32)
  1. J. Tan, H. Westermann, and K. Benyekhlef, “Chatgpt as an artificial lawyer,” Artificial Intelligence for Access to Justice (AI4AJ 2023), 2023.
  2. M. Cascella, J. Montomoli, V. Bellini, and E. Bignami, “Evaluating the feasibility of chatgpt in healthcare: an analysis of multiple clinical and research scenarios,” Journal of Medical Systems, vol. 47, no. 1, p. 33, 2023.
  3. V. Taecharungroj, ““what can chatgpt do?” analyzing early reactions to the innovative ai chatbot on twitter,” Big Data and Cognitive Computing, vol. 7, no. 1, p. 35, 2023.
  4. T. Dave, S. A. Athaluri, and S. Singh, “Chatgpt in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations,” Frontiers in Artificial Intelligence, vol. 6, p. 1169595, 2023.
  5. S. Gilbert, H. Harvey, T. Melvin, E. Vollebregt, and P. Wicks, “Large language model ai chatbots require approval as medical devices,” Nature Medicine, pp. 1–3, 2023.
  6. B. Gordijn and H. t. Have, “Chatgpt: evolution or revolution?” Medicine, Health Care and Philosophy, vol. 26, no. 1, pp. 1–2, 2023.
  7. A. J. Thirunavukarasu, D. S. J. Ting, K. Elangovan, L. Gutierrez, T. F. Tan, and D. S. W. Ting, “Large language models in medicine,” Nature Medicine, pp. 1–11, 2023.
  8. S. Kulkarni, “11. future of technology chatgpt: Optimizing language models for dialogue.” ICP Monogram on Digital Technology in Clinical Medicine, p. 62.
  9. A. Gilson, C. W. Safranek, T. Huang, V. Socrates, L. Chi, R. A. Taylor, D. Chartash et al., “How does chatgpt perform on the united states medical licensing examination? the implications of large language models for medical education and knowledge assessment,” JMIR Medical Education, vol. 9, no. 1, p. e45312, 2023.
  10. H. Nori, N. King, S. M. McKinney, D. Carignan, and E. Horvitz, “Capabilities of gpt-4 on medical challenge problems,” arXiv preprint arXiv:2303.13375, 2023.
  11. S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer, “Rethinking the role of demonstrations: What makes in-context learning work?” arXiv preprint arXiv:2202.12837, 2022.
  12. P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” ACM Computing Surveys, vol. 55, no. 9, pp. 1–35, 2023.
  13. K. Yang, S. Ji, T. Zhang, Q. Xie, and S. Ananiadou, “On the evaluations of chatgpt and emotion-enhanced prompting for mental health analysis,” arXiv preprint arXiv:2304.03347, 2023.
  14. Y. Yang, M. S. Islam, J. Wang, Y. Li, and X. Chen, “Traditional chinese medicine in the treatment of patients infected with 2019-new coronavirus (sars-cov-2): a review and perspective,” International journal of biological sciences, vol. 16, no. 10, p. 1708, 2020.
  15. H. Fu, Z. Xu, X.-l. Zhang, and G.-q. Zheng, “Kaixinsan, a well-known chinese herbal prescription, for alzheimer’s disease and depression: a preclinical systematic review,” Frontiers in Neuroscience, vol. 13, p. 1421, 2020.
  16. C.-L. Lu, X. Li, H.-M. Zhou, C. Zhang, Y.-Y. Yang, R.-L. Feng, C.-J. Long, F.-Y. Deng, J.-C. Li, Z.-M. Cao et al., “Randomised controlled trials of traditional chinese medicine in cancer care published in chinese: An overview,” The Lancet, vol. 394, p. S26, 2019.
  17. D. Roosan, M. Samore, M. Jones, Y. Livnat, and J. Clutter, “Big-data based decision-support systems to improve clinicians’ cognition,” in 2016 IEEE International Conference on Healthcare Informatics (ICHI).   IEEE, 2016, pp. 285–288.
  18. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “Training language models to follow instructions with human feedback,” Advances in Neural Information Processing Systems, vol. 35, pp. 27 730–27 744, 2022.
  19. R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill et al., “On the opportunities and risks of foundation models,” arXiv preprint arXiv:2108.07258, 2021.
  20. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.
  21. T. H. Kung, M. Cheatham, A. Medenilla, C. Sillos, L. De Leon, C. Elepaño, M. Madriaga, R. Aggabao, G. Diaz-Candido, J. Maningo et al., “Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models,” PLoS digital health, vol. 2, no. 2, p. e0000198, 2023.
  22. C. Ahn, “Exploring chatgpt for information of cardiopulmonary resuscitation,” Resuscitation, vol. 185, 2023.
  23. T. Pettinato Oltz, “Chatgpt, professor of law,” Professor of Law (February 4, 2023), 2023.
  24. J. H. Choi, K. E. Hickman, A. Monahan, and D. Schwarcz, “Chatgpt goes to law school,” Available at SSRN, 2023.
  25. X. Yang, A. Chen, N. PourNejatian, H. C. Shin, K. E. Smith, C. Parisien, C. Compas, C. Martin, A. B. Costa, M. G. Flores et al., “A large language model for electronic health records,” NPJ Digital Medicine, vol. 5, no. 1, p. 194, 2022.
  26. Y. Gu, R. Tinn, H. Cheng, M. Lucas, N. Usuyama, X. Liu, T. Naumann, J. Gao, and H. Poon, “Domain-specific language model pretraining for biomedical natural language processing,” ACM Transactions on Computing for Healthcare (HEALTH), vol. 3, no. 1, pp. 1–23, 2021.
  27. J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang, “Biobert: a pre-trained biomedical language representation model for biomedical text mining,” Bioinformatics, vol. 36, no. 4, pp. 1234–1240, 2020.
  28. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361, 2020.
  29. T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large language models are zero-shot reasoners,” Advances in neural information processing systems, vol. 35, pp. 22 199–22 213, 2022.
  30. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Information Processing Systems, vol. 35, pp. 24 824–24 837, 2022.
  31. X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” arXiv preprint arXiv:2203.11171, 2022.
  32. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders et al., “Webgpt: Browser-assisted question-answering with human feedback,” arXiv preprint arXiv:2112.09332, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Li Yizhen (1 paper)
  2. Huang Shaohan (1 paper)
  3. Qi Jiaxing (1 paper)
  4. Quan Lei (1 paper)
  5. Han Dongran (1 paper)
  6. Luan Zhongzhi (1 paper)