Leveraging ChatGPT in Pharmacovigilance Event Extraction: An Empirical Study (2402.15663v1)
Abstract: With the advent of LLMs, there has been growing interest in exploring their potential for medical applications. This research aims to investigate the ability of LLMs, specifically ChatGPT, in the context of pharmacovigilance event extraction, of which the main goal is to identify and extract adverse events or potential therapeutic events from textual medical sources. We conduct extensive experiments to assess the performance of ChatGPT in the pharmacovigilance event extraction task, employing various prompts and demonstration selection strategies. The findings demonstrate that while ChatGPT demonstrates reasonable performance with appropriate demonstration selection strategies, it still falls short compared to fully fine-tuned small models. Additionally, we explore the potential of leveraging ChatGPT for data augmentation. However, our investigation reveals that the inclusion of synthesized data into fine-tuning may lead to a decrease in performance, possibly attributed to noise in the ChatGPT-generated labels. To mitigate this, we explore different filtering strategies and find that, with the proper approach, more stable performance can be achieved, although constant improvement remains elusive.
- Large language models are few-shot clinical information extractors. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1998–2022.
- Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416.
- Mttlade: A multi-task transfer learning-based method for adverse drug events extraction. Information Processing & Management, 58(3):102473.
- Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports. Journal of biomedical informatics, 45(5):885–892.
- Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models. PLOS Digital Health, 2:e0000198.
- Few-shot learning with multilingual generative language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9019–9052.
- Wanli: Worker and ai collaboration for natural language inference dataset creation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6826–6847.
- NapSS: Paragraph-level medical text simplification via narrative prompting and sentence-matching summarization. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1079–1091, Dubrovnik, Croatia. Association for Computational Linguistics.
- Unified structure generation for universal information extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5755–5772, Dublin, Ireland. Association for Computational Linguistics.
- OpenAI. 2022. Chatgpt.
- Boosting low-resource biomedical QA via entity-aware masking strategies. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1977–1985, Online. Association for Computational Linguistics.
- Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.
- PHEE: A dataset for pharmacovigilance event extraction from text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5571–5587, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Event temporal relation extraction with Bayesian translational model. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1125–1138, Dubrovnik, Croatia. Association for Computational Linguistics.
- Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
- Improvements to bm25 and language models examined. In Proceedings of the 2014 Australasian Document Computing Symposium, pages 58–65.
- Llm-powered data augmentation for enhanced crosslingual performance. arXiv preprint arXiv:2305.14288.
- World Health Organization. 2004. Pharmacovigilance: ensuring the safe use of medicines. Technical report, World Health Organization.
- Towards transforming fda adverse event narratives into actionable structured data for improved pharmacovigilance. In Proceedings of the Symposium on Applied Computing, pages 777–782.
- Disentangled learning of stance and aspect topics for vaccine attitude detection in social media. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1566–1580, Seattle, United States. Association for Computational Linguistics.
- Disentangling aspect and stance via a Siamese autoencoder for aspect clustering of vaccination opinions. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1827–1842, Toronto, Canada. Association for Computational Linguistics.
- Zhaoyue Sun (6 papers)
- Gabriele Pergola (26 papers)
- Byron C. Wallace (82 papers)
- Yulan He (113 papers)