Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Enhancing Adverse Drug Event Detection with Multimodal Dataset: Corpus Creation and Model Development (2405.15766v2)

Published 24 May 2024 in cs.AI, cs.CL, and cs.CV

Abstract: The mining of adverse drug events (ADEs) is pivotal in pharmacovigilance, enhancing patient safety by identifying potential risks associated with medications, facilitating early detection of adverse events, and guiding regulatory decision-making. Traditional ADE detection methods are reliable but slow, not easily adaptable to large-scale operations, and offer limited information. With the exponential increase in data sources like social media content, biomedical literature, and Electronic Medical Records (EMR), extracting relevant ADE-related information from these unstructured texts is imperative. Previous ADE mining studies have focused on text-based methodologies, overlooking visual cues, limiting contextual comprehension, and hindering accurate interpretation. To address this gap, we present a MultiModal Adverse Drug Event (MMADE) detection dataset, merging ADE-related textual information with visual aids. Additionally, we introduce a framework that leverages the capabilities of LLMs and VLMs for ADE detection by generating detailed descriptions of medical images depicting ADEs, aiding healthcare professionals in visually identifying adverse events. Using our MMADE dataset, we showcase the significance of integrating visual cues from images to enhance overall performance. This approach holds promise for patient safety, ADE awareness, and healthcare accessibility, paving the way for further exploration in personalized healthcare.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Extraction of adverse drug effects from clinical records. In MEDINFO 2010, pages 739–743. IOS Press.
  2. Identifying potential adverse effects using the web: A new approach to medical hypothesis generation. Journal of biomedical informatics, 44(6):989–996.
  3. Multi-task pharmacovigilance mining from social media posts. In Proceedings of the 2018 World Wide Web Conference, pages 117–126.
  4. Instructblip: Towards general-purpose vision-language models with instruction tuning.
  5. Biodex: Large-scale biomedical adverse drug event extraction for real-world pharmacovigilance. arXiv preprint arXiv:2305.13395.
  6. Clipsyntel: CLIP and LLM synergy for multimodal question summarization in healthcare. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 22031–22039. AAAI Press.
  7. Medsumm: A multimodal approach to summarizing code-mixed hindi-english clinical queries. In Advances in Information Retrieval - 46th European Conference on Information Retrieval, ECIR 2024, Glasgow, UK, March 24-28, 2024, Proceedings, Part V, volume 14612 of Lecture Notes in Computer Science, pages 106–120. Springer.
  8. Identification of adverse drug event assertive sentences in medical case reports. In First international workshop on knowledge discovery and health care management (KD-HCM), European conference on machine learning and principles and practice of knowledge discovery in databases (ECML PKDD), pages 16–27.
  9. An empirical evaluation of resources for the identification of diseases and adverse effects in biomedical literature. In 2nd Workshop on Building and evaluating resources for biomedical text mining (7th edition of the Language Resources and Evaluation Conference), pages 15–22.
  10. Extraction of potential adverse drug events from medical case reports. Journal of biomedical semantics, 3(1):1–10.
  11. Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports. Journal of biomedical informatics, 45(5):885–892.
  12. Percentage of patients with preventable adverse drug reactions and preventability of adverse drug reactions–a meta-analysis. PloS one, 7(3):e33236.
  13. Combing signals from spontaneous reports and electronic health records for detection of adverse drug reactions. Journal of the American Medical Informatics Association, 20(3):413–419.
  14. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778.
  15. Adverse drug reaction classification with deep neural networks. Coling.
  16. Cadec: A corpus of adverse drug event annotations. Journal of biomedical informatics, 55:73–81.
  17. Text and data mining techniques in adverse drug reaction detection. ACM Computing Surveys (CSUR), 47(4):1–39.
  18. Towards internet-age pharmacovigilance: extracting adverse drug reactions from user posts in health-related social networks. In Proceedings of the 2010 workshop on biomedical natural language processing, pages 117–125.
  19. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597.
  20. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888–12900. PMLR.
  21. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  22. Azadeh Nikfarjam and Graciela H Gonzalez. 2011. Pattern mining for extraction of mentions of adverse drug reactions from user comments. In AMIA annual symposium proceedings, volume 2011, page 1019. American Medical Informatics Association.
  23. Pharmacovigilance from social media: mining adverse drug reaction mentions using sequence labeling with word embedding cluster features. Journal of the American Medical Informatics Association, 22(3):671–681.
  24. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  25. Unveiling hallucination in text, image, video, and audio foundation models: A comprehensive survey.
  26. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927.
  27. Abeed Sarker and Graciela Gonzalez. 2015. Portable automatic text classification for adverse drug reaction detection via multi-corpus training. Journal of biomedical informatics, 53:196–207.
  28. Social media mining shared task workshop. In Biocomputing 2016: Proceedings of the Pacific Symposium, pages 581–592. World Scientific.
  29. Re-evaluating word mover’s distance. In International Conference on Machine Learning, pages 19231–19249. PMLR.
  30. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
  31. Clinical and economic burden of adverse drug reactions. Journal of Pharmacology and Pharmacotherapeutics, 4(1_suppl):S73–S77.
  32. Xraygpt: Chest radiographs summarization using medical vision-language models. arXiv preprint arXiv:2306.07971.
  33. Combination of deep recurrent neural networks and conditional random fields for extracting adverse drug reactions from user reviews. Journal of healthcare engineering, 2017.
  34. Understanding interobserver agreement: the kappa statistic. Fam med, 37(5):360–363.
  35. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100.
  36. Active computerized pharmacovigilance using natural language processing, statistics, and electronic health records: a feasibility study. Journal of the American Medical Informatics Association, 16(3):328–337.
  37. Feature selection for entity extraction from multiple biomedical corpora: A pso-based approach. Soft Computing, 22(20):6881–6904.
  38. Multi-task learning framework for mining crowd intelligence towards clinical treatment.
  39. Relation extraction from biomedical and clinical text: Unified multitask learning framework. IEEE/ACM transactions on computational biology and bioinformatics, 19(2):1105–1116.
  40. Vision-language models for vision tasks: A survey.
  41. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.
  42. An ensemble method for binary classification of adverse drug reactions from social media. In Proceedings of the Social Media Mining Shared Task Workshop at the Pacific Symposium on Biocomputing, volume 1.
  43. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. arXiv preprint arXiv:1909.02622.
  44. Juexiao Zhou and Xin Gao. 2023. Skingpt: A dermatology diagnostic system with vision large language model. arXiv preprint arXiv:2304.10691.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Pranab Sahoo (5 papers)
  2. Ayush Kumar Singh (2 papers)
  3. Sriparna Saha (48 papers)
  4. Aman Chadha (109 papers)
  5. Samrat Mondal (11 papers)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets