Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Assessing the Usability of GutGPT: A Simulation Study of an AI Clinical Decision Support System for Gastrointestinal Bleeding Risk (2312.10072v1)

Published 6 Dec 2023 in cs.HC, cs.AI, cs.LG, and stat.AP

Abstract: Applications of LLMs like ChatGPT have potential to enhance clinical decision support through conversational interfaces. However, challenges of human-algorithmic interaction and clinician trust are poorly understood. GutGPT, a LLM for gastrointestinal (GI) bleeding risk prediction and management guidance, was deployed in clinical simulation scenarios alongside the electronic health record (EHR) with emergency medicine physicians, internal medicine physicians, and medical students to evaluate its effect on physician acceptance and trust in AI clinical decision support systems (AI-CDSS). GutGPT provides risk predictions from a validated machine learning model and evidence-based answers by querying extracted clinical guidelines. Participants were randomized to GutGPT and an interactive dashboard, or the interactive dashboard and a search engine. Surveys and educational assessments taken before and after measured technology acceptance and content mastery. Preliminary results showed mixed effects on acceptance after using GutGPT compared to the dashboard or search engine but appeared to improve content mastery based on simulation performance. Overall, this study demonstrates LLMs like GutGPT could enhance effective AI-CDSS if implemented optimally and paired with interactive interfaces.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (23)
  1. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785–794, 2016.
  2. Human trust in artificial intelligence: Review of empirical research. Academy of Management Annals, 14(2):627–660, 2020.
  3. Applied artificial intelligence and trust—the case of autonomous vehicles and medical assistance devices. Technological Forecasting and Social Change, 105:105–120, 2016.
  4. Harold Hotelling. Relations between two sets of variates. In Breakthroughs in statistics: methodology and distribution, pages 162–190. Springer, 1992.
  5. Provider trust and perceived usefulness of machine learning risk stratification tool for acute upper gastrointestinal bleeding using the technology acceptance model: a pilot study. Gastroenterology, 164(6S):S1168–S1169.
  6. Technology-enhanced simulation in emergency medicine: a systematic review and meta-analysis. Academic Emergency Medicine, 20(2):117–127, 2013.
  7. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  8. Rene F Kizilcec. How much information? effects of transparency on trust in an algorithmic interface. CHI ’16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pages 2390–2395, 2016.
  9. Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models. PLoS digital health, 2(2):e0000198, 2023.
  10. Acg clinical guideline: upper gastrointestinal and ulcer bleeding. Official journal of the American College of Gastroenterology— ACG, 116(5):899–917, 2021.
  11. An integrative review on the acceptance of artificial intelligence among healthcare professionals in hospitals. NPJ digit Med., 6(1).
  12. Artificial intelligence in undergraduate medical education: a scoping review. Academic Medicine, 96(11S):S62–S70, 2021.
  13. Christoph Molnar. Interpretable machine learning. Lulu. com, 2020.
  14. Karl Pearson. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science, 2(11):559–572, 1901.
  15. Kathleen R Rosen. The history of medical simulation. Journal of critical care, 23(2):157–166, 2008.
  16. Learning representations by back-propagating errors. nature, 323(6088):533–536, 1986.
  17. Validation of a machine learning model that outperforms clinical risk scoring systems for upper gastrointestinal bleeding. Gastroenterology, 158(1):160–167, 2020.
  18. Making sense of cronbach’s alpha. International journal of medical education, 2:53–55, 2011.
  19. Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267–288, 1996.
  20. User acceptance of information technology: Toward a unified view. MIS quarterly, pages 425–478, 2003.
  21. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523):1228–1242, 2018.
  22. Ethical considerations of using chatgpt in health care. J Med Internet Res., 11(25).
  23. Jeremy J Webb. Proof of concept: Using chatgpt to teach emergency physicians how to break bad news. Cureus, 15(5), 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (13)
  1. Colleen Chan (3 papers)
  2. Kisung You (16 papers)
  3. Sunny Chung (2 papers)
  4. Mauro Giuffrè (3 papers)
  5. Theo Saarinen (4 papers)
  6. Niroop Rajashekar (3 papers)
  7. Yuan Pu (11 papers)
  8. Yeo Eun Shin (1 paper)
  9. Loren Laine (4 papers)
  10. Ambrose Wong (1 paper)
  11. René Kizilcec (6 papers)
  12. Jasjeet Sekhon (10 papers)
  13. Dennis Shung (13 papers)
Citations (3)
View on Semantic Scholar