An LLM-Powered Clinical Calculator Chatbot Backed by Verifiable Clinical Calculators and their Metadata

Abstract: Clinical calculators are widely used, and LLMs make it possible to engage them using natural language. We demonstrate a purpose-built chatbot that leverages software implementations of verifiable clinical calculators via LLM tools and metadata about these calculators via retrieval augmented generation (RAG). We compare the chatbot's response accuracy to an unassisted off-the-shelf LLM on four natural language conversation workloads. Our chatbot achieves 100% accuracy on queries interrogating calculator metadata content and shows a significant increase in clinical calculation accuracy vs. the off-the-shelf LLM when prompted with complete sentences (86.4% vs. 61.8%) or with medical shorthand (79.2% vs. 62.0%). It eliminates calculation errors when prompted with complete sentences (0% vs. 16.8%) and greatly reduces them when prompted with medical shorthand (2.4% vs. 18%). While our chatbot is not ready for clinical use, these results show progress in minimizing incorrect calculation results.