MedMCP-Calc: Clinical Calculator & Benchmark
- The paper presents MedMCP-Calc, a dual artifact combining an LLM-powered clinical calculator chatbot with a benchmark evaluating multi-step, realistic medical workflows.
- It employs JSON-LD metadata, tool manifests, and local Python execution to achieve verifiable, error-free numerical outputs in clinical scenarios.
- Empirical evaluations reveal significant performance improvements over off-the-shelf LLMs, with higher accuracy and reduced calculation errors across diverse medical domains.
Searching arXiv for MedMCP-Calc and related papers to ground the article. MedMCP-Calc denotes a recent line of research on medical calculator use by LLMs in which calculator selection, parameter acquisition, and numerical execution are treated as tool-grounded processes rather than as unsupported free-text generation. In the current literature, the name refers principally to two related but distinct artifacts: an LLM-powered clinical calculator chatbot backed by verifiable local calculator implementations and structured calculator metadata, and a benchmark for evaluating realistic medical calculator scenarios through Model Context Protocol (MCP) integration (Kumar et al., 21 Mar 2025, Zhu et al., 30 Jan 2026).
1. Scope, nomenclature, and conceptual focus
The 2025 MedMCP-Calc system is a purpose-built chatbot that leverages software implementations of verifiable clinical calculators via LLM tools and metadata about these calculators via retrieval augmented generation. The 2026 MedMCP-Calc benchmark evaluates LLMs in realistic medical calculator scenarios characterized by fuzzy task descriptions, structured EHR database interaction, external reference retrieval, and process-level evaluation (Kumar et al., 21 Mar 2025, Zhu et al., 30 Jan 2026).
| Artifact | Core purpose | Key mechanisms |
|---|---|---|
| MedMCP-Calc chatbot | Natural-language access to verifiable clinical calculators | RAG, JSON-LD metadata, function calling, local Python execution |
| MedMCP-Calc benchmark | Evaluation of realistic end-to-end calculator workflows | MCP servers, EHR querying, external retrieval, process-level scoring |
The common research premise is that medical calculator use in practice is not limited to single-step arithmetic. The benchmark paper states that real-world use is an adaptive, multi-stage process requiring proactive EHR data acquisition, scenario-dependent calculator selection, and multi-step computation, whereas current benchmarks focus only on static single-step calculations with explicit instructions (Zhu et al., 30 Jan 2026). The chatbot paper approaches the same problem from a narrower engineering angle by combining metadata retrieval, typed tool manifests, and local execution so that calculator outputs are verifiable rather than solely inferred by the model (Kumar et al., 21 Mar 2025).
A separate recipe in the supplied literature describes how bias-corrected estimators for censored Weibull regression can be slotted into a “MedMCP-Calc–style tool” for the MCP-Mod approach to time-to-event endpoints, with arguments such as method = c("MLE","BCE","FIRTH") and covCorrect = TRUE|FALSE (Diniz et al., 2023). This suggests a naming overlap between the MedMCP-Calc clinical-calculator work and an unrelated statistical workflow centered on Multiple Comparison Procedures with Modeling Techniques.
2. Chatbot architecture and verifiable calculator execution
The chatbot architecture consists of a natural-language front end, a retrieval pipeline, an auto-generated tool manifest, LLM orchestration, and local calculator execution. Users type clinical requests in full English or medical shorthand, and each new user query is appended to the conversation history. At runtime, the user query is embedded with OpenAI’s text-embedding-3-small model, and a Qdrant vector store retrieves the single most similar JSON-LD metadata object. The eleven calculators are exposed as LLM “tools” via OpenAI’s function-calling interface, with each tool definition auto-generated from the JSON-LD metadata. The LLM logic uses OpenAI gpt-4o: it ingests a fixed system prompt, conversation history, retrieved metadata, and the tool manifest; replies directly if no calculation is requested; or identifies the tool, requests missing parameters, and emits a structured “Calculator Call” JSON if a calculation is requested. The runtime validates the call via OpenAI’s Structured Outputs schema, executes the matching calculator locally in Python 3, and returns the numeric result to the LLM, which composes a final reply and clearly marks verifiable outputs with asterisks (Kumar et al., 21 Mar 2025).
MedMCP-Calc includes eleven calculators drawn from MDCalc’s top ten, with BMI and BSA counted separately. The implemented set comprises Cockcroft–Gault Creatinine Clearance, CKD-EPI GFR Equations (2021), MDRD GFR Equation, CHA₂DS₂-VASc Stroke Risk Score, ASCVD 2013 Risk Calculator, Body Mass Index, Mosteller Body Surface Area, Albumin-Corrected Calcium, Wells’ Criteria for Pulmonary Embolism, and NIH Stroke Scale. Each calculator is implemented in Python per its original publications and unit-tested against reference cases. Each calculator is encapsulated in a Knowledge Object with versioning and unit tests against published example cases, and the code resides in the local environment and is never exposed to the LLM provider (Kumar et al., 21 Mar 2025).
The system description provides representative formulas for several calculators. For example, BMI is given as
and Mosteller BSA is given as
These formulas are not used as free-form prompts to the model; instead, MedMCP-Calc operationalizes them through local, verifiable implementations (Kumar et al., 21 Mar 2025).
3. Metadata design and retrieval-augmented control
A central technical feature of the chatbot is its metadata model. Calculator descriptions are encoded as JSON-LD linked data following the Knowledge Object paradigm. Core fields include id, name, shortDescription, longDescription, version, authors, publicationReferences, structured inputs, structured outputs, evidenceLevel, intendedUse, and contraindications. Each input object includes name, dataType, units, description, a required flag, range or enumerated constraints, and optional defaults. Outputs are likewise typed and documented (Kumar et al., 21 Mar 2025).
The retrieval system embeds flattened or minified JSON-LD in Qdrant and stores vector identifiers linked to metadata. At query time, the full JSON-LD is reloaded both for context injection and for construction of tool definitions. The prompt is composed from system instructions, conversation history, the retrieved metadata block, and the derived tool manifest. The paper states that surfacing the exact parameter list—names, units, ranges, and required flags—allows the LLM to choose the correct calculator and request or validate each parameter before tool invocation. OpenAI’s structured-outputs schema further enforces that only permitted parameters and types appear in the calculator call (Kumar et al., 21 Mar 2025).
The examples in the system paper illustrate the control flow. For a complete-sentence query requesting CKD-EPI eGFR, the RAG step retrieves CKD-EPI metadata, the LLM identifies that all required inputs are present, emits a structured tool call, local Python computes the result, and the final reply explicitly states that the output came from the verifiable CKD-EPI calculator. For a shorthand query such as “pt 70 yo M Scr 2.0 mg/dL wt 80 kg compute Cockcroft-Gault,” the pipeline analogously retrieves Cockcroft–Gault metadata, resolves required inputs, executes the calculator, and returns a verifiable creatinine-clearance estimate (Kumar et al., 21 Mar 2025).
4. Empirical performance of the chatbot
The chatbot was evaluated against an unassisted off-the-shelf LLM on four natural-language conversation workloads. On full-sentence calculations, MedMCP-Calc achieved 86.4% ± 0.9% accuracy versus 61.6% ± 3.8%, with p=5.54×10⁻⁷. The corresponding calculation-error rates were 0% versus 16.8% ± 3.0%. On shorthand calculations, it achieved 79.2% ± 5.0% versus 62.0% ± 3.2%, with p=1.91×10⁻⁴, and calculation-error rates of 2.4% ± 1.7% versus 18.0% ± 3.7%. On calculator-inquiry prompts such as asking which inputs are needed for CHA₂DS₂-VASc, accuracy was 100% versus 75.2%. On metadata-inquiry prompts such as asking the version of the ASCVD calculator, accuracy was 100% versus 21.0% (Kumar et al., 21 Mar 2025).
The reported latency and cost profile reflects the additional machinery required for retrieval and local execution. Per-query latency for full calculations was 4.8 s ± 0.17 s for MedMCP-Calc versus 7.7 s ± 2.15 s for the unassisted model; shorthand queries were 4.8 s versus 6.0 s; and metadata queries were slower for MedMCP-Calc, 2.6–4.0 s versus 1.3–2.6 s, due to RAG overhead. Cost per query was \$0.036 ± 0.0008` versus `\$0.015–\$0.013, described as a 2–6× increase while remaining approximately 1–4 cents/query (Kumar et al., 21 Mar 2025).
The system paper is explicit that the chatbot is not ready for clinical use. Reported limitations include occasional misassignment of parameter values under heavy medical shorthand, lack of support for chaining multiple calculators automatically, and the possibility that tool-calling reliability may degrade if more than twenty tools are exposed, whereas MDCalc has more than two hundred calculators. The listed future directions are fine-tuning on medical shorthand, reliability alignment or tool-selector LLMs for scaling beyond twenty tools, multi-step workflows in which outputs of one calculator feed into another, and formal clinical validation in real-world workflows before patient-care deployment (Kumar et al., 21 Mar 2025).
5. MedMCP-Calc as a realistic MCP benchmark
The 2026 MedMCP-Calc benchmark reframes calculator use as an end-to-end agentic task. Its motivation is that clinicians often begin with vague, goal-driven questions, must fetch scattered data from an EHR, decide which calculators to chain together, and carry out a sequence of numerical computations, sometimes consulting external references, before arriving at an actionable answer. Existing benchmarks, by contrast, typically present an explicit request to calculate a named score and evaluate only the final arithmetic (Zhu et al., 30 Jan 2026).
The benchmark embeds three MCP servers into a unified agent environment: a PostgreSQL Executor with run_sql, list_tables, and describe_tables for a realistic MIMIC-IV–derived EHR database; Google Search with search and fetch for up-to-date calculator definitions or guideline excerpts; and a Python Executor with run_python for precise numerical computations. The EHR database is described as having nine tables, including patient_information, visit_inpatient, laboratory_result, and vital_signs. At runtime, the LLM agent issues planning actions and tool invocations formatted as MCP calls (Zhu et al., 30 Jan 2026).
MedMCP-Calc comprises 118 scenario tasks across 4 clinical domains: Critical Care & Perioperative Medicine, Internal Medicine & Organ Systems, Neurology & Psychiatry, and Special Populations & Universal Tools. These tasks were distilled from an initial pool of 125 scenarios and approximately 1,970 calculators through a four-stage pipeline consisting of calculator curation, GPT-5–assisted workflow instantiation, database construction with synthetic patients, and physician-led quality control. Prompts are deliberately fuzzy rather than explicit. A representative example asks how to stratify the risk of a patient with chest pain, mildly elevated troponin, blood pressure 140/85, and heart rate 110, and requests a walkthrough of the reasoning rather than naming a specific score (Zhu et al., 30 Jan 2026).
Each task is formalized as a partially observable Markov decision process tuple . Evaluation is process-level rather than limited to the terminal answer. The benchmark reports Task Fulfillment, Calculator Selection, Evidence Acquisition, and Quantitative Precision, and it also defines an overall scenario success rate
where only if the final answer is both valid and internally consistent. In addition, every run_sql, run_python, search, and fetch call is logged, and the paper reports average invocation counts per model (Zhu et al., 30 Jan 2026).
6. Benchmark findings and the CalcMate model
The benchmark evaluates 23 leading models and reports substantial deficits even among top systems. Claude Opus 4.5 is reported as the best open-world performer, with CS=66.66%, EA=71.08%, and QP=33.92% overall. GPT-5 and Gemini 3-Pro are reported at approximately CS≈60%, EA≈53–55%, and QP≈25–27%. Among open-source systems, GLM-4.7 is described as the strongest, with CS=51.89%, EA=50.06%, and QP=15.59%. Many smaller or domain-specific models are reported below 40% on Calculator Selection and under 10% on Quantitative Precision (Zhu et al., 30 Jan 2026).
Performance varies markedly across domains. Models do best in Critical Care & Perioperative Medicine and Internal Medicine & Organ Systems, while performance dips sharply in Neurology & Psychiatry and Special Populations. For Claude Opus in Critical Care & Perioperative Medicine, the paper reports CS=71.30%, EA=73.51%, and QP=36.00%. The paper identifies three principal failure modes in top models: calculator selection under fuzzy queries remains difficult, SQL-based EHR interaction is brittle, and models are reluctant to call run_python or fetch external references, instead trusting imperfect internal arithmetic and static knowledge (Zhu et al., 30 Jan 2026).
An ablation in Appendix C shows that explicit instructions such as “Calculate X using data A, B, C” can increase Calculator Selection by approximately 15–20 points and Quantitative Precision by approximately 10–15 points, but this bypasses the planning steps that the benchmark is intended to assess. The benchmark therefore uses fuzzy prompts specifically to expose planning deficits rather than to maximize arithmetic success under tightly specified instructions (Zhu et al., 30 Jan 2026).
To address these deficits, the authors introduce CalcMate. CalcMate starts from Qwen3-4B-Instruct and incorporates two modules during fine-tuning: Scenario Planning and Tool Augmentation. Scenario Planning uses a synthetic curriculum of 1,000 diverse calculator workflows generated with GPT-5 and annotated with a global chain-of-thought so that the model learns explicit multi-step planning. Tool Augmentation expands each calculator invocation into a full ReAct pattern involving schema inspection, retrieval of reference documents, assembly of run_python code, and interpretation of results. The paper states that no novel mathematical loss was required; the gains arise from richer scenario-planning data and explicit tool-use demonstrations. Reported results are CalcMate-4B: CS=56.97% (+37 pp vs. base Qwen3-4B 19.95%), EA=71.06% (+61.7 pp), QP=15.02% (+11.94 pp) and CalcMate-30B: CS=65.66%, EA=75.31%, QP=20.07% (Zhu et al., 30 Jan 2026).
7. Limitations, misconceptions, and research significance
A recurring misconception is to equate medical-calculator competence with the ability to produce a correct final number from an explicit prompt. The benchmark directly rejects that framing by treating calculator use as a sequence involving intent interpretation, EHR evidence acquisition, calculator selection, optional external reference checking, precise numerical execution, and branching on intermediate results (Zhu et al., 30 Jan 2026). The chatbot system addresses a more restricted version of the same problem by constraining the model to a verified calculator inventory with typed metadata and local execution, thereby reducing calculation errors and improving metadata fidelity (Kumar et al., 21 Mar 2025).
A second misconception is that MedMCP-Calc names a single fixed artifact. In the supplied literature, it names both a deployable chatbot and a benchmarking framework. The chatbot emphasizes verifiable execution over a curated set of eleven calculators, whereas the benchmark emphasizes realistic scenario complexity, MCP-based tool use, and process-level scoring across 118 tasks (Kumar et al., 21 Mar 2025, Zhu et al., 30 Jan 2026).
The main limitations are also different across the two lines of work. In the chatbot, the limiting factors are shorthand parsing errors, lack of automatic calculator chaining, degradation when the tool set grows beyond twenty tools, RAG overhead, and the absence of formal clinical validation (Kumar et al., 21 Mar 2025). In the benchmark, the limiting factors are SQL hallucinations, tool avoidance, domain-specific performance collapses—especially on complex cognitive scores—and efficiency costs associated with long context and aggressive tool augmentation (Zhu et al., 30 Jan 2026).
Taken together, the two MedMCP-Calc projects separate two complementary research problems: constructing a verifiable calculator assistant and evaluating broader agentic workflows for realistic medical calculator scenarios. This suggests a progression in the field from narrow calculator grounding toward full clinical task orchestration, with MCP-based tool ecosystems, structured EHR access, and explicit numerical execution serving as the central technical mechanisms (Kumar et al., 21 Mar 2025, Zhu et al., 30 Jan 2026).