CalcMate: Medical Calculator Agent

• CalcMate is a fine-tuned medical calculator agent that decomposes fuzzy clinical queries into sequential calculator calls.
• It integrates scenario planning with ReAct-style tool augmentation, enabling proactive EHR data retrieval and multi-step computations.
• Benchmark results show enhanced calculator selection and evidence acquisition, although robust quantitative precision remains a challenge.

Searching arXiv for the cited papers to ground the article in the source literature. CalcMate is a fine-tuned medical-calculator agent introduced in "MedMCP-Calc: Benchmarking LLMs for Realistic Medical Calculator Scenarios via MCP Integration" (Zhu et al., 30 Jan 2026). It is designed for realistic clinical calculator use rather than static single-step arithmetic, targeting workflows that require interpretation of fuzzy clinical requests, proactive structured EHR retrieval, scenario-dependent calculator selection, and multi-step numerical computation. In the published system, CalcMate combines scenario-planning supervision with ReAct-style tool augmentation over Model Context Protocol (MCP) servers, and is reported to achieve state-of-the-art performance among open-source models on the MedMCP-Calc benchmark (Zhu et al., 30 Jan 2026).

1. Benchmark setting and problem formulation

CalcMate is defined in relation to MedMCP-Calc, a benchmark intended to evaluate LLMs in realistic medical calculator scenarios. MedMCP-Calc comprises 118 scenario tasks across 4 clinical domains: Critical Care & Perioperative Medicine; Internal Medicine & Organ Systems; Neurology & Psychiatry; and Special Populations & Universal Tools. The tasks feature fuzzy task descriptions mimicking natural queries, structured EHR database interaction, external reference retrieval, and process-level evaluation (Zhu et al., 30 Jan 2026).

The benchmark is motivated by a gap between conventional calculator benchmarks and real clinical use. The paper characterizes real-world calculator use as an adaptive, multi-stage process that requires proactive EHR data acquisition, scenario-dependent calculator selection, and multi-step computation, whereas prior benchmarks focus only on static single-step calculations with explicit instructions. In the authors’ evaluation of 23 leading models, even top performers such as Claude Opus 4.5 exhibit difficulty selecting appropriate calculators for end-to-end workflows given fuzzy queries, poor performance in iterative SQL-based database interactions, and marked reluctance to leverage external tools for numerical computation (Zhu et al., 30 Jan 2026).

MedMCP-Calc tasks are modeled as a POMDP $(\mathsf G,\mathcal S,\mathcal A,\mathcal O,T,E)$. Within that formulation, CalcMate is not merely a calculator executor; it is a planning-and-tool-use agent that must transform an underspecified goal into a sequence of calculator calls, database queries, retrieval steps, and numerical operations. This framing is central to the model’s design because the benchmark evaluates intermediate process quality as well as final task completion.

2. Model basis and architectural design

CalcMate is built by fine-tuning Qwen3-4B-Instruct-2507; a larger variant is built by fine-tuning Qwen3-30B-Instruct-2507. No new Transformer layers or external neural modules are added beyond those in the base model (Zhu et al., 30 Jan 2026). The system therefore relies on targeted supervision and tool-use traces rather than architectural expansion.

The first major modification is the scenario planning module. To address poor multi-step planning, the authors generated 1,000 synthetic scenario tasks using GPT-5. Each example pairs a fuzzy clinical prompt with a full global reasoning plan, defined as a step-by-step sequence of calculators plus decision branches. During fine-tuning, these <prompt, plan> pairs are used as demonstration examples so that the model learns to decompose an underspecified clinical request into a sequence of substeps, which the paper terms scenario planning (Zhu et al., 30 Jan 2026).

The second major modification is tool-augmentation integration. CalcMate is trained to follow the ReAct framework: at each step it either emits a natural-language reasoning trace or a tool-call action. The available MCP servers are the PostgreSQL Executor with list_tables, describe_tables, and run_sql; Google Search with search and fetch; and Python Executor with run_python. All calculator invocations in the training traces follow a fixed pattern consisting of retrieval of the calculator definition via search and fetch, SQL queries to acquire patient inputs, and Python code to perform the numeric computation (Zhu et al., 30 Jan 2026).

This design implies a division of labor between the base LLM and the tool substrate. The LLM is responsible for scenario interpretation, plan decomposition, and action selection, while database access, evidence acquisition, and numerical execution are externalized through MCP tools. A plausible implication is that CalcMate’s gains arise less from latent memorization of medical formulae than from improved orchestration of tool-mediated workflows.

3. Supervision regime and optimization objective

The paper describes CalcMate’s training as supervised fine-tuning rather than RLHF or reward-model tuning. The reported training data consist of 1,000 GPT-5-generated scenario-planning traces together with tool-augmented demonstration traces, in ReAct format, for each required calculator call. No low-level details such as learning rates, batch sizes, or additional loss terms are given beyond supervised fine-tuning (Zhu et al., 30 Jan 2026).

The optimization objective is standard next-token maximum likelihood over the combined reasoning-and-action trace. The published objective is

$$\mathcal{L}(\theta) \;=\; - \sum_{i} \sum_{t} \log p_\theta( y_{i,t} \mid y_{i,<t},\, x_i )\,.$$

Here $D = \{ (x_i, y_i) \}$ denotes the training pairs, with $x_i$ as prompt plus history and $y_i$ as the next-token trace. The model is therefore optimized over a single serialized sequence that includes natural-language reasoning, tool calls, and returned results (Zhu et al., 30 Jan 2026).

The planning workflow is also supervised explicitly. In training, the model sees examples of interpreting a fuzzy clinical request $\mathsf G$, generating a global plan $\pi = [c_1, c_2, \dots, c_K]$ where each $c_k$ is a calculator plus conditional logic, and executing each step by retrieving EHR inputs via SQL, retrieving calculator rules via search and fetch, computing via run_python, and using the output to select the next calculator or branch. The summary further states that the underlying logic follows standard ReAct, enriched by the scenario-planning demonstrations so that the model learns a scoring function for selecting the next calculator $c$ from state $s$ (Zhu et al., 30 Jan 2026).

A frequent misconception in evaluating such systems is to treat tool use as a purely post hoc wrapper around an unchanged LLM. CalcMate is trained directly on enhanced traces in which reasoning and tool-use are interleaved, so its tool dependence is part of the learned policy rather than an external prompt-only convention.

4. MCP-agent execution semantics

CalcMate operates as an MCP-agent within a POMDP. At each step $$\mathcal{L}(\theta) \;=\; - \sum_{i} \sum_{t} \log p_\theta( y_{i,t} \mid y_{i,<t},\, x_i )\,.$$0, the state $$\mathcal{L}(\theta) \;=\; - \sum_{i} \sum_{t} \log p_\theta( y_{i,t} \mid y_{i,<t},\, x_i )\,.$$1 comprises the full chat history, including previous tool calls and observations. The model then emits either a natural-language next step, denoted $$\mathcal{L}(\theta) \;=\; - \sum_{i} \sum_{t} \log p_\theta( y_{i,t} \mid y_{i,<t},\, x_i )\,.$$2, or a tool-call action

$$\mathcal{L}(\theta) \;=\; - \sum_{i} \sum_{t} \log p_\theta( y_{i,t} \mid y_{i,<t},\, x_i )\,.$$3

The environment executes $$\mathcal{L}(\theta) \;=\; - \sum_{i} \sum_{t} \log p_\theta( y_{i,t} \mid y_{i,<t},\, x_i )\,.$$4 and returns observation $$\mathcal{L}(\theta) \;=\; - \sum_{i} \sum_{t} \log p_\theta( y_{i,t} \mid y_{i,<t},\, x_i )\,.$$5, after which the internal history is updated as

$$\mathcal{L}(\theta) \;=\; - \sum_{i} \sum_{t} \log p_\theta( y_{i,t} \mid y_{i,<t},\, x_i )\,.$$6

This state-update rule makes the agent history-explicit: subsequent reasoning conditions on prior tool trajectories rather than only on the initial prompt (Zhu et al., 30 Jan 2026).

The paper’s ReAct loop is summarized as follows:

$D = \{ (x_i, y_i) \}$7

Because CalcMate was fine-tuned on traces that always follow the pattern of retrieving calculator definitions, issuing SQL to obtain inputs, and then running Python for the numeric step, the model learns to chain these tools reliably (Zhu et al., 30 Jan 2026).

From a systems perspective, this workflow couples knowledge retrieval, data access, and executable computation into a single recurrent policy loop. The significance of the design is not simply that external tools are available, but that the training traces normalize a particular ordering of evidence acquisition and computation. That ordering directly targets the benchmark’s observed failure modes: tool avoidance, under-specified planning, and weak iterative SQL behavior.

5. Reported benchmark performance

MedMCP-Calc evaluates systems using four process-level metrics defined in Appendix A.2: Task Fulfillment (TF), Calculator Selection (CS), Quantitative Precision (QP), and Evidence Acquisition (EA). Table 3 of the paper reports the following overall averages (Zhu et al., 30 Jan 2026).

Model	TF	CS	EA	QP
Claude Opus 4.5	100 %	66.66 %	71.08 %	33.92 %
GPT-5	100 %	59.81 %	53.12 %	26.70 %
GLM-4.7	96.61 %	51.89 %	50.06 %	15.59 %
Qwen3-4B (base)	100 %	33.42 %	30.60 %	7.91 %
CalcMate-4B	100 %	56.97 %	71.06 %	15.02 %
CalcMate-30B	100 %	65.66 %	75.31 %	20.07 %

These results position CalcMate-4B as the strongest reported open-source model on CS and EA, while also nearly doubling QP relative to the base Qwen3-4B. The 30B variant narrows the gap to proprietary systems, especially on calculator selection and evidence acquisition (Zhu et al., 30 Jan 2026).

The paper’s synthesized findings attribute these gains to the two principal design choices. Scenario planning fine-tuning improves decomposition of fuzzy clinical requests into the correct sequence of calculators, with CS rising from approximately $$\mathcal{L}(\theta) \;=\; - \sum_{i} \sum_{t} \log p_\theta( y_{i,t} \mid y_{i,<t},\, x_i )\,.$$7 to approximately $$\mathcal{L}(\theta) \;=\; - \sum_{i} \sum_{t} \log p_\theta( y_{i,t} \mid y_{i,<t},\, x_i )\,.$$8. Aggressive tool augmentation raises EA from approximately $$\mathcal{L}(\theta) \;=\; - \sum_{i} \sum_{t} \log p_\theta( y_{i,t} \mid y_{i,<t},\, x_i )\,.$$9 to approximately $D = \{ (x_i, y_i) \}$0 and QP from approximately $D = \{ (x_i, y_i) \}$1 to approximately $D = \{ (x_i, y_i) \}$2 in the 4B model, or approximately $D = \{ (x_i, y_i) \}$3 in the 30B model (Zhu et al., 30 Jan 2026).

At the same time, the metric profile shows that perfect or near-perfect TF does not imply high numerical reliability. CalcMate-30B reaches $D = \{ (x_i, y_i) \}$4 TF but only $D = \{ (x_i, y_i) \}$5 QP. The evaluation therefore distinguishes successful end-to-end task progression from exact quantitative execution, and the distinction is consequential for any medical-calculator deployment.

6. Limitations, domain variability, and related research directions

The paper identifies several remaining challenges. No open model yet achieves high numeric precision; even CalcMate-30B remains at approximately $D = \{ (x_i, y_i) \}$6 QP, leaving substantial headroom. The authors also note increased inference latency and context length from proactive tool augmentation, which may create deployment constraints in resource-constrained settings. In addition, performance varies considerably across clinical domains: all models, including CalcMate, perform substantially worse on Neurology & Psychiatry and Special Populations, where scoring systems are more complex (Zhu et al., 30 Jan 2026).

These observations delimit what CalcMate has and has not solved. It improves scenario decomposition, evidence acquisition, and tool engagement, but it does not eliminate numeric brittleness. Nor does it make domain difficulty uniform. The published evidence therefore supports a view of CalcMate as a targeted advance in open-source medical calculator workflows rather than as a complete solution to calculator-grounded clinical reasoning.

A plausible extension concerns formal mathematical interoperability. "Semantic Preserving Bijective Mappings of Mathematical Formulae between Document Preparation Systems and Computer Algebra Systems" describes a bijective mapping between semantic LaTeX expressions built from the DLMF/DRMF macro set and CAS representations in Maple and Mathematica, using a lexicon-based translation pipeline, Youssef’s Part-of-Math tagger, and inverse translation over Maple Inert_Form or analogous CAS structures (Cohl et al., 2021). This suggests a route by which a CalcMate-like system could couple executable calculator workflows with a semantic LaTeX↔CAS layer for formula verification, canonicalization, or MathML generation. Such an interoperability layer is described as suitable for integration into a system such as CalcMate, but it is not presented as part of CalcMate itself (Cohl et al., 2021).

Within the literature represented here, CalcMate occupies a specific niche: it is an MCP-integrated, ReAct-trained, scenario-planning medical calculator agent optimized for realistic multi-step clinical workflows. Its main contribution is to show that focused supervision over fuzzy-query decomposition and aggressive tool-use traces can materially improve open-source performance on calculator selection and evidence acquisition, while also making clear that robust quantitative precision remains an open problem (Zhu et al., 30 Jan 2026).