DiaBENCH: Dual AI & Diabetes Benchmark

• DiaBENCH is a dual-purpose benchmark suite assessing LLMs in both enterprise tool-calling and patient-facing diabetes decision-making.
• DexBench uses real-world CGM and behavioral data across 360,600 queries to evaluate numeric accuracy, reasoning, and personalized actionability.
• The agentic benchmark stresses multi-turn dialogue and tool-disambiguation with 5,000 API specifications to quantify slot-filling errors in enterprise settings.

DiaBENCH refers to two distinct, high-impact benchmarks released in the mid-2020s: one in the field of agentic evaluation for enterprise tool-calling LLMs, and one in diabetes self-management decision making. Both are prominent in their respective domains and are commonly encountered in the literature under either the label “DiaBENCH” or, in the health context, “DexBench.” The following entry covers both instantiations, distinguishing their targets, methodologies, and evaluation criteria.

1. Overview: Dual Usage of DiaBENCH

The term DiaBENCH designates:

• DexBench (a.k.a. DiaBENCH): A benchmark suite for evaluating LLM performance on personalized, context-rich, patient-facing decision making in diabetes care. It leverages real-world CGM and behavioral data, stresses numeric reasoning and interpretability, and dissects LLM errors across seven operationalized patient-facing tasks (Cardei et al., 26 Sep 2025).
• DiaBENCH (Agentic Tool-Use): The dynamic evaluation harness for the DiaFORGE framework, designed to expose and quantify realistic tool-calling errors by LLMs in enterprise settings. It emphasizes agentic, multi-turn dialogue where tool disambiguation and slot-filling under ambiguity are core challenges (Hathidara et al., 4 Jul 2025).

Each instantiation reflects a focus on real-world complexity and interactive error surfaces absent from earlier, more static benchmarks.

2. DexBench: Patient-Facing Diabetes Decision-Making Evaluation

DexBench (DiaBENCH), as introduced in (Cardei et al., 26 Sep 2025), operationalizes seven categories of patient queries using 30-day CGM and behavioral traces from 15,000 subjects (T1D, T2D, prediabetes/HW):

Task Number	Task Category	Data Context
1	Glucose Math	1 day CGM
2	Education	None (templated)
3	Simple Reasoning	1 day CGM+behavior
4	Advanced Reasoning	30 days CGM+behavior
5	Decision Making	7 days CGM+behavior
6	Planning	30 days CGM+behavior
7	Alert/Triage	30 days CGM+behavior

Tasks span from calculation of specialist indices (e.g., MAGE, CONGA) to multi-turn planning and escalation scenarios. A total of 360,600 questions enables granular error breakdown by patient sub-cohort, reasoning type, and temporal granularity. Synthetic, privacy-preserving extensions are used for Calibration/Math questions.

3. DexBench: Dataset Generation and Evaluation Process

Question generation in DexBench is a hybrid of template filling (for numeric/mathematical tasks) and LLM-conditioned drafting (for behavioral and reasoning-focused queries). For Tasks 3–7 and 2, an LLM generates individualized questions, further refined through LLM-based evaluation along five binary axes: fluency, relevance, originality, difficulty, and answerability. De-duplication and human review ensure coverage and minimal redundancy.

Answers are graded with an LLM (Gemini 2.5 Pro) against five binary metrics:

• Accuracy: Numeric or semantic correctness (e.g., must match ground truth within ±2 mg/dL for Task 1).
• Groundedness: No hallucinated data; strict contextual fidelity.
• Safety: Restriction of diagnostic/prognostic statements and high-risk suggestions.
• Clarity: Flesch–Kincaid Grade <8, conciseness.
• Actionability: Concrete, personalized steps; sequential plans for long-term queries.

Aggregate performance is expressed both as percentage pass rates and with standard error of the mean (SEM): $\mathrm{SEM} = \sqrt{\frac{p(1-p)}{n}}$ for $p$ as the pass rate, $n$ as sample size.

4. DexBench: Empirical Results and Limitations

Evaluation of eight LLMs reveals no model dominates across all dimensions. Key outcomes (Cardei et al., 26 Sep 2025):

• GPT-5 achieves highest accuracy (92.0% ± 0.05), strong groundedness (89.0% ± 0.05), but only 58.6% clarity.
• Gemini 2.5 Pro yields higher clarity (70.7% ± 0.08), lower accuracy (83.2% ± 0.06), overall 86.7% average.
• Deepseek R1 leads in clarity (88.8%) but lags in overall accuracy and actionability.
• Open-source, smaller-scale models (Llama 3.1 8B / MedGemma 4B) cluster near 47% overall pass rate.
• Safety rates are high (≥89%) for all models; clarity is the principal area of deficit, particularly on planning and advanced reasoning.
• Latency–performance tradeoff: the highest-performing models are order-of-magnitude slower (GPT-5 ≈ 48s/response).

Tasks involving advanced numeric calculation (MAGE, CONGA), behavioral association, or long-term planning consistently expose weaknesses; several models have <10% accuracy on the most technical glucose-math prompts. For planning, true sequential structured output is rare and actionable responses can be as low as 13% for weaker models.

5. DexBench: Practical Significance and Future Work

DexBench demonstrates that patient-facing diabetes AI must merge domain-specific numeric reasoning, real-world time-series context, low-grade-level articulation, and strict safety. The benchmark recommends integration of calculator modules, context grounding via retrieval, fine-tuning with real patient data, and style adapters for constrained readability. A plausible implication is that future model improvements must target both mathematical dexterity and user-tailored dialogic interface design. Plans for DexBench include multimodal prompt augmentation, richer demographic/medication inputs, and expanded coverage of non-diabetes metabolic health.

6. DiaBENCH: Benchmarking Agentic Tool-Calling in Enterprise LLMs

DiaBENCH is the agentic evaluation suite for DiaFORGE, focused on stress-testing LLMs’ tool-disambiguation and slot-filling behaviors under realistic, adversarial multi-turn conditions (Hathidara et al., 4 Jul 2025). It addresses two critical failure surfaces ignored by static tool-use benchmarks:

• Disambiguation among semantically similar, near-duplicate tools.
• Multi-turn dialogue for eliciting missing required arguments before tool invocation.

The corpus consists of 5,000 production-grade API specifications, with 119 held-out test tools and hard negative distractor pools constructed via embedding similarity to maximize ambiguity. The evaluation harness simulates both a “user-proxy” agent (sampling utterances from a frozen LLM policy) and the candidate assistant, enacting the workflow until a correct, schema-conformant tool-call is produced or a turn limit is hit.

Metrics include:

• Tool-Call Accuracy ($Acc$): indicator for first correct invocation with all required parameters.
• False-Positive Tool-Call Rate ($FTR$): sum of incorrect/hallucinated tool invocations per dialogue.
• Tool-Call Abstention Rate ($TAR$): fraction of dialogues with no tool call issued.

For model $A$ and $B$,

$$\Delta Acc_{A \rightarrow B} = 100\% \times (Acc_B - Acc_A) ~\mathrm{pp}$$

Llama-3.3-Nemotron-DiaFORGE-49B, fine-tuned with DiaFORGE, achieves $Acc=0.89$ ($p$0, $p$1), compared with GPT-4o ($p$2) and Claude-3.5 ($p$3), underscoring the necessity of disambiguation-centric pre-training.

7. DiaBENCH: Impact, Diagnostic Utility, and Community Directions

DiaBENCH for tool use shifts benchmarking toward diagnostically rich, on-policy, adversarial evaluation. It enables precise isolation of slot-filling and tool-disambiguation errors and establishes open, scalable validation practices. Its release, along with a corpus of API specs and dialogue templates, facilitates reproducibility and community extension. Future expansions may incorporate multi-tool workflows, reinforcement learning-based clarification, and more efficient dynamic evaluation with reduced human involvement.

Editor’s term: “DiaBENCH” is sometimes used as a generic short-form to reference either the diabetes or the enterprise agentic-tool benchmark; context in the literature and in each publication determines the intended scope. Both paradigms exemplify recent benchmarking advances that favor context-rich, fully-interactive, and human-realistic evaluation over prior static or off-policy frameworks.