Clinical Safety-Effectiveness Dual-Track Benchmark

• Clinical Safety-Effectiveness Dual-Track Benchmark (CSEDB) is a framework that separately evaluates clinical safety and effectiveness by decoupling technological outputs from clinical outcomes.
• It employs expert-designed, scenario-based testing with consequence-weighted scoring to assess key risk factors such as critical illness recognition and medication safety.
• The dual-track structure informs evaluations across diverse applications, including decision-support LLMs, radiology QA systems, and adaptive clinical trial methodologies.

Clinical Safety-Effectiveness Dual-Track Benchmark (CSEDB) denotes a family of evaluation frameworks in which clinical safety and clinical effectiveness are assessed as distinct but linked objects rather than collapsed into a single undifferentiated score. In its narrow sense, the term refers to a multidimensional benchmark for clinical decision-support LLMs built from expert-authored open-ended scenarios and consequence-weighted scoring; in a broader sense, later work uses the same dual-track logic to organize evaluation of clinical computer-use agents, consultation agents, refusal-safe-completion behavior, radiology QA systems, and even adaptive clinical-trial and safety-analysis methods. A key conceptual precursor is the separation of treatment-agnostic technological assurance from therapy-specific clinical assurance, which formalizes the idea that machine outputs and patient outcomes should be argued separately and then explicitly linked (Deevy et al., 2023, Wang et al., 31 Jul 2025).

1. Definition and conceptual lineage

The core idea of CSEDB is separation of concerns. In the medical-device assurance setting, “technological effects” consider the device solely as a machine that produces deterministic outputs given specific inputs, whereas “clinical effects” refer to the physiological response of a human patient to the use of the medical device and its operating procedures during a specific clinical treatment. The corresponding Technological Assurance Case (TAC) demonstrates the safety of the manufacturer’s device independent of any clinical treatment, and the Clinical Assurance Case (CAC) demonstrates the safety and clinical effectiveness of the device when it is used in a specific clinical treatment. The CAC may reference items in the TAC, but “it is crucial that there are no references from the TAC to any dependent CAC,” establishing a one-way linkage discipline that later dual-track benchmarks mirror (Deevy et al., 2023).

The specific benchmark named CSEDB was introduced as “a multidimensional, expert-consensus benchmark that jointly evaluates safety and effectiveness in clinical decision support via open-ended scenarios.” Its design separates a safety gate and an effectiveness gate, uses risk weights from 1 to 5, and defines high-risk scenarios as weight levels 4–5. Safety emphasizes critical illness recognition and medication safety, including absolute contraindications and complex risk control, while effectiveness emphasizes guideline adherence and optimization of diagnostic and therapeutic pathways, including evidence-based planning and patient-centric considerations (Wang et al., 31 Jul 2025).

A broader interpretation appears in subsequent work. Several later papers explicitly map their own evaluation frameworks into a Clinical Safety-Effectiveness Dual-Track Benchmark structure, even when the original task is not open-ended question answering. This suggests that CSEDB has evolved from a single benchmark name into a general evaluation pattern for clinical AI systems whose utility cannot be characterized adequately by correctness, task completion, or average accuracy alone (Yu et al., 2 Jun 2026, Ding et al., 18 Nov 2025).

2. Architectural patterns across benchmark families

Across the literature, CSEDB-style evaluation appears in several recurring architectural forms: open-ended clinical reasoning benchmarks, interactive GUI benchmarks, consultation-process benchmarks, safety-calibration benchmarks for borderline prompts, and deployment-condition benchmarks for multiple-choice clinical QA. The common structure is not a single metric, but a decomposition in which clinical usefulness is measured alongside a separately operationalized safety track.

Context	Effectiveness side	Safety side
Clinical decision-support LLMs	Diagnostic and management effectiveness	Critical illness recognition, medication safety
Clinical computer-use agents	Task completion, intent/step success	Five clinical safety dimensions
Consultation agents	History taking, diagnosis, follow-up quality	Regimen safety critic, dynamic constraint satisfaction
Borderline health prompts	Safe completion quality	Over-refusal and harmful compliance control
Radiology QA deployment	Accuracy	High-risk error, unsafe answer, contradiction, dangerous overconfidence

The open-ended LLM benchmark contains 2,069 clinical Q&A items spanning 26 clinical departments and 30 indicators, split into 17 safety criteria and 13 effectiveness criteria. Safety criteria named in the paper include S01 “Critical illness recognition,” S02 “Fatal diagnostic errors,” S03 “Absolute contraindicated medications,” S05 “Errors in drug dosage calculation,” S06 “Fatal drug–drug interactions,” S09 “Accounting for severe allergy history,” S11 “Fabrication of medical information,” and S17 “Compliance with standardized procedural practices.” Effectiveness criteria named in the paper include E01 “Diagnosing common conditions,” E03 “Differential diagnosis,” E04 “Guideline adherence,” E05 “Prioritization in multimorbidity,” E09 “Follow-up planning and monitoring,” E10 “Appropriateness of laboratory and imaging test recommendations,” and E13 “Scientific validity of combination therapy regimens” (Wang et al., 31 Jul 2025).

In MedCUA-Bench, the dual-track split is between clinical reasoning and UI execution. Each task appears twice: once as an intent-level goal and once as a step-level goal. The benchmark is screenshot-only, uses low-level pixel actions through the BrowserGym pixel interface, and evaluates performance with a deterministic checker over strict task completion and five clinical safety dimensions: Patient identity, Data accuracy, Information fidelity, Record integrity, and Workflow safety. This design was motivated by the claim that “task-completion-only” evaluation is invalid for clinical GUIs because of domain knowledge demands, distinctive UI design, lack of public testbeds, and safety needs (Yu et al., 2 Jun 2026).

MedConsultBench applies the same separation within a full consultation cycle. It evaluates history taking, diagnosis, treatment planning, and follow-up Q&A through Atomic Information Units (AIUs), Minimum Necessary Information (MNI) sets, inquiry-logic templates, and a regimen safety critic. Its 22 fine-grained metrics are divided into process-oriented effectiveness measures such as MNI-Comp, IGE, F1$^{core}$, and SWDS, and safety-oriented measures such as PSC, DDIV, PCR, and DCSR, with safety gating for hard violations such as absolute contraindications, major DDIs, and gross dosing errors (Qiao et al., 19 Jan 2026).

Health-ORSC-Bench reinterprets the dual-track problem as one of calibration under intent ambiguity. It measures Over-Refusal on benign health-related prompts and Safe Completion quality on borderline prompts that should receive helpfully constrained answers rather than blanket refusal. Its dataset contains 31,920 benign boundary prompts across seven health categories, with difficulty tiers Easy-5K, Medium-5K, and Hard-1K, and its judge rubric separately scores Safety, Helpfulness, and Intent (Zhang et al., 25 Jan 2026).

MedBench v4 generalizes the dual-track idea into a large benchmarking infrastructure rather than a single task type. It comprises over 700,000 expert-curated tasks across 24 primary and 91 secondary specialties, with LLM, multimodal, and agent tracks. These are organized conceptually into an effectiveness track for clinical capability and a safety/ethics track for harm avoidance, guideline compliance, medication safety, ethical governance, and adversarial robustness (Ding et al., 18 Nov 2025).

3. Scoring systems and mathematical formulations

The original CSEDB scoring model is consequence-weighted. For binary items, the score is either 0 or 1 depending on full adherence to the gold standard. For graded items, the score is normalized by the total possible rule weights:

$$\mathrm{Score}_{\mathrm{graded}} = \frac{\sum_{i=1}^{n} r_i}{\sum_{i=1}^{n} s_i}.$$

Each case carries a consequence weight $w_i \in \{1,2,3,4,5\}$, and the aggregate benchmark score is

$$S_{\mathrm{total}} = \frac{\sum_{i=1}^{n} w_i \cdot \mathrm{Score}_i}{\sum_{i=1}^{n} w_i},$$

with analogous gate-specific forms for $S_{\mathrm{safety}}$ and $S_{\mathrm{effectiveness}}$. The effectiveness gate additionally uses a 70%-20%-10 structural emphasis over high-value diagnostic decisions, intermediate management tasks, and patient experience optimization (Wang et al., 31 Jul 2025).

Other CSEDB-style frameworks retain the same dual-track separation but use different observables. Health-ORSC-Bench defines Safe Completion Rate as

$$\text{SCR} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}[R \in sc],$$

where $sc$ includes both Partial Answer and Full Answer under Safe. The benchmark discusses Over-Refusal Rate as the rate at which models refuse benign health-related prompts, and it stratifies both ORR and SCR by difficulty and category rather than collapsing them into a single utility score (Zhang et al., 25 Jan 2026).

MedCUA-Bench formalizes a task as $$T = (S_0, E, C, G_{\mathrm{intent}}, G_{\mathrm{step}})$$, uses strict task completion $s_{\mathrm{task}} \in \{0,1\}$, tracks a progress vector $$\mathrm{Score}_{\mathrm{graded}} = \frac{\sum_{i=1}^{n} r_i}{\sum_{i=1}^{n} s_i}.$$0, and audits a deduplicated violation set $$\mathrm{Score}_{\mathrm{graded}} = \frac{\sum_{i=1}^{n} r_i}{\sum_{i=1}^{n} s_i}.$$1 across the five safety dimensions. Its strict success criterion is safety-gated: $$\mathrm{Score}_{\mathrm{graded}} = \frac{\sum_{i=1}^{n} r_i}{\sum_{i=1}^{n} s_i}.$$2 iff the episode reward is $$\mathrm{Score}_{\mathrm{graded}} = \frac{\sum_{i=1}^{n} r_i}{\sum_{i=1}^{n} s_i}.$$3, which occurs only when the task is completed and no safety violation is recorded. The benchmark therefore treats “completion” and “safe completion” as different outcomes (Yu et al., 2 Jun 2026).

MedBench v4 uses track-level averaging and judge-calibrated scores. All task-specific metrics are rescaled to 0–100, and the overall score is

$$\mathrm{Score}_{\mathrm{graded}} = \frac{\sum_{i=1}^{n} r_i}{\sum_{i=1}^{n} s_i}.$$4

where each dimension score is itself the macro-average of rescaled task scores. Open-ended responses are scored by an LLM-as-a-judge calibrated to human ratings, with judge outputs on a 0–5 scale converted by $$\mathrm{Score}_{\mathrm{graded}} = \frac{\sum_{i=1}^{n} r_i}{\sum_{i=1}^{n} s_i}.$$5 (Ding et al., 18 Nov 2025).

In deployment-focused radiology evaluation, SaFE-Scale retains accuracy as the effectiveness axis but disaggregates safety into High-risk error, Unsafe answer, Contradiction, and Dangerous overconfidence. Dangerous overconfidence is defined jointly over incorrectness, clinical risk, and high confidence, with the main threshold set at $$\mathrm{Score}_{\mathrm{graded}} = \frac{\sum_{i=1}^{n} r_i}{\sum_{i=1}^{n} s_i}.$$6. The paper explicitly argues that safety is not exhausted by accuracy, because confidence concentration can remain high on clinically consequential wrong answers (Wind et al., 5 May 2026).

4. Empirical findings and safety-effectiveness trade-offs

The original CSEDB study reported moderate aggregate capability and a clear safety deficit. Across six LLMs, the average total score was 57.2%, safety 54.7%, and effectiveness 62.3%. High-risk scenarios showed a significant 13.3% performance drop relative to ordinary-risk scenarios, with $$\mathrm{Score}_{\mathrm{graded}} = \frac{\sum_{i=1}^{n} r_i}{\sum_{i=1}^{n} s_i}.$$7 after Holm-adjusted testing. MedGPT achieved the strongest reported gate scores, with safety 0.912 and effectiveness 0.861, whereas general-purpose models showed weaker safety, especially on contraindications, dosing, DDIs, allergy history, hallucinated clinical details, and procedural compliance (Wang et al., 31 Jul 2025).

MedCUA-Bench showed that task completion alone is an inadequate surrogate for clinically reliable software use. Across 23 agents, the best closed-source model reached 54.2% strict success, while all models remained below 9% on the real OpenEMR. Open-source agents averaged only 2.5%, with the best reaching 16.2%. Across 9,936 episodes, there were 0 critical, 53 major, and 15 minor violations, but the paper states that the zero-critical count reflects capability limits—agents often time out before reaching harmful endpoints—rather than proof of safety (Yu et al., 2 Jun 2026).

MedBench v4 reported a similarly strong gap between raw capability and safety. Base LLMs reached a mean overall score of 54.1/100, but the safety/ethics dimension averaged only 18.4/100. Multimodal models performed worse overall at 47.5/100. Agentic systems built on the same backbones substantially improved end-to-end performance, reaching a mean overall score of 79.8/100 and an average safety score of 73.4/100, with the best Claude Sonnet 4.5–based agent achieving up to 85.3/100 overall and 88.9/100 on safety tasks. The paper nevertheless states that real-world risk mitigation still requires prospective validation beyond the benchmark (Ding et al., 18 Nov 2025).

MedConsultBench exposed a related discrepancy between apparent diagnostic competence and process integrity. Systematic evaluation of 19 LLMs showed that high diagnostic accuracy often masks deficiencies in information-gathering efficiency and medication safety, and follow-up adaptation was identified as a universal bottleneck. For example, DCSR remained at 0.49 for Gemini-3-Pro-Preview and 0.46 for gpt-5.1 (high), versus a clinician baseline of 0.814. The benchmark’s ablation results showed that static accuracy substantially overestimates readiness once process and safety constraints are enforced (Qiao et al., 19 Jan 2026).

Health-ORSC-Bench identified a different trade-off: safety optimization can manifest as excessive refusal rather than safe helpfulness. Safety-optimised models frequently refused up to 80% of “Hard” benign prompts, while domain-specific models often sacrificed safety for utility. On Hard-1K, GPT-OSS-120B reached 81.10% overall rejection, GPT-5 66.80%, whereas Qwen-Max and Qwen-Plus were at 0.10%; Qwen-Max combined near-zero over-refusal with approximately 70% SCR on Hard-1K. The benchmark characterizes this as a refusal-compliance calibration problem rather than a simple safety gain (Zhang et al., 25 Jan 2026).

SaFE-Scale makes the strongest explicit statement against the common assumption that accuracy implies safety. In RadSaFE-200, clean evidence increased mean accuracy from 73.5% to 94.1%, while reducing high-risk error from 12.0% to 2.6%, contradiction from 12.7% to 2.3%, and dangerous overconfidence from 8.0% to 1.6%. By contrast, standard RAG and agentic RAG did not reproduce this profile: agentic RAG improved accuracy over standard RAG and reduced contradiction, but high-risk error and dangerous overconfidence remained elevated. Max-context prompting increased latency without closing the safety gap, and additional inference-time compute produced only limited gains (Wind et al., 5 May 2026).

5. Extension beyond AI benchmarking to devices and clinical trials

The dual-track logic also appears in medical-device assurance and clinical-trial methodology. In the TAC/CAC framework, the benchmark-like separation is between technological safety-effectiveness and clinical safety-effectiveness. The TAC covers deterministic outputs, hazard analyses, environmental interfaces, and therapy-agnostic operating procedures; the CAC covers therapy-specific requirements, patient variability, treatment goals, and outcome acceptability. A “single TAC could be linked with multiple CACs” for multiple treatments, which makes reuse and complexity reduction central design principles for later CSEDB-style structures (Deevy et al., 2023).

In adaptive dose-finding, SEEDA and SEEDA-Plateau instantiate a dual-track evaluation in explicitly statistical terms. The objective is to maximize cumulative efficacy while satisfying a toxicity safety constraint with high probability:

$$\mathrm{Score}_{\mathrm{graded}} = \frac{\sum_{i=1}^{n} r_i}{\sum_{i=1}^{n} s_i}.$$8

The corresponding CSEDB-style metrics are cumulative efficacy, regret, recommendation success probability, allocation success probability, ToxicityViolationAvg, UnsafeAllocCount, and sample efficiency. The benchmark blueprint derived from this work therefore treats efficacy gains and safety adherence as co-primary evaluation targets rather than as a single maximum tolerated dose criterion (Shen et al., 2020).

SAFE extends the same principle to post hoc clinical-trial safety profiling. It defines clinically meaningful Synergy Areas (SAs), requires “compelling evidence” from at least two elementary safety findings within an SA, and then controls FDR across SAs. Its partial conjunction alternative is

$$\mathrm{Score}_{\mathrm{graded}} = \frac{\sum_{i=1}^{n} r_i}{\sum_{i=1}^{n} s_i}.$$9

and the within-SA decision rule is

$w_i \in \{1,2,3,4,5\}$0

Simulation studies showed nominal error-rate control, and the case studies highlighted an important dual-track principle: single extreme signals can be screened out when they do not satisfy the benchmark’s stronger evidence requirement (Zhan et al., 4 May 2026).

CWTA provides a different extension, one centered on risk-benefit integration over time. It defines a patient-level trajectory score

$w_i \in \{1,2,3,4,5\}$1

where efficacy and toxicity states are jointly weighted. In the advanced cancer simulations, the 6 × 5 CWTA-RBA matrix reduced sample size requirements relative to efficacy-only CWTA when the experimental drug was both more effective and less toxic: 44 vs 53 at hazard ratio 0.6, 87 vs 106 at 0.7, and 212 vs 279 at 0.8. In the PCLX-001-01 phase I trial, Cohort 6 (210 mg) was significantly better than all other cohorts by CWTA RBA ($w_i \in \{1,2,3,4,5\}$2), providing objective validation for the selected dose (Chauhan et al., 2024).

6. Limitations, misconceptions, and future directions

A recurrent misconception in this literature is that stronger average performance implies safer clinical behavior. Multiple benchmarks reject that inference, but they do so in different ways: MedCUA-Bench shows that task completion can conceal unsafe actions; MedConsultBench shows that diagnostic accuracy can conceal process failures; Health-ORSC-Bench shows that aggressive refusal can degrade clinical usefulness; and SaFE-Scale shows that accuracy, contradiction, high-risk error, and dangerous overconfidence follow different empirical patterns under different deployment conditions (Yu et al., 2 Jun 2026, Qiao et al., 19 Jan 2026, Zhang et al., 25 Jan 2026, Wind et al., 5 May 2026).

Standardization remains incomplete. The original CSEDB is single-turn and primarily Chinese, with limited multimodal coverage; MedCUA-Bench’s synthetic HTML scenarios are explicitly described as upper bounds relative to real systems; MedConsultBench notes simulator–reality gap, LLM-judge bias, and locale dependence; Health-ORSC-Bench is English-only and depends on LLM-as-judge scoring; MedBench v4 notes limited institutional diversity and no formal significance testing across models in the reported release; TAC/CAC, although structurally influential, does not specify full evidence catalogs or formal interface specifications between the two cases (Wang et al., 31 Jul 2025, Yu et al., 2 Jun 2026, Qiao et al., 19 Jan 2026, Zhang et al., 25 Jan 2026, Ding et al., 18 Nov 2025, Deevy et al., 2023).

The main future direction is therefore not simply larger benchmark scale, but stronger formalization of the safety-effectiveness interface. The literature points toward several convergent needs: reusable templates for linked safety and effectiveness arguments; paired or multi-view task formulations that separate planning, execution, and patient risk; better calibration of LLM-as-judge systems to clinician ratings; broader multimodal and longitudinal clinical settings; empirical validation of safety gating and complexity reduction; and prospective real-world validation for systems that score well under current benchmark conditions. Taken together, these developments suggest that CSEDB is becoming a general methodology for evaluating clinical systems under a safety-first, explicitly decomposed view of performance rather than a single benchmark tied to one modality or one model class (Ding et al., 18 Nov 2025, Qiao et al., 19 Jan 2026, Zhan et al., 4 May 2026).