- The paper introduces the SaFE-Scale framework, which evaluates the safety and accuracy of large language models (LLMs) in clinical settings across various deployment conditions and model scales.
- Safety metrics for 34 LLMs reveal that clinician-written clean evidence significantly improves accuracy and reduces high-risk errors, improving accuracy of 34 locally deployed LLMs by 20.6 percentage points.
- Accurate reliability and self consistency rather than the confidence level is critical for the safety of clinical LLMs.
Overview
This paper introduces SaFE-Scale (Safety-Focused Evaluation of Scaling), a framework for measuring how clinical LLM safety changes across model scale, evidence quality, retrieval strategy, context exposure, and inference-time compute. The framework is instantiated on RadSaFE-200, a 200-question radiology multiple-choice benchmark with clinician-defined clean evidence, conflict evidence, and option-level labels for high-risk error, unsafe answer, and evidence contradiction. The central empirical claim is that clinical LLM safety is not a passive consequence of scaling but a deployment property governed primarily by the quality of the evidence supplied to the model.
The evaluation covers 34 locally deployed LLMs from seven families (Qwen, Llama, Gemma/MedGemma, DeepSeek, Mistral, OpenAI-OSS) under six deployment conditions: closed-book prompting, clean evidence, conflict evidence, standard RAG, agentic RAG, and max-context prompting, yielding 40,800 model-condition-question evaluations. Two secondary experiments probe inference-time compute via self-consistency and fixed three-model ensembles.
Benchmark construction
RadSaFE-200 pools three subsets: 104 questions from RadioRAG [(2605.04039) sources], 65 five-option questions from RaR, and 31 newly curated Radiopaedia-derived questions written by a board-certified radiologist. Each of the 865 answer options carries three binary safety labels: high-risk (selection could plausibly cause clinical harm or mismanagement), unsafe (the option directly supports an unsafe diagnosis or recommendation), and contradiction with clean evidence. In total, 33% of options are labeled high-risk, 10% unsafe, and 39% contradictory; 69% of questions contain at least one high-risk option.
Clean evidence is a concise clinician-written explanation of the correct answer (median 32 words); conflict evidence adds one distracting or partially conflicting sentence without changing the reference answer. This paired design isolates the effect of mild evidence perturbation while holding question content constant.
Safety and accuracy decouple across deployment conditions
The headline result is that clinician-written clean evidence produces by far the largest improvement in both accuracy and safety. Model-averaged accuracy rises from 73.5% (closed-book) to 94.1%, while high-risk error falls from 12.0% to 2.6%, contradiction from 12.7% to 2.3%, and dangerous overconfidence from 8.0% to 1.6%. Critically, all 34 models improved on both axes under clean evidence — the effect is not driven by strong models.
| Condition |
Accuracy (%) |
High-risk error (%) |
Contradiction (%) |
Dangerous overconf. (%) |
Latency (s) |
| Closed-book |
73.5 |
12.0 |
12.7 |
8.0 |
18.0 |
| Clean evidence |
94.1 |
2.6 |
2.3 |
1.6 |
17.3 |
| Conflict evidence |
92.5 |
3.5 |
2.6 |
2.3 |
17.5 |
| Standard RAG |
76.0 |
9.6 |
11.7 |
5.7 |
18.4 |
| Agentic RAG |
78.1 |
10.3 |
9.0 |
8.0 |
19.9 |
| Max context |
74.0 |
10.6 |
11.6 |
6.0 |
27.0 |
Two findings stand out against common deployment assumptions. First, agentic RAG improves accuracy over standard RAG (76.0% → 78.1%) and reduces contradiction (11.7% → 9.0%), yet dangerous overconfidence increases to match closed-book levels (8.0%) and high-risk error remains elevated at 10.3% — accuracy and safety move in opposite directions. Second, max-context prompting yields almost no accuracy gain while increasing latency from 18.0 to 27.0 seconds, leaving high-risk error at 10.6%.
Conflict evidence acts as a graded stress test: relative to clean evidence it costs only 1.6 percentage points of accuracy, but safety degrades measurably before any accuracy collapse — high-risk error rises to 3.5% and confidence among high-risk errors increases from 85.4% to 88.3%. A single distracting sentence therefore makes residual failures more dangerous even when aggregate accuracy appears stable.
Confidence is not a reliable safety signal
Reported confidence is uniformly high across correct, incorrect, high-risk, and unsafe outputs under every condition. Under closed-book prompting, mean confidence among high-risk errors is 87.8%; under agentic RAG it reaches 93.0%. Per-model confidence on high-risk errors lies almost exactly on the diagonal against confidence on correct answers, meaning the confidence attached to clinically dangerous wrong answers is indistinguishable from that attached to correct ones. The median gap between these two quantities remains well below the ~30-point threshold above which confidence could plausibly serve as a deployment filter.
A downstream system that suppresses only low-confidence outputs would therefore miss many of the most consequential errors. The dangerous-overconfidence metric — jointly requiring incorrectness, a high-risk or unsafe selected option, and entropy-normalized repeated-sampling confidence ≥ 0.80 — separates conditions cleanly: clean evidence improves this metric in 34 of 34 models, whereas the closed-book-to-agentic-RAG transition is non-uniform (17 improve, 15 worsen). One caveat: confidence here derives from repeated-sampling stability rather than calibrated token probabilities, so this should be read as an answer-stability finding, not a full calibration study.
Different scaling laws for safety and accuracy
Within-family scaling curves fan out widely under closed-book prompting (accuracy ranging from 41.5 to 88.4 within the panel), but collapse to a narrow band under clean evidence, with family-level accuracies between 90.4 (Llama) and 97.0 (DeepSeek). Families with the weakest baselines gain the most — Qwen and Gemma each gain 24.7 percentage points — an equalization pattern indicating that curated evidence compresses scale-driven differences.
A two-way variance decomposition quantifies this: deployment condition explains 43–45% of variance in accuracy and high-risk error and 38% in dangerous overconfidence, whereas model family explains only 9–17%. Within-family standard deviation of high-risk error collapses from 3.8 points (closed-book) to 1.0 point (clean evidence). Scale and family choice thus do not substitute for reliable evidence; they shape the baseline but not the safety ceiling.
Inference-time compute does not produce safety
Self-consistency (20 stochastic samples, majority vote) in eight representative models produced condition-mean accuracy changes of only +0.1 to +0.6 percentage points, with essentially stationary high-risk and unsafe rates. Mean single-to-self-consistency changes across 24 model-condition pairs were +0.39 points for accuracy and +0.27 for high-risk safety — small relative to the ~20-point effects of curated evidence. Confidence remained flat near 96.8 regardless of high-risk error rate.
Fixed three-model ensembles improved aggregate performance relative to the panel average (e.g., 83.0 vs. 73.5 accuracy closed-book) but did not consistently beat their best individual member; averaged across 12 ensemble-condition cases, ensembles were −0.88 points worse than their best member on accuracy and −1.67 worse on dangerous overconfidence. More importantly, ensembles introduce synchronized failure — all three members selecting the same wrong option — at rates up to 9.0% (Frontier ensemble under standard RAG), with most ensemble high-risk errors lying on or near the unanimous-failure diagonal. Agreement among strong models is not equivalent to safety, and may create false reassurance precisely where risk is highest.
Worst-case failures are structured, not random
Clinically consequential errors concentrate in a small recurrent subset. The 15 highest-risk questions under closed-book prompting each produced high-risk errors in more than half of the model panel (rates 52.9–97.1%). The top-ranked case was answered incorrectly by all 34 models, produced high-risk errors in 33, and contradicted the provided evidence in all 34. Notably, unsafe-answer rates were zero in this subset — the dominant failure mode is high-risk misclassification, often accompanied by contradiction. These same questions overlap with ensemble synchronized-failure cases, meaning the items that defeat individual models also defeat agreement among them. Any future mitigation should be evaluated directly against this recurrent failure set.
Limitations
The paper is explicit about several constraints. RadSaFE-200 is text-based and multiple-choice, omitting image interpretation, longitudinal comparison, and free-text reasoning; at 200 questions it limits subgroup precision, and its composition skews toward diagnostic questions (59%). Safety labels rest on a single annotator's clinical judgment, sometimes requiring subjective counterfactual reasoning (particularly for technical, physics, radiation therapy, and negation-style questions); future versions need multiple annotators with inter-rater agreement analysis. Null responses after majority-vote aggregation were scored incorrect but received no safety labels, potentially underestimating option-level failures. Confidence derives from repeated-sampling stability rather than calibrated probabilities. The self-consistency and ensemble experiments are targeted secondary analyses, not exhaustive searches over compute regimes, so more advanced safety-oriented aggregation methods are not ruled out. Finally, the retrieval and agentic implementations represent specific pipeline choices; other corpora, rankers, or controllers could yield different safety profiles.
Conclusion
Across 34 models and six deployment conditions, evidence quality dominates every other scaling axis: clean clinician-written evidence delivers roughly a 20-point accuracy gain and a four-to-five-fold reduction in high-risk error and dangerous overconfidence, while larger models, longer contexts, retrieval, agentic reasoning, self-consistency, and ensembling each fail to reproduce that safety profile and, in some cases (agentic RAG, synchronized ensemble failure), actively introduce new risks. The practical implication is that clinical LLM systems should report high-risk error, contradiction, unsafe-answer rate, dangerous overconfidence, and synchronized failure alongside accuracy, and be evaluated under the actual evidence conditions of deployment. The open questions left by this work concern whether multimodal and free-text extensions preserve these decoupling patterns, whether alternative retrieval designs can close the gap to curated evidence, and whether safety-oriented aggregation can suppress correlated failures.