Counterfactual Evaluation Reveals Hidden Capability Profiles in Clinical LLMs and Agents

Abstract: Two clinical AI systems can score nearly identically on coverage-based rubrics yet behave radically differently when their patient inputs change: one updates its recommendations to match the new clinical signal, while the other produces the same output regardless. We introduce the Causal Sensitivity Score (CSS), a pre-registered interventional metric that mutates oncology tumor-board cases along five clinically meaningful dimensions - biomarker flips, prior-treatment failures, biomarker removals, surgery-status changes, and stage perturbations - and scores whether each model updates its recommendations in the pre-registered correct direction using a {0, 0.5, 1.0} scale. Benchmarked against the Consensus Match Score (CMS), a coverage-based weighted recall metric, six frontier models from three labs evaluated in single-shot inference across 224 cases rank in nearly opposite orders: all six models change rank, the CMS-worst model becomes CSS-best, and one upper-mid CMS model ranks last on CSS. We further surface a universal safety blind spot: every frontier model fails on surgery-status interventions (at most 17.2% CSS on Family D), a finding CMS does not expose. The metric also transfers to tool-using agents: in a ReAct-style experiment, tool use improves CSS for five of six models (+2.5 to +20.3 percentage points), yet the lowest-CSS model retrieves the same chart sections and still fails to update its recommendations - revealing a structural responsiveness deficit visible only under counterfactual evaluation. Cross-judge replication and three-rater medical-professional validation confirm the aggregate findings. Interventional pre-registered metrics like CSS complement coverage-based evaluation for clinical AI agents: they capture responsiveness that coverage metrics miss and offer a candidate dense reward signal for future agentic RL systems.

• The paper introduces the Causal Sensitivity Score (CSS) to evaluate how clinical LLMs adapt outputs when input patient case facts are systematically altered.
• It shows a rank inversion compared to consensus-based metrics, exposing hidden capabilities and critical safety blind spots, especially in surgery-related inputs.
• The study validates CSS through oncology tumor-board cases and agent settings, demonstrating its utility for reinforcement learning and safer clinical AI deployment.

Counterfactual Evaluation Reveals Hidden Capability Profiles in Clinical LLMs and Agents

Introduction

This paper introduces and validates the Causal Sensitivity Score (CSS), a pre-registered, intervention-based evaluation metric for clinical LLMs and agents. Motivated by the limitation of coverage-based evaluation, which emphasizes output overlap with reference values but neglects behavioral sensitivity to clinically critical input changes, the CSS measures whether model recommendations update appropriately when presented with systematically altered patient-case facts. The experimental design encompasses six frontier LLMs from three major labs, evaluated on both single-shot inference and a ReAct-style agent setting, using detailed oncology tumor-board cases.

Conventional Evaluation Limitations and Motivations for CSS

The prevailing evaluation paradigm for clinical LLMs utilizes string-similarity or LLM-as-judge rubrics to measure alignment with consensus outputs. These metrics, exemplified by the Consensus Match Score (CMS), are insensitive to the causal relationship between patient input variation and model output adaptation. An LLM that always recommends a particular regimen, regardless of relevant clinical input, can still optimize such metrics if the recommendation happens to align with consensus. Conversely, in agentic deployment, an LLM agent may retrieve relevant patient data but fail to incorporate new information into its recommendations. This distinction highlights the need for a metric that interrogates the responsiveness of models to counterfactual input variation.

The Causal Sensitivity Score: Metric Definition and Evaluation Protocol

The CSS systematically mutates patient cases across five clinically meaningful axes: biomarker status flips, prior-treatment failures, biomarker information stripping, discrete changes in surgery status, and stage perturbations. Each mutation is specified in a pre-registered catalog with expected output shifts and case-level scoring rules in $\{0.0, 0.5, 1.0\}$. For each model and intervention, the CSS pipeline generates both baseline and mutated-case outputs, and an independent LLM judge (with safeguards against self-judging bias) evaluates whether outputs reflect the correct adaptive change. The final CSS aggregates these per-case/intervention scores, providing both a global and per-family estimate of behavioral sensitivity.

Key Findings

CMS vs. CSS: Rank Inversion and Revealed Heterogeneity

A critical finding is the near-complete inversion of model rankings when moving from coverage-based (CMS) to counterfactual (CSS) evaluation. For all six models, ranking changes; the CMS-best model (gpt-5) drops to fourth in CSS, while the CMS-worst (grok-4.20-reasoning) emerges as the most causally sensitive. Spearman $\rho = -0.49$ demonstrates substantial ordinal discordance. This supports the claim that coverage-based metrics conflate rote pattern-matching with genuine clinical understanding.

Figure 1: Rank disagreement between CMS and CSS for six frontier models, visualizing complete rank inversion across the two evaluation paradigms.

Per-family decomposition further reveals distinct capability profiles: different models "win" different clinical perturbation families, indicating that coarse-grained aggregation masks important heterogeneity.

Figure 2: Per-family CSS, showing that model strengths are highly family-dependent, and that all models fail on surgery status interventions (Family D).

Universal and Severe Failure on Surgery-Status Perturbations

The CSS exposes a critical safety blind spot not surfaced by CMS: all six models catastrophically fail to adapt their recommendations when surgery status is mutated (Family D), with best-case performance below 18%. This failure directly implicates the models' inability to condition on crucial clinical events, which have a direct impact on therapeutic sequencing.

Agent Setting: Robustness, Tool Use, and Structural Responsiveness Deficits

Transitioning to a tool-using ReAct agent setting, the CSS protocol transfers directly with no design modifications. Tool use markedly improves CSS for five models (gains +2.5 to +20.3 percentage points) as the agent context more closely matches deployment reality. However, gpt-5.4 demonstrates a persistent deficit: its scores are unaffected despite retrieving the same information as other models—a pattern indicative not of retriever failure but a deficit in behavioral responsiveness.

Figure 3: Tool use elevates CSS for all but one model; gpt-5.4 remains unresponsive, implicating a process-level insensitivity rather than an information-access issue.

Per-case analysis corroborates this result, showing that for gpt-5.4, output adaptation is decoupled from information retrieval.

Figure 4: Per-case scatter between single-shot and tool-using CSS, with most models showing improvement while gpt-5.4 clusters on the diagonal, indicating no agentic benefit.

Score distribution tracking reveals that tool use meaningfully reduces wrong-direction recommendations for responsive models, but gpt-5.4 remains inert.

Figure 5: Stacked barplots visualizing the distributional shift under tool use—reduction in wrong/partial outputs for five models, little change for the structurally unresponsive model.

Analysis of tool-calling patterns confirms that the information accessed does not explain these discrepancies: all models access the same sections at similar rates, isolating the failure to behavioral response rather than upstream retrieval.

Figure 6: Tool-call distribution is consistent across models, exonerating retrieval frequency as a confound for observed capability divergence.

Human-Judge Validation and Consistency

Three medical professionals independently annotated a subset of tuples. Human and LLM-judge aggregate CSS scores closely align in key families (Family D: LLM mean 0.10, human mean 0.09), although per-case agreement is moderate. This indicates that, at the population level, the LLM-judged CSS metric tracks medically meaningful differences, but that case-level variability and incoherence in some interventions can privilege or penalize certain models unfairly. Annotator error and medical incoherence are most prevalent in the complex Family D scenarios, highlighting the need for future catalog refinement and imputation guards.

Figure 7: LLM judge vs. human-majority validation scatter, highlighting population-level agreement and case-level scoring discrepancies.

Theoretical and Practical Implications

By employing pre-registered, mutation-based evaluation, CSS advances the causal evaluation paradigm for agentic AI, preventing post-hoc metric design and reducing selection bias. Practically, the methodology demonstrates that coverage-based scoring is an insufficient proxy for safety- and reliability-critical responsiveness in clinical recommendation tasks. The revealed capability axis—causal sensitivity to input—is both a diagnostic for deployment risk and a candidate dense reward signal for reinforcement learning of agentic models. The observed blind spots and possible architectural insensitivities (e.g., gpt-5.4's retriever-actor decoupling) motivate a research agenda toward improved alignment and error correction in high-consequence domains.

Limitations and Future Directions

CSS inherits limitations from the regex-based mutation process, which may produce semantically inert or medically incoherent counterfactuals. The strict pre-registration design defends against metric leak, but cannot adjudicate cases of justified model refusal when the input is nonsensical or inconsistent—a scenario scored as unresponsive under current rules. Future work will focus on catalog refinement, medical vetting, semantic and propagation audits for counterfactual generation, more diverse clinical domains, and larger validation cohorts including additional models (e.g., Google Gemini and others). The metric structure motivates future reinforcement learning experiments employing CSS as a dense shaping reward.

Conclusion

This work establishes that input-responsiveness is a distinct and overlooked capability axis for clinical LLM evaluation. The Causal Sensitivity Score effectively surfaces both performance heterogeneity not observable via coverage metrics and identifies critical, domain-specific safety blind spots, under both single-shot and agentic tool-using conditions. These findings motivate broader incorporation of interventional, pre-registered counterfactual metrics for agent evaluation in safety-critical AI deployment, and challenge the sufficiency of consensus-based rubrics for attesting LLM clinical utility.

(Figure 8)

Figure 8: Summary visual showing the effect of tool use on CSS score distributions; tool use closes some but not all gaps exposed by interventional metrics.

(Figure 9)

Figure 9: LLM-vs-human majority scatter plot for validation tuples, demonstrating the population-level reliability of CSS as an aggregate metric.

---

For comprehensive details and attributions, see "Counterfactual Evaluation Reveals Hidden Capability Profiles in Clinical LLMs and Agents" (2605.30590).