Causal Sensitivity Score (CSS): A Robust Metric

• Causal Sensitivity Score (CSS) is a scalar metric that measures model responsiveness by quantifying how outputs change under controlled counterfactual interventions.
• It operationalizes sensitivity by comparing baseline and mutated recommendations using a pre-registered scoring system, effectively evaluating clinical models’ directional updates.
• CSS also serves as a robustness threshold in causal analyses, summarizing the minimum strength of hidden confounding needed to overturn causal conclusions.

Causal Sensitivity Score (CSS) denotes a family of scalar quantities that summarize how a model, estimand, or substantive causal conclusion changes under intervention or under bounded departures from ignorability. The term is introduced explicitly as a pre-registered interventional metric for clinical LLMs and agents, where it measures whether recommendations update in the clinically correct direction after controlled counterfactual mutations of oncology cases (Turk, 28 May 2026). Closely related CSS-like constructions appear in causal sensitivity analysis for observational studies, where the score is the minimal sensitivity parameter $\Lambda$ or $\Gamma$ needed for hidden confounding to overturn a conclusion, or the width of the resulting partial-identification bounds (Jesson et al., 2022, Frauen et al., 2023, Frauen et al., 2023, Zhao, 2017).

1. Explicit definition in clinical LLM and agent evaluation

In the explicit named usage, CSS measures causal responsiveness of a clinical model to clinically meaningful input changes. For each model $m$, intervention $i$, and case $c$ where $i$ applies, baseline recommendations are generated from the unmodified packet and intervened recommendations from the mutated packet. A judge LLM receives both recommendations, the pre-registered expected change, and the scoring rule, and emits a score in $\{0.0, 0.5, 1.0\}$ for no change, acknowledged but unchanged, or updated correctly. CSS is then the mean score across valid $(c,i)$ tuples: $$\text{CSS}(m) = \frac{1}{N_m} \sum_{(c,i)\in \mathcal{T}_m} s(m,c,i),$$ with $s(m,c,i)\in\{0.0,0.5,1.0\}$. Family-wise CSS is defined analogously by restricting the average to tuples in a given intervention family $\Gamma$0 (Turk, 28 May 2026).

The three-level scoring scale is intervention-specific but pre-registered. In a HER2-positive to HER2-negative flip, a score of $\Gamma$1 requires all HER2-targeted recommendations to be dropped; $\Gamma$2 is assigned when some are dropped or the rationale hedges without fully updating the treatment list; $\Gamma$3 corresponds to unchanged HER2-targeted recommendations. In a surgery-status intervention, $\Gamma$4 requires treatment timing to shift as specified, $\Gamma$5 requires only rationale-level acknowledgement, and $\Gamma$6 corresponds to no change. The defining property of the metric is therefore not textual similarity to a reference answer, but directional updating under intervention (Turk, 28 May 2026).

This operationalization makes CSS an interventional metric rather than a static agreement score. It evaluates whether the input–output mapping has the clinically expected local behavior under counterfactual perturbation, rather than whether a single output overlaps with an expert recommendation set.

2. Counterfactual design and intervention families

The clinical CSS protocol is built around 224 real oncology tumor-board cases and 12 pre-registered mutations grouped into five families. Family A applies biomarker flips such as HER2, ER, or PD-L1 replacement; Family B injects prior-line progression or failed therapy; Family C removes biomarker mentions; Family D toggles resection history in procedure sections; and Family E perturbs stage, for example by inserting metastasis. Before no-op removal, eligible tuples per family are $\Gamma$7, $\Gamma$8, $\Gamma$9, $m$0, and $m$1. After dropping 73 regex no-ops, the final scoring set contains 789 mutated tuples (Turk, 28 May 2026).

Each intervention is specified in a pre-registered YAML catalog with an applicability filter, mutation rule, expected output change, scoring rule in $m$2, and family label. Counterfactual cases are constructed by applying a single replace, delete, or insert mutation while otherwise leaving the packet unchanged. If the required regex pattern does not match, the tuple is dropped rather than scored. Correct-direction behavior is also specified in advance: biomarker flips require targeted agents to disappear or appear as appropriate; prior-failure interventions require failed regimens to be dropped from future-line recommendations; biomarker stripping requires hedging or more generic treatment proposals; surgery-status mutations require treatment timing to move between adjuvant and primary or definitive framing; stage perturbations require a shift toward metastatic rather than curative-intent recommendations (Turk, 28 May 2026).

The pre-registration constraint is methodologically central. It fixes the intervention catalog and the scoring semantics before model execution, thereby limiting post hoc metric redesign. In this sense, CSS inherits some of the logic of classical causal sensitivity analysis—explicitly define an admissible perturbation class, then measure how conclusions behave over that class—but applies it directly to model behavior rather than to an identified causal estimand.

3. Computation, rankings, and empirical profiles

In the reported benchmark, six frontier models from three labs were evaluated in single-shot inference. Aggregate CSS scores are $m$3 for grok-4.20-reasoning, $m$4 for gpt-5.4-mini, $m$5 for claude-opus-4-7, $m$6 for gpt-5, $m$7 for claude-sonnet-4-6, and $m$8 for gpt-5.4. Family-wise scores reveal heterogeneous capability profiles: Family B, prior treatment failure, is the strongest family across models, while Family D, surgery status, is uniformly poor, with the best model reaching only $m$9 and the worst $i$0. The score distributions further show that gpt-5.4 has $i$1 wrong, $i$2 partial, and $i$3 correct tuple-level judgments, whereas grok-4.20 has $i$4 wrong, $i$5 partial, and $i$6 correct (Turk, 28 May 2026).

The clinical study also transfers CSS to tool-using agents in a ReAct-style setting focused on surgery-status interventions. Tool use improves Family D CSS for five of six models, with gains ranging from $i$7 to $i$8 percentage points, but one model remains unchanged at $i$9 despite similar tool-call counts and similar retrieval of the procedures section. The paper interprets this as a structural responsiveness deficit: the model accesses the relevant chart sections yet still fails to update the recommendation appropriately (Turk, 28 May 2026).

Validation is conducted at two levels. Cross-judge replication with a uniform Opus judge preserves the rank ordering exactly, with Spearman $c$0 between the default and replicated rankings. Three medical-professional annotators independently score 100 tuples; human–human Cohen’s $c$1 ranges from $c$2 to $c$3, while LLM judge versus human-majority agreement is $c$4. Aggregate family-wise means remain close between LLM and human scoring, including Family D, where the LLM mean is $c$5 and the human mean $c$6 (Turk, 28 May 2026).

Taken together, these results establish CSS as a diagnostic for hidden capability profiles. Models that appear similar under static evaluation can differ sharply in whether they react to altered causal signals, and the decomposition by intervention family localizes where that responsiveness succeeds or fails.

4. Relation to coverage metrics and counterfactual evaluation

The principal comparison metric in the clinical study is the Consensus Match Score (CMS), a coverage-based weighted recall metric: $c$7 CMS evaluates how well a model’s recommendations overlap with oncologist consensus for a static case. CSS instead evaluates whether recommendations change in the correct direction after intervention on clinically meaningful case attributes (Turk, 28 May 2026).

The two metrics produce nearly opposite rankings. In the reported benchmark, all six models change rank between CMS and CSS: the CMS-worst model becomes CSS-best, and one upper-mid CMS model ranks last on CSS. The observed CMS range is $c$8 to $c$9, whereas CSS spans $i$0 to $i$1. The rank correlation is negative, with Spearman $i$2, although the study notes that $i$3 makes this test underpowered (Turk, 28 May 2026).

This divergence clarifies the conceptual status of CSS. CMS is a static output-coverage metric; CSS is a counterfactual responsiveness metric. A model can achieve strong weighted recall by recommending broadly plausible regimens while remaining insensitive to whether a biomarker was removed, a therapy was declared to have failed, or surgery was no longer documented. CSS is designed to expose exactly that failure mode.

5. CSS as a robustness threshold in causal sensitivity analysis

Outside clinical LLM evaluation, the same label naturally attaches to tipping-point summaries of partial-identification bounds. In continuous-treatment sensitivity analysis under the Continuous Treatment-Effect Marginal Sensitivity Model (CMSM), a scalar sensitivity parameter $i$4 bounds the density ratio between the true treatment assignment with hidden confounding and the observed generalized propensity score. A natural CSS is the minimal $i$5 required for hidden confounding to materially change a causal conclusion. For a sign conclusion on an average effect between $i$6 and $i$7, the proposed score is

$i$8

and for a threshold $i$9,

$\{0.0, 0.5, 1.0\}$0

If CSS is close to $\{0.0, 0.5, 1.0\}$1, small violations of ignorability can overturn the conclusion; if it is large, stronger hidden confounding is required (Jesson et al., 2022).

A closely related formulation appears in generalized neural sensitivity analysis. NeuralCSA represents unobserved confounding as a latent distribution shift constrained by a sensitivity model and computes sharp or tight bounds on a general causal query $\{0.0, 0.5, 1.0\}$2. In that framework, one natural CSS is the minimal $\{0.0, 0.5, 1.0\}$3 such that the partially identified interval includes $\{0.0, 0.5, 1.0\}$4: $\{0.0, 0.5, 1.0\}$5 The same paper also identifies bound width,

$\{0.0, 0.5, 1.0\}$6

as an alternative scalar summary of sensitivity at a fixed $\{0.0, 0.5, 1.0\}$7 (Frauen et al., 2023).

The sharp-bounds framework based on the generalized marginal sensitivity model (GMSM) makes the same logic explicit for discrete, continuous, and time-varying treatments, as well as mediation and path-specific effects. There, CSS is again the smallest $\{0.0, 0.5, 1.0\}$8 for which $\{0.0, 0.5, 1.0\}$9 enters the sharp identified set $(c,i)$0 derived from worst-case latent distribution shifts. Because the bounds are sharp, the resulting score is calibrated to the stated sensitivity model rather than to a conservative approximation (Frauen et al., 2023).

In pair-matched observational studies, Zhao’s sensitivity value provides an older and highly specific precursor. Under Rosenbaum’s sensitivity model, the truncated sensitivity value is

$(c,i)$1

with the untruncated version

$(c,i)$2

This is defined as the minimum strength of unmeasured confounding needed to change the qualitative conclusions of a naive analysis assuming no unmeasured confounder. In CSS terminology, it is the tipping-point robustness score for a matched-pair design under a Rosenbaum-type odds-ratio model (Zhao, 2017).

Across these formulations, a common structure recurs: define an admissible perturbation model for unobserved confounding, derive lower and upper bounds on the estimand of interest, then summarize robustness either by the smallest sensitivity parameter that invalidates a claim or by the width of the bound interval at a fixed parameter value.

6. Related operationalizations, limitations, and broader significance

The CSS label is also natural in other domains where intervention-induced change is the object of interest. In sentiment-analysis bias auditing, controlled interventions on gender and race markers produce scalar summaries such as the Weighted Rejection Score,

$(c,i)$3

which aggregates t-test rejections across protected-attribute comparisons, and the Deconfounding Impact Estimate,

$(c,i)$4

That work does not use the term CSS, but explicitly presents WRS and DIE as quantities that can be interpreted as causal sensitivity scores at instance, group, or system level (Lakkaraju et al., 2023).

In data management, an analogous renaming is proposed for the generalized causal-effect score of tuples on query answers in probabilistic databases: $(c,i)$5 For Boolean queries this reduces to the difference in query truth probability under tuple inclusion versus exclusion; under a uniform tuple-independent embedding of a deterministic database, it coincides with the Banzhaf power index. The paper explicitly notes that this quantity can be interpreted as a Causal Sensitivity Score because it measures sensitivity of the expected query answer to interventions on tuples (Azua et al., 4 Feb 2025).

These variants also delimit the concept’s limitations. In the clinical LLM setting, CSS is a population-level metric: per-row agreement with human annotators is moderate to low for some families, regex-based mutations can create semantic no-ops, incomplete propagation, or medical incoherence, and the pre-registered scoring rule can assign $(c,i)$6 even when a model refuses to update because the mutation is itself incoherent (Turk, 28 May 2026). In hidden-confounding sensitivity analysis, CSS depends on the chosen sensitivity model and scaling of $(c,i)$7 or $(c,i)$8, and it reflects worst-case hidden confounding rather than a model of how confounding actually operates (Jesson et al., 2022, Frauen et al., 2023). In matched-pair studies, the sensitivity value is tied to Rosenbaum’s odds-ratio model and to pair-matched signed-score procedures (Zhao, 2017). In data management, open problems remain beyond monotone queries and beyond the current axiomatization for monotone Boolean queries (Azua et al., 4 Feb 2025).

This suggests that CSS is best understood not as a single universal statistic, but as a recurring design pattern in causal methodology. The pattern has three principal realizations: a direct interventional response score over counterfactual input mutations, a robustness threshold against bounded unobserved confounding, and an expected marginal effect under explicit interventions. What unifies these realizations is the attempt to replace purely correlational or coverage-based evaluation with a scalar summary of how outputs or conclusions move when the underlying causal conditions are changed.