Attribution Disagreement Score (ADS)

• Attribution Disagreement Score (ADS) is a rank-correlation metric that quantifies per-instance divergence between structured model (XGBoost with SHAP) and LLM feature rankings.
• It computes the distance by comparing top-5 feature rankings using Kendall’s τ, assigning missing features a default rank to ensure complete union.
• ADS acts as an external uncertainty signal to detect epistemic misalignment, improving calibration by highlighting when LLM predictions diverge from trusted clinical models.

Attribution Disagreement Score (ADS) is a per-instance, rank-correlation–based distance introduced to quantify cross-model attribution divergence on clinical tabular data. In the reported AKI prediction study, ADS compares the local feature-attribution ranking from a structured model, XGBoost explained with SHAP, against the top-$K$ risk-driving features self-reported by Qwen 2.5 7B Instruct for the same patient encounter. The score is used as both a diagnostic of feature-level reasoning misalignment and an external uncertainty signal for calibration, within a broader argument that LLMs on structured tasks face a cold start problem and can exhibit epistemic blind spots even when they verbalize high confidence (Dasula et al., 17 Jun 2026).

1. Formal definition

ADS is defined in the Methods subsection “Attribution Disagreement Score (ADS)” as a rank-correlation–based distance between two local top-$K$ feature rankings for the same instance: $R_A$ from XGBoost’s SHAP explanation and $R_B$ from the LLM’s self-reported “important features” list. With

$$R_A = (f_1,\dots,f_K), \qquad R_B = (g_1,\dots,g_K),$$

the method first forms the union of features

$\mathcal{U} = R_A \cup R_B.$

Any feature in $\mathcal{U}$ that is missing from one model’s top-$K$ list is assigned rank $K+1$ in that model’s ranking. Kendall’s rank correlation $\tau$ is then computed over the completed rankings, and ADS is defined as

$K$0

Because $K$1, ADS lies in $K$2. An ADS of $K$3 corresponds to perfect agreement, and an ADS of $K$4 corresponds to perfect anti-correlation. The score is defined per instance, here per patient encounter, and the reported values for each prompt condition are mean ADS over the 300-patient evaluation subset. The study also reports Jaccard overlap, Top-1 match rate, and Directional agreement, but these are complementary per-instance attribution metrics rather than part of the ADS definition itself (Dasula et al., 17 Jun 2026).

2. Experimental role and operationalization

The study instantiates ADS in a two-model comparison on serialized clinical tabular data. The reference structured model is an XGBoost classifier on 50 selected clinical features, trained on 8,000 encounters, evaluated on 2,000 test encounters, with a 300-patient subset used for the LLM experiments. For each patient, SHAP provides local per-instance explanations, and the pipeline records the predicted label, predicted probability, and the top-5 SHAP features with their SHAP values. The LLM side uses Qwen 2.5 7B Instruct under four prompting conditions: zero-shot (ZS), zero-shot + SHAP evidence (ZS+S), few-shot (FS), and few-shot + SHAP evidence (FS+S). In every condition, the model must output a binary AKI prediction, a verbalized confidence in $K$5, and a ranked list of top-5 features with directionality (“increase/decrease”) (Dasula et al., 17 Jun 2026).

Operationally, ADS uses only rank information. The XGBoost ranking is derived from the top-5 SHAP features in descending order of SHAP $K$6, and the LLM ranking is taken from the order in its JSON output. Missing features receive the default rank $K$7, after which Kendall’s $K$8 is computed over the union of features. No additional normalization is reported beyond the $K$9 treatment for missing features and the normalization inherent in Kendall’s $R_A$0.

The motivation is explicitly epistemic rather than merely predictive. XGBoost is described as a well-performing, calibrated structured model on the AKI prediction task, with AUROC $R_A$1 and accuracy $R_A$2, and its SHAP attributions are treated as a data-grounded reference for what features matter. ADS therefore serves as a proxy for epistemic misalignment: high ADS indicates that the LLM attends to different factors or orders them differently, while low ADS indicates closer alignment of local feature-level reasoning (Dasula et al., 17 Jun 2026).

3. Behavior across prompting conditions

The reported ADS values change sharply across prompt regimes, and these changes are coupled to large differences in predictive performance. The central pattern is that few-shot examples and SHAP-derived feature evidence act as orthogonal, super-additive interventions.

Condition	Accuracy	ADS
ZS	0.490	1.536
ZS+S	0.520	0.881
FS	0.683	1.395
FS+S	0.753	0.378

In the zero-shot condition, the LLM is at chance, with accuracy $R_A$3, AUROC $R_A$4, and ADS $R_A$5. This is the highest disagreement regime and corresponds to substantial attributional divergence. Adding SHAP evidence in zero-shot reduces ADS from $R_A$6 to $R_A$7, while Jaccard overlap rises from $R_A$8 to $R_A$9 and Top-1 match rises from $R_B$0 to $R_B$1, yet accuracy improves only from $R_B$2 to $R_B$3. The authors interpret this pattern as “shallow adoption”: the LLM starts talking about the same features without materially changing its decisions.

Few-shot prompting without SHAP evidence yields a different profile. ADS decreases only slightly relative to zero-shot, from $R_B$4 to $R_B$5, while Jaccard remains $R_B$6 and Top-1 match is $R_B$7. However, accuracy rises strongly to $R_B$8, with AUROC $R_B$9. The stated interpretation is that few-shot prompting improves “task comprehension”: the model learns the mapping from features to labels, but still focuses on different features than XGBoost.

The combined FS+S condition produces the strongest alignment and the best LLM performance: ADS $$R_A = (f_1,\dots,f_K), \qquad R_B = (g_1,\dots,g_K),$$0, accuracy $$R_A = (f_1,\dots,f_K), \qquad R_B = (g_1,\dots,g_K),$$1, AUROC $$R_A = (f_1,\dots,f_K), \qquad R_B = (g_1,\dots,g_K),$$2, Jaccard $$R_A = (f_1,\dots,f_K), \qquad R_B = (g_1,\dots,g_K),$$3, Top-1 match $$R_A = (f_1,\dots,f_K), \qquad R_B = (g_1,\dots,g_K),$$4, and Directional agreement $$R_A = (f_1,\dots,f_K), \qquad R_B = (g_1,\dots,g_K),$$5. The paper emphasizes super-additivity in ADS reduction: ZS$$R_A = (f_1,\dots,f_K), \qquad R_B = (g_1,\dots,g_K),$$6FS gives $$R_A = (f_1,\dots,f_K), \qquad R_B = (g_1,\dots,g_K),$$7ADS $$R_A = (f_1,\dots,f_K), \qquad R_B = (g_1,\dots,g_K),$$8, ZS$$R_A = (f_1,\dots,f_K), \qquad R_B = (g_1,\dots,g_K),$$9ZS+S gives $\mathcal{U} = R_A \cup R_B.$0ADS $\mathcal{U} = R_A \cup R_B.$1, whereas ZS$\mathcal{U} = R_A \cup R_B.$2FS+S gives $\mathcal{U} = R_A \cup R_B.$3ADS $\mathcal{U} = R_A \cup R_B.$4, which is larger than $\mathcal{U} = R_A \cup R_B.$5. This is presented as evidence that task guidance and structured evidence jointly pull the LLM’s reasoning closer to the structured model’s reasoning (Dasula et al., 17 Jun 2026).

4. ADS as an uncertainty and calibration signal

A central claim of the study is that the LLM’s own verbalized confidence is epistemically vacuous. Across prompt regimes, the model emits a near-constant confidence range of $\mathcal{U} = R_A \cup R_B.$6–$\mathcal{U} = R_A \cup R_B.$7 even when accuracy ranges from $\mathcal{U} = R_A \cup R_B.$8 to $\mathcal{U} = R_A \cup R_B.$9. In the few-shot condition used for calibration analysis, raw LLM confidence is nearly constant at $\mathcal{U}$0, with AUROC $\mathcal{U}$1 for separating correct from incorrect predictions and expected calibration error $\mathcal{U}$2. ADS is introduced as a patient-specific external signal that can replace this uninformative self-report (Dasula et al., 17 Jun 2026).

The cross-model calibrator predicts

$\mathcal{U}$3

from cross-model features: ADS, Jaccard overlap, Directional agreement, Top-1 match, XGBoost confidence, LLM verbalized confidence, and $\mathcal{U}$4. Two calibrators are evaluated: logistic regression and XGBoost + Platt scaling.

Confidence source	AUROC (correct vs incorrect)	ECE
Raw LLM confidence	0.533	0.254
Logistic regression	0.551	0.043
XGBoost + Platt scaling	0.710	0.080

The XGBoost-based calibrator with ADS and the other cross-model features raises correctness-discrimination AUROC to $\mathcal{U}$5 and reduces ECE from $\mathcal{U}$6 to $\mathcal{U}$7, a reduction described as about $\mathcal{U}$8. The paper frames this as replacing uninformative verbalized confidence with patient-specific reliability estimates, without accessing model internals or requiring repeated inference.

Within this framing, ADS functions as an external uncertainty signal. High ADS, especially when XGBoost confidence is high, is treated as evidence that the LLM is relying on features that the structured model deems unimportant or orders differently, and is therefore more likely to be wrong. Low ADS indicates that the LLM is reasoning along similar lines as a high-performing structured model and can be trusted more. This is the mechanism by which ADS is linked to the detection of epistemic blind spots (Dasula et al., 17 Jun 2026).

5. Design choices, assumptions, and limitations

ADS is explicitly relative to a designated reference model. The study assumes that XGBoost is more trustworthy on this AKI prediction task, and that SHAP provides faithful local explanations of XGBoost’s behavior. Consequently, ADS does not measure disagreement with clinical ground truth directly; it measures disagreement with XGBoost+SHAP as a grounding reference. This is a substantive design choice rather than a model-independent property of the data (Dasula et al., 17 Jun 2026).

Several methodological choices follow from that framing. The score uses Kendall’s $\mathcal{U}$9 over top-$K$0 rankings rather than distances on raw attribution magnitudes, with $K$1 fixed for both models. Missing features are assigned rank $K$2, which penalizes disjoint top-$K$3 lists. The LLM attributions are self-reported rather than gradient-based or perturbation-based, and ADS therefore compares SHAP-based local attributions to narrative feature rankings emitted by the LLM. The shared feature universe is controlled by serializing a common set of 50 selected top features into the prompt, and JSON-formatted output is enforced via Jsonformer for parseability.

The limitations are also explicit. The paper does not systematically vary the attribution method for XGBoost, the value of $K$4, or the LLM architecture beyond the four prompt conditions. It notes that SHAP is one of many explanation methods and is not perfect, and that self-reported LLM attributions may reflect narrative preferences rather than causal feature importance. The authors acknowledge faithfulness concerns and note that they are pursuing clinician validation. At the same time, they show that ADS is sensitive to prompting condition: zero-shot, few-shot, and SHAP injection alter ADS dramatically, which they interpret as sensitivity to task alignment and feature guidance rather than numerical instability alone (Dasula et al., 17 Jun 2026).

6. Position within attribution-disagreement research

The ADS formulation in clinical tabular prediction is specific: it is a per-instance scalar built from the disagreement between a structured model’s top-5 SHAP ranking and an LLM’s top-5 self-reported ranking. Related work uses adjacent but non-identical formalisms. “Reckoning with the Disagreement Problem: Explanation Consensus as a Training Objective” does not introduce ADS by name, but formalizes Post hoc Explainer Agreement Regularization and evaluates agreement with Feature Agreement, Rank Agreement, Sign Agreement, Signed Rank Agreement, Rank Correlation, and Pairwise Rank Agreement (Schwarzschild et al., 2023). “Dynamic Top-k Estimation Consolidates Disagreement between Feature Attribution Methods” likewise centers agreement@k and shows that disagreement depends strongly on top-$K$5 selection (Kamp et al., 2023). In explainable news summarization, disagreement is quantified with Feature Agreement, Rank Agreement, Spearman’s rank correlation, and Pairwise Rank Agreement, and regional explanations reduce feature-level disagreement (Aswani et al., 2024).

These adjacent literatures clarify what is distinctive about ADS in the clinical study. First, ADS is cross-model rather than cross-explainer: it compares an LLM with a structured tabular model rather than comparing two explainers on the same predictor. Second, it is used as an uncertainty feature in a calibrator rather than solely as a stability or consensus diagnostic. Third, it privileges rank order over magnitude, which parallels work in explainable data management defining alignment as two scores inducing the same total preorder over tuples (Azua et al., 18 Mar 2025).

A further conceptual caution comes from generative-model attribution research. “The Attribution Contract: Feature Attribution for Generative LLMs” argues that many apparent disagreements arise from different unstated explanatory contracts rather than from attribution algorithms alone (Nguyen, 21 May 2026). A plausible implication is that the clinical ADS is most coherent precisely because its contract is tightly specified: one patient, one structured prediction task, one shared feature vocabulary, one top-$K$6 ranking from SHAP, and one top-$K$7 ranking from the LLM. Within that fixed setting, ADS becomes a concrete operational measure of how aligned the LLM’s stated feature-level reasoning is with a high-performing structured reference, and therefore of how much trust to place in the LLM on a given structured clinical case.