Rethinking XAI Evaluation: A Human-Centered Audit of Shapley Benchmarks in High-Stakes Settings

Abstract: Shapley values are a cornerstone of explainable AI, yet their proliferation into competing formulations has created a fragmented landscape with little consensus on practical deployment. While theoretical differences are well-documented, evaluation remains reliant on quantitative proxies whose alignment with human utility is unverified. In this work, we use a unified amortized framework to isolate semantic differences between eight Shapley variants under the low-latency constraints of operational risk workflows. We conduct a large-scale empirical evaluation across four risk datasets and a realistic fraud-detection environment involving professional analysts and 3,735 case reviews. Our results reveal a fundamental misalignment: standard quantitative metrics, such as sparsity and faithfulness, are decoupled from human-perceived clarity and decision utility. Furthermore, while no formulation improved objective analyst performance, explanations consistently increased decision confidence, signaling a critical risk of automation bias in high-stakes settings. These findings suggest that current evaluation proxies are insufficient for predicting downstream human impact, and we provide evidence-based guidance for selecting formulations and metrics in operational decision systems.

• The paper audits eight Shapley formulations across five risk datasets, two model classes, 37 participants, and 3,735 case reviews using amortized surrogates to isolate semantic effects from implementation noise.
• Human evaluations show that no formulation improves analyst accuracy or review time, while conditional, counterfactual, and mean-baseline explanations significantly increase confidence without corresponding performance gains.
• Quantitative metrics fail to predict human outcomes: joint-marginal explanations improve perceived clarity, whereas filtered conditional explanations score highly on offline benchmarks but are confusing to analysts, underscoring the need for behavioral XAI evaluation.

Motivation and problem statement

Shapley values have become the de facto standard for local feature attribution in tabular machine learning, yet the framework has fragmented into competing formulations that differ in how they operationalize feature absence. The authors of this paper observe that, although the theoretical distinctions among these variants are well documented, XAI evaluation remains dominated by quantitative proxies—faithfulness, sparsity, stability—whose alignment with human decision utility has never been systematically verified. This gap matters acutely in high-stakes, low-latency domains such as fraud detection, credit assessment, and healthcare, where model predictions are reviewed by human analysts under time and regulatory constraints, and where explanations are treated as a compliance and accountability requirement rather than an empirical guarantee.

The paper's central contribution is to treat XAI evaluation itself as an object of scientific study. The authors audit whether standard quantitative benchmarks predict human-perceived clarity, decision confidence, and objective decision quality, using Shapley formulations as a controlled experimental variable. The study is large by human-centered XAI standards: 8 Shapley variants, 5 risk datasets, 2 model classes, 37 participants (including 5 professional fraud analysts), and 3,735 recorded case reviews, with the interaction data released publicly.

A unified framework for isolating semantic effects

A recurring confound in prior Shapley comparisons is that algorithmic implementation differences (KernelSHAP sampling noise, TreeSHAP heuristics, per-instance optimization) mask the semantic differences among value functions. The authors address this by using amortized surrogates: a parametric network $\hat{\phi}_\theta(\boldsymbol{x})$ is trained to approximate each Shapley definition by minimizing the kernel-weighted coalition loss over the data distribution, subject to the efficiency constraint. Inference reduces to a single forward pass, satisfying millisecond-level SLAs while ensuring that all eight formulations—fixed baselines (zero, mean), uniform, marginal, joint-marginal, conditional, search-counterfactual, and filtered conditional—are evaluated under identical computational conditions.

The authors validate this control empirically: amortized attributions achieve low attribution error and high Recall@$k$ against high-sample KernelSHAP ground truths tailored to each definition. Consequently, observed differences across formulations can be attributed to the semantics of the value function rather than approximation artifacts. This is an important methodological point: it establishes that any divergence in downstream behavior is a property of the definition itself.

Quantitative benchmark results

The quantitative evaluation covers deletion/insertion AUC (faithfulness), perturbation sensitivity, counterfactual contrastivity, and sparsity, aggregated across dataset–model pairs with bootstrapped standard errors. Three structural findings emerge:

• No formulation dominates. High sparsity and contrastivity improve insertion AUC but increase perturbation sensitivity; stable, low-sensitivity formulations produce denser, less selective explanations. The landscape exhibits systematic trade-offs rather than a Pareto-optimal choice.
• Formulations cluster into families. Cross-agreement analysis shows that marginal and joint-marginal variants align closely and behave in a balanced manner; conditional Shapley produces dense, sensitive attributions that reflect data correlations rather than model behavior; counterfactual variants are the least stable.
• Fixed baselines are inconsistent. The zero baseline achieves the best deletion AUC and Recall@1 (0.90) but is unaligned with empirical formulations; the mean baseline performs poorly on both stability and selectivity.

Notably, the filtered conditional variant maximizes sparsity, contrastivity, and insertion AUC (0.95) yet is later rated as confusing by analysts—a first signal that quantitative superiority does not translate into human utility.

Human-in-the-loop results

The behavioral study uses a blinded, randomized within-subjects design with a no-explanation control, a fixed review interface (score, attribution bar chart, reason codes, data explorer), and mixed-effects models controlling for case difficulty, model entropy, prediction error, analyst experience, and self-reported expertise.

The headline result is a sharp asymmetry: no Shapley formulation reliably improves objective analyst performance—neither decision accuracy nor decision time—while several formulations significantly inflate self-reported confidence. Conditional (odds ratio 1.58), counterfactual (1.55), and mean-baseline (1.49) formulations substantially increase confidence relative to the no-explanation baseline, with no commensurate accuracy gain. The authors interpret this directly as a deployment risk of automation bias: current evaluation practices may select explanations that encourage over-reliance without improving decision quality.

Perceived clarity, by contrast, is genuinely sensitive to formulation choice. Joint-marginal explanations significantly improve clarity ratings (odds ratio 1.52), while the zero baseline, uniform, and filtered conditional variants are consistently rated as confusing. The authors suggest that the joint-marginal variant's "true-to-the-model" independence logic may be easier for humans to parse than distributionally realistic alternatives. Clarity and confidence remain bounded by case-level difficulty signals—model entropy strongly suppresses confidence (odds ratio 0.44) and score error suppresses accuracy (0.19)—so explanations modulate perception but do not override task difficulty.

Misalignment between proxies and human outcomes

The synthesis stage regresses perceived clarity and confidence on the quantitative explanation properties, with full confounder controls. The result is a fundamental decoupling: standard metrics show no positive association with either clarity or confidence, and sparsity is negatively associated with confidence. This directly contradicts the widely assumed "less is more" hypothesis—that concentrated, low-dimensional explanations enhance intelligibility—and implies that offline proxy benchmarks, while useful for debugging model faithfulness, cannot substitute for behavioral evaluation in operational settings. The practical guidance offered follows from this: practitioners should treat deletion AUC, sensitivity, and related metrics as model-faithfulness debugging tools, prioritize empirical formulations (marginal, joint-marginal) for clarity, and be wary of conditional and counterfactual variants that inflate confidence without accuracy gains.

Limitations and open questions

The authors are explicit about scope. The audit covers low-latency tabular risk models; results may differ in vision or language domains where feature semantics and attribution granularity differ. The experiments are conducted in controlled, single-session settings and cannot capture longer-term effects such as analyst learning, adaptation, or shifts in institutional decision norms—effects that could alter the automation-bias dynamics observed here. Only 5 of 37 participants are professional analysts, so the generalization of confidence-inflation to expert populations rests on a modest sample, though the authors note that high ML and Shapley familiarity across the pool suggests the effect is not mere user naivety. Causal and generative in-manifold Shapley variants are excluded on computational grounds, leaving open whether they would behave differently with human reviewers. Finally, the study establishes that proxies fail to predict human outcomes but does not propose a replacement metric; identifying behaviorally grounded evaluation criteria that remain practical at production scale remains unresolved.

Conclusion

This paper provides a controlled, large-scale empirical demonstration that the quantitative benchmarks governing XAI evaluation are decoupled from the outcomes they are implicitly assumed to serve. Under strictly matched data, models, interfaces, and computational budgets, eight Shapley formulations produce systematic differences in quantitative properties, yet none improves analyst accuracy or speed, while several significantly inflate decision confidence—a concrete automation-bias risk in high-stakes workflows. The released corpus of 3,735 human–AI interaction measurements, together with the amortized evaluation framework, offers the community both the evidence and the infrastructure to move toward behaviorally grounded XAI evaluation, and makes clear that proxy metrics cannot stand in for user studies until their alignment with human utility is demonstrated.