Explainability & Sensitivity Analysis

• Explainability and sensitivity analysis approaches are techniques that quantify how variations in model inputs affect outputs, enhancing model transparency.
• These methods encompass global metrics like Sobol’ indices and local surrogates such as LIME and SHAP, enabling precise feature attribution and robustness evaluation.
• Robust performance relies on smoothing, benchmarking, and uncertainty quantification, ensuring reliable explanations in safety-critical and complex application domains.

Explainability and Sensitivity Analysis Approaches

Explainability and sensitivity analysis in machine learning encompass a broad class of techniques for quantifying, attributing, and evaluating the influence of model inputs on outputs, with the goal of generating actionable and trustworthy explanations for complex models. These methods are central to eXplainable Artificial Intelligence (XAI), as they provide insight into black-box models’ decision pathways, support model auditability, enable feature selection, and quantify robustness with respect to data or parameter perturbations. The landscape spans classical global sensitivity metrics, local surrogate-based explanations, derivative-based approaches, and causal/counterfactual frameworks.

1. Mathematical Foundations of Sensitivity and Explainability

Sensitivity analysis formalizes how variation in inputs propagates to outputs. For a function $f: \mathbb{R}^d \rightarrow \mathbb{R}$, local (pointwise) sensitivity is quantified by the partial derivative $\frac{\partial f}{\partial x_j}$, while global (distributional) measures capture average, variance-based, or distribution-shifting effects. Key metrics include:

• Variance-based indices (Sobol’): Partition $\mathrm{Var}[f(X)]$ among inputs via first-order $$S_j = \mathrm{Var}_{x_j}(\mathbb{E}_{x_{-j}}[f])/\mathrm{Var}(f(X))$$ and total-order $S_{T_j}$ indices. These are typically estimated via Monte Carlo or surrogate modeling (Palar et al., 12 Dec 2025, Schuler, 6 Aug 2025, Scholbeck et al., 2023).
• Derivative-based measures (DGSM): Compute $\mu_j = \mathbb{E}[(\partial f/\partial x_j)^2]$ to proxy total effect, especially for differentiable models (Scholbeck et al., 2023, Duan et al., 2023).
• Moment-independent indices (Borgonovo’s $\Delta$): Evaluate $L^1$-distance between unconditional and conditional output PDFs: $$\Delta_j = \mathbb{E}_{x_j}\left[\int |f_Y(y) - f_{Y|x_j}(y)| dy\right]$$ (Carlo et al., 2024, Scholbeck et al., 2023).
• Shapley value-based decompositions: Attribute output variance or function expectation to input feature coalitions, satisfying desirable axioms of efficiency and symmetry (Schuler, 6 Aug 2025, Duan et al., 2023).

These global metrics lay the foundation for both global feature-importance rankings and process-robustness analysis.

2. Model-Agnostic and Model-Specific Explainability Techniques

Several taxonomic axes delimit the explainability toolkit:

Intrinsic (interpretable) models: Models such as decision trees, sparse linear models, and rule lists offer built-in global explanations via inspection of learned parameters or rules (Anton et al., 2022).

Post-hoc model-agnostic methods:

• LIME (Local Interpretable Model-agnostic Explanations): Fits a local surrogate (typically a sparse linear model) to the black-box’s decision surface in a neighborhood of $x$ by minimizing a locality-weighted loss $L(f, g, \pi_x) + \Omega(g)$, generating local attributions (Kantz et al., 2024, Rizvi et al., 18 Apr 2025).
• SHAP (SHapley Additive exPlanations): For input $x$ and model $f$, computes Shapley value $\phi_i$ that satisfies local accuracy, missingness, and consistency, commonly approximated with Kernel or Tree SHAP for computational tractability (Kantz et al., 2024, Schuler, 6 Aug 2025, Rizvi et al., 18 Apr 2025).

Gradient and effect-based methods:

• Gradient saliency: $\nabla_x f(x)$ provides a local linear sensitivity map, directly interpretable as the effect of infinitesimal input perturbation (Horel et al., 2018, Kantz et al., 2024, Bocchi et al., 12 Mar 2025).
• SmoothGrad and effect-smoothing: Average gradients over a local neighborhood to mitigate variance, improving stability for highly nonlinear or noisy models (Kantz et al., 2024).
• ALE (Averaged Local Effects): Integrates and averages local partial derivatives across the conditional data distribution, offering bias-reduced, model-agnostic feature effects (Kantz et al., 2024, Scholbeck et al., 2023).

Perturbation-based and surrogate frameworks:

• Perturbation approaches analyze effect on output of masking/occluding inputs or input regions, as in LIME for textual/audio data or occlusion maps for images (Mishra et al., 2020, Chatterjee et al., 2021).

A summary of method types and their mathematical cores is provided below:

Approach	Core Formula/Mechanism	Sensitivity Principle
Sobol’ indices	$$\mathrm{Var}_{x_j}(\mathbb{E}_{x_{-j}}[f])/\mathrm{Var}(f)$$	Variance partitioning / ANOVA
LIME	Local surrogate $g^* = \arg\min_{g\in G} L(f,g,\pi_x)+\Omega(g)$	Local finite-difference, surrogate fit
SHAP	$$\phi_j = \mathbb{E}_S [f(x_{S \cup \{j\}}) - f(x_S)]$$	Marginal contribution, game theory
Gradient/Saliency	$|\partial f/\partial x_j(x)|$	Local linear effect
ALE	$$ALE_j(x) = \int E[\partial f/\partial x_j | x_j = u] du$$	Conditional average derivative
Delta/δ-XAI	$$\Delta_j = \mathbb{E}_{x_j}[ \int | f_Y(y) - f_{Y|x_j}(y)| dy]$$	PDF shift, moment-independent
Counterfactual	$$\xi_Y(\vee_k W_k) = \mathrm{Var}[f(W)-f(W’_{\mathcal{S}}, W_{-\mathcal{S}})]/(2\,\mathrm{Var}[f(W)])$$	Causal, interventional variance

(Kantz et al., 2024, Schuler, 6 Aug 2025, Horel et al., 2018, Scholbeck et al., 2023, Carlo et al., 2024, Gao et al., 2024)

3. Robustness, Correctness, and Fidelity of XAI and Sensitivity Methods

Recent comparative studies highlight the tight coupling between model fit and explanation fidelity. In industrial process modeling, explicit scoring protocols benchmark XAI methods against automatic differentiation-based ground-truth sensitivities:

• Scoring methodology: For each sample $i$, compare normalized, scaled XAI output $W_i$ to ground-truth gradient $w^*_i$ via per-sample Brier-style error $$s_i=\frac{1}{d}\sum_{j=1}^d (\hat{w}_{i,j}-\hat{w}^*_{i,j})^2$$; final score $S=1-(1/n)\sum_i s_i$ quantifies faithfulness, with $S=1$ indicating perfect agreement (Kantz et al., 2024).
• Findings: Effect-based explainers (smoothed gradient, ALE) achieve high $S$ only when model $R^2$ is high and drop sharply with increased noise. Additive XAI (LIME/SHAP) generally underperform on sensitivity recovery ($S\leq 0.4$ even in low-noise settings) due to their construction not directly approximating $\partial f/\partial x$.
• Robustness requirements: Smoothing (e.g., cohort-based SG, ALE) is critical for stable gradients in noisy or high-variance domains. Uncertainty quantification, such as via bootstrapping, becomes essential in cost- or safety-critical applications.

A plausible implication is that reliance on gradient/effect-based XAI is justified primarily when model fit to the true process is demonstrably adequate (Kantz et al., 2024).

4. Novel Sensitivity and Explainability Indices

Several recent approaches extend and generalize classical measures:

• Metric-space α-curves: Instead of a single norm, $\alpha$-curves $$M_\alpha = (\frac{1}{N}\sum_{i=1}^N |(\partial f/\partial x_j)(x_i)|^\alpha)^{1/\alpha}$$ chart the spectrum from mean, RMS, to worst-case (max-norm) sensitivity, identifying features with localized or global impact (Pizarroso et al., 2023). Empirical studies show that rare, high-sensitivity regions missed by RMS methods are captured at large $\alpha$.
• Derivative-based Shapley values (DerSHAP): Attribute global importance as $$\phi_j = \mathbb{E}[(\partial f_j)^2] + \frac{1}{2}\sum_{k\neq j} |\mathbb{E}[\partial f_j \partial f_k]|$$, merging variance and covariance (feature-interaction) structure in polynomial computational complexity (Duan et al., 2023).
• δ-XAI: Leverages the Borgonovo $\Delta$ index for local explanations by fixing the value $x_j^*$ and computing the PDF shift at the observed output $y^*$. The normalized difference $$\delta_j(x^*) = \frac{|d_j(x^*)|}{\sum_k |d_k(x^*)|}$$ gives local feature rankings; this approach is robust to feature correlation and highlights outlier or dominant features more clearly than Shapley, especially in presence of distributional skew (Carlo et al., 2024).
• ICE-based global sensitivity: Aggregates spread and correlation of Individual Conditional Expectation (ICE) curves across data instances, quantifying not just global effect but also interaction-induced trend modifications. Specifically, mean and standard deviation of ICE-curve variances, and ICE-vs-PDP correlation, discriminate between pure mean effects and heterogeneous or interaction-modified effects (Palar et al., 12 Dec 2025).

These methods complement classical XAI by probing second-order effects, rare-impact phenomena, PDF shape changes, and the context-dependence of feature relevance.

5. Sensitivity Analysis in Practice: Benchmarking, Visual Tools, and Stability

The applications and benchmarking of explainability and sensitivity techniques span systems biology, vision, audio, and engineering design:

• Visual and GUI frameworks (e.g., SAInT, TorchEsegeta): Enable global Sobol’ and local LIME/SHAP explanations, coupled with feature-pruning workflows and real-time diagnosis (Schuler, 6 Aug 2025, Chatterjee et al., 2021).
• Quantitative validation metrics: Metrics such as infidelity (expected squared difference between attributed and true function change for perturbations) and input-noise sensitivity are standard for comparing explanation map fidelity; low values indicate both faithfulness and local stability (Chatterjee et al., 2021). Sensitivity tests using model component randomization (layer, embedding, etc.) and similarity metrics (SSIM, Spearman correlation, JSD) form necessary sanity checks for any explainer (Sriram et al., 2023).
• Sensitivity in sequential, vision, and mixed-modality domains: Specialized explainers such as LeGrad compute gradients w.r.t. internal attention maps for ViTs, aggregating signal across layers to yield high spatial fidelity and robustness to perturbation (Bousselham et al., 2024). In machine listening, the reliability of local surrogates is shown to depend not only on perturbation strategy but also on the stability and alignment with domain ground truth (Mishra et al., 2020).
• Counterfactual and causal extensions: Counterfactual explainability generalizes association-based sensitivity indices to account for causal structures, interventions, and interaction explanations, notably via an “explanation algebra” that subsumes main and higher-order effects (Gao et al., 2024). This extends the reach of sensitivity analysis to settings with dependent features and explicitly modeled interventions.

6. Domain-Specific Insights and Best Practices

• Process modeling and industrial applications: When the goal is understanding “how does a small change in input $j$ change the prediction?”, effect-based XAI that approximates $\partial f/\partial x_j$ (with smoothing) is preferable. If high-fidelity simulators exist, their gradients serve as gold standards for offline benchmarking of XAI methods under varying noise (Kantz et al., 2024).
• Feature selection and model refinement: By combining global sensitivity indices (e.g., Sobol’, ICE, or SHAP) with iterative feature pruning and local explanation audits, practitioners can improve model performance, reduce complexity, and enhance interpretability (Schuler, 6 Aug 2025, Palar et al., 12 Dec 2025).
• Regulatory and safety-critical contexts: Explanations must be paired with uncertainty quantification and stability analysis (e.g., bootstrapped gradient estimates, repetition under noisy data), as unhinged explanations in unstable or misfit models are unreliable and may mislead end users (Kantz et al., 2024, Sriram et al., 2023).
• Interpreting mixed and highly variable data: For cases with real-world data heterogeneity or complex interaction structure (e.g., engineering design, genomics), averaging-based (e.g., PDP) explainability may underestimate importance, while ICE/SHAP/delta-based approaches are better suited to expose trend changes and context-sensitive effects (Palar et al., 12 Dec 2025, Carlo et al., 2024).

7. Future Directions and Unifying Perspectives

• Unified frameworks: Interpreting explainability as sensitivity analysis aligns the XAI literature with decades of work in uncertainty quantification, enabling the principled transfer of advanced global (e.g., variogram, elementary effects), local, and screening strategies to machine learning (Scholbeck et al., 2023, Anton et al., 2022).
• Feature dependency and causality: There is an ongoing effort to develop sensitivity indices and XAI tools robust to feature correlation, exploiting conditional expectations, high-dimensional model representations, and causal structure to avoid misattribution (Scholbeck et al., 2023, Gao et al., 2024).
• Tooling and reproducibility: Growing ecosystem of open-source sensitivity-analysis toolkits (e.g., sensitivity, sensobol, neuralSens) is being integrated into XAI pipelines, emphasizing scalability, robustness, and convergence diagnostics (Scholbeck et al., 2023). Benchmarking explainers under the SA lens, particularly w.r.t. convergence and reproducibility, is a recommended best practice.

In summary, explainability and sensitivity analysis approaches constitute a mature, interconnected discipline within XAI that is characterized by rigorous mathematical underpinnings, empirical fidelity benchmarking, and broad applicability across model classes and domains. Advances in effect quantification, robust benchmarking, and causal/interaction-sensitive methods continue to expand both their theoretical expressivity and practical utility.

References:

(Kantz et al., 2024, Schuler, 6 Aug 2025, Horel et al., 2018, Duan et al., 2023, Pizarroso et al., 2023, Carlo et al., 2024, Bocchi et al., 12 Mar 2025, Gao et al., 2024, Palar et al., 12 Dec 2025, Chatterjee et al., 2021, Sriram et al., 2023, Anton et al., 2022, Rizvi et al., 18 Apr 2025, Mishra et al., 2020, Scholbeck et al., 2023, Bousselham et al., 2024, Jaroudi et al., 2022)