Post-hoc Explanation Methods

• Post-hoc Explanation Methods are techniques that approximate black-box models with interpretable surrogates to provide clear, actionable insights.
• They leverage approaches like feature attribution, counterfactual examples, and local approximations to ensure high fidelity, sensitivity, and robustness.
• Recent innovations focus on scalability, user-knowledge adaptation, and addressing limitations such as explanatory inversion and spurious correlations.

A post-hoc explanation method is any algorithmic procedure designed to interpret or elucidate the predictions of an already-trained, typically black-box, model—such as a neural network or ensemble—without altering or requiring transparency from the underlying model. Post-hoc explanation strategies are central to the interpretability of modern machine learning, where the model’s internal logic is inaccessible or opaque, but informed reasoning about the output is essential for trust, regulatory compliance, and scientific understanding.

1. Formal Foundations and Taxonomy

Post-hoc explainers operate on the premise that a black-box model $F: \mathbb{R}^d \rightarrow \mathbb{R}$ can be queried but not internally inspected. These methods seek to construct an interpretable mapping (surrogate) $p^*$ from a simpler, human-comprehensible family $\mathcal{P}$ such that $p^*$ approximates $F$ locally or globally. Formally, for an input $x$:

$$p^* = \arg\min_{p \in \mathcal{P}} \mathbb{E}[d(F(x'), p(x'))]_{x' \sim N(x)}$$

where $d(\cdot)$ is a fidelity metric and $N(x)$ is a neighborhood of $x$ (Oh, 2024).

Major categories include:

• Feature-attribution (additive) explainers: Assign scores $\{\phi_j(x)\}$ to input features such that $\sum_j \phi_j(x) \approx F(x)$. Examples: SHAP, LIME, Integrated Gradients (Carmichael et al., 2021).
• Surrogate models: Fit a simple interpretable model (e.g., decision tree) to locally or globally approximate $F$ (Oh, 2024).
• Feature-selection explainers: Identify minimal subsets $S(x)$ of features sufficient for prediction, maximizing mutual information $I(X_S; F(X))$ under a size constraint (Camburu et al., 2019).
• Counterfactual explanations: Generate perturbed examples $x'$ with $F(x') \ne F(x)$ that are minimally different from $x$.
• Holistic/descriptor methods: Compute functionals (e.g., global importance, PDP, SAGE) reflecting overall feature or concept impact.

2. Unifying Methodologies: Local Approximation View

A broad unification is achieved by casting many prominent methods as local function approximations (LFA). The general form involves minimizing

$$g^* = \arg\min_{g \in \mathcal{G}} \mathbb{E}_{z \sim \pi_x} [\ell(F(z), g(z))]$$

where $g$ is a linear surrogate, $\pi_x$ defines the neighborhood, and $\ell$ is the loss. Methods correspond to distinct choices of $N(x)$ (perturbation regime), loss, and regularization (Han et al., 2022):

Method	Neighborhood/Perturbation	Loss	Attribution
LIME	Binary masking	Weighted error	$w_i$
KernelSHAP	Binary masking, Shapley kernel	Weighted error	Shapley values $\phi_i$
Occlusion	One-hot masking	Difference	$F(x)-F(x_{\setminus i})$
Vanilla Grad	Infinitesimal additive noise	Gradient match	$\nabla_x F(x)$
Integrated Grad	Path interpolation	Gradient match	$\int_{0}^1 \nabla_x F(x'+\alpha(x-x')) d\alpha$

This framework demonstrates that method-specific differences (e.g., LIME vs. KernelSHAP) are due to perturbation sampling/simulation and loss definitions rather than divergent conceptual goals (Han et al., 2022).

3. Faithfulness, Robustness, and Evaluation Metrics

The principal desiderata for post-hoc explanations are:

• Faithfulness: Degree to which attributions faithfully track the model’s functional dependence. Evaluated using ground-truth additive models when available—calculating $\|a - b\|_2$, cosine distance, nRMSE, and Spearman $\rho$ between attribution error and predictive accuracy (Carmichael et al., 2021), or via insertion/deletion perturbation tests (Seth et al., 5 Feb 2025).
• Sensitivity: Stability of explanations to small input perturbations, estimated via local Lipschitz constants or the change in attributions under Gaussian input noise (Seth et al., 5 Feb 2025).
• Robustness: Invariance of explanations under random or adversarial perturbations, measured using quantities such as mean pixelwise robustness (MPRT) in images (Seth et al., 5 Feb 2025).
• Benchmark reliability: Variability of method rankings across test images quantified using Krippendorff’s $\alpha$. Reliable benchmarks require model training modifications that increase inter-image concordance in method rankings (Gomez et al., 2023).

Recent work has highlighted that, while methods like SHAP and LIME are empirically reliable in low-dimensional, low-interaction regimes, their faithfulness decreases with model complexity and nonlinearity. Models can be globally accurate while explanations grossly misattribute feature importance locally—a potentially dangerous property in high-stakes domains (Carmichael et al., 2021).

4. Key Algorithmic Innovations and Scaling

• SHapley Estimated Explanation (SHEP): Reduces SHAP’s complexity from $O(2^d)$ to $O(dn)$ by using “add” (inject feature $i$ into baseline) and “remove” (replace feature $i$ with baseline value) approximations. Patch-wise aggregation further improves scalability for high-dimensional data, trading off granularity for computational tractability (Chen et al., 3 Apr 2025).
• Span-based and dynamic-$k$ token explanations in NLP: Explainer disagreement at token-level can be largely attributed to systematic linguistic preferences. Agreement increases when attributions are compared at the syntactic span level and when $k$ (number of important tokens) is set dynamically via peak-finding above a mean positive attribution threshold (Kamp et al., 2024).
• User-knowledge adaptation: Explanations can be tailored to user knowledge by augmenting the explanation objective with a compatibility term $C(e, K)$, e.g., penalizing attributions outside the user’s known feature set (KICE), resulting in lower-cost and more user-aligned counterfactuals (Jeyasothy et al., 2022).
• Vision Transformer explanation (TokenTM): Standard attention-based methods are insufficient for ViTs. TokenTM integrates measures of patch vector scaling and alignment (norm and cosine) in addition to attention weights, improving segmentation fidelity and perturbation robustness (Wu et al., 2024).

5. Limitations, Failure Modes, and Theoretical Barriers

Theoretical developments have revealed foundational limitations:

• Non-informativeness in rich hypothesis classes: For large model families (e.g., all differentiable functions, deep decision trees), post-hoc explanations such as raw gradients, SHAP, and counterfactuals prove non-informative—they fail to reduce the Rademacher complexity (function space size) and thus cannot rule out plausible alternative models (Günther et al., 15 Aug 2025).
• Explanatory inversion: Standard post-hoc methods may “justify” outputs post-hoc rather than reflecting the true decision process, especially in the presence of spurious correlations. The Inversion Quantification (IQ) framework explicitly measures reliance on the output (vs. input) and faithfulness, demonstrating that methods such as LIME and SHAP are vulnerable to inversion—correct output attributions can misrepresent internal causality (Tan et al., 11 Apr 2025).
• Detecting unknown spurious signals: Feature attribution, concept activation, and influence ranking methods are ineffective at uncovering unknown, non-salient spurious correlations. They frequently highlight artifact regions even for models not relying on these artifacts, leading to high rates of false alarms and rendering them unreliable in the absence of strong prior hypotheses (Adebayo et al., 2022).
• Additivity vs. sufficiency: Additive attributions do not guarantee that top-ranked features are actually sufficient for the prediction, and vice versa. Feature-selection explainers often rank zero-contribution features highly even in controlled architectures, challenging their trustworthiness (Camburu et al., 2019).

6. Empirical Evaluation Practices and Practical Recommendations

Rigorous evaluation frameworks are now standard:

• Ground-truth additive models: Synthetic models with known per-feature contributions allow objective benchmarking of explainer accuracy and misattribution (Carmichael et al., 2021).
• xai_evals toolkit: Provides standardized metrics (faithfulness, sensitivity, robustness) and pipelines over diverse explainers and data modalities. Model-agnostic methods such as SHAP and LIME typically yield higher faithfulness but lower robustness than gradient-based methods (Seth et al., 5 Feb 2025).
• Application-grounded user studies: Direct measurement of the effect of explanations on real decision tasks reveals that explanations do not always improve human accuracy versus data-only baselines, with strong method-by-task and user-preference variation (Jesus et al., 2021, Jalali et al., 2023).
• User-alignment and adaptive explanations: Explicitly incorporating user knowledge and preferences can yield explanations that simultaneously minimize cognitive burden and maintain fidelity (Jeyasothy et al., 2022).

Best practices include (i) explicit reporting and validation of explanation fidelity, (ii) task-specific explainer selection, (iii) user-aligned presentation (e.g., via span-level or counterfactual examples), (iv) comprehensive benchmarking across diverse metrics, and (v) abstaining from strong interpretive claims in “theory-poor” or high-complexity regimes unless informativeness can be proven (Oh, 2024, Günther et al., 15 Aug 2025).

7. Philosophical and Regulatory Perspectives

The philosophical stance of “Computational Interpretabilism” (CI) reframes post-hoc explainability as a mediated, empirically bounded process whereby scientific knowledge arises not from full transparency but from an iterative cycle of model behavior, explanation, hypothesis, and empirical validation. CI recognizes that even incomplete or imperfect explanations can yield justified insight provided their scope and limitations are documented, and that they are empirically tested. Regulatory frameworks (e.g., GDPR’s "right to explanation," EU AI Act) are increasingly demanding provable fidelity or informativeness, placing new demands on both model structure and explanation delivery (Oh, 2024, Günther et al., 15 Aug 2025).

---

References:

• (Carmichael et al., 2021) A Framework for Evaluating Post Hoc Feature-Additive Explainers
• (Kamp et al., 2024) The Role of Syntactic Span Preferences in Post-Hoc Explanation Disagreement
• (Chen et al., 3 Apr 2025) SHapley Estimated Explanation (SHEP): A Fast Post-Hoc Attribution Method
• (Seth et al., 5 Feb 2025) xai_evals : A Framework for Evaluating Post-Hoc Local Explanation Methods
• (Tan et al., 11 Apr 2025) Are We Merely Justifying Results ex Post Facto? Quantifying Explanatory Inversion in Post-Hoc Model Explanations
• (Oh, 2024) In Defence of Post-hoc Explainability
• (Camburu et al., 2019) Can I Trust the Explainer? Verifying Post-hoc Explanatory Methods
• (Günther et al., 15 Aug 2025) Informative Post-Hoc Explanations Only Exist for Simple Functions
• (Adebayo et al., 2022) Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlation
• (Wu et al., 2024) Token Transformation Matters: Towards Faithful Post-hoc Explanation for Vision Transformer
• (Han et al., 2022) Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanations
• (Jeyasothy et al., 2022) Integrating Prior Knowledge in Post-hoc Explanations
• (Jalali et al., 2023) Predictability and Comprehensibility in Post-Hoc XAI Methods: A User-Centered Analysis
• (Gomez et al., 2023) Enhancing Post-Hoc Explanation Benchmark Reliability for Image Classification
• (Jesus et al., 2021) How can I choose an explainer? An Application-grounded Evaluation of Post-hoc Explanations