Explainability (XAI): Methods & Implications

• Explainability (XAI) is a framework that renders AI models transparent by translating complex decisions into human-understandable insights.
• XAI techniques include inherently interpretable models and post-hoc methods such as LIME, SHAP, and gradient-based approaches to provide local and global explanations.
• The field enhances trust in critical applications like healthcare and finance by aligning technical model behavior with ethical, regulatory, and practical requirements.

Explainability, also called Explainable Artificial Intelligence (XAI), encompasses a broad set of methodologies, definitions, and engineering protocols aimed at providing human-understandable insights into the inner workings, decision processes, and output rationales of AI models. The objective is to transition from opaque “black-box” models to systems whose behaviors can be comprehensively interrogated, justified, audited, and trusted in high-stakes contexts such as finance, healthcare, law, and autonomous vehicles (Benchekroun et al., 2020). The field emphasizes the need for principled, standardized frameworks, formalizes various methods for both inherently interpretable and post-hoc explanation, and integrates concepts from logic, cognitive science, domain-specific knowledge, and regulatory requirements.

1. Key Definitions and Conceptual Distinctions

Explainability (XAI) is formally defined as the condition that allows users to understand the decisions made by a model and its subcomponents through all phases—before, during, and after model construction (Benchekroun et al., 2020). Closely related concepts are:

• Interpretability: The property that enables understanding of the direct relationship between model input and output, providing explicit mapping of inputs to outputs.
• Auditability: The capacity to rigorously question whether a model behaves as expected, exposing unwanted biases or errors, and assuring compliance with standards and regulations.
• Transparency: The degree to which both model logic and internal parameters are available for direct inspection and analysis, encompassing interpretability and explainability.

A further distinction is drawn between local explanations (pertaining to an individual prediction) and global explanations (pertaining to the overall model behavior), with implications for both human understanding and regulatory compliance (Islam et al., 2021).

2. Taxonomies and Methodological Landscape

A taxonomy reflecting the machine learning workflow segments explainability approaches as follows (Benchekroun et al., 2020):

Phase	XAI Methods Examples	Key Focus
Pre-modelling	Data description, statistical plots	Understanding data context
Modelling	Decision trees, linear predictors	Mechanistic interpretability
Post-modelling	LIME, DeepLift, SHAP	Approximate black-box models

• Inherently interpretable models: Linear regression, logistic regression, decision trees, rule-based models, and k-nearest neighbors (KNN) are designed for transparency. The weights, paths, or rule sets afford direct interpretation, though sometimes at the expense of model fidelity for complex tasks.
• Post-hoc model-agnostic methods: E.g., LIME (Local Interpretable Model-agnostic Explanations) constructs sparse, local surrogate models to approximate the original predictor,

$$\xi(x) = \arg\min_{g \in G} L(f, g, \pi_x) + \Omega(g)$$

where $f$ is the black-box model, $g$ is a simple interpretable one, and $L$/Ω regularize fidelity and complexity.

• Attribution and importance methods: SHAP assigns additive attributions to features based on the cooperative game theory formula,

$g(z') = \phi_0 + \sum_{j=1}^{M} \phi_j z'_j$

where $z'$ is the coalition vector and $\phi_j$ is the feature attribution.

• Gradient-based methods: DeepLift, Integrated Gradients, and saliency maps quantify input feature contributions via explicit or baseline-referenced gradient calculations.
• Counterfactual and contrastive methods: Explore “what-if” scenarios to enhance causal understanding and reveal the minimum set of changes required to flip model decisions.

The literature also highlights the need to relate the interpretability of explanation mechanisms to their cognitive and practical complexity, often assessed via model size, rule depth, or number of cognitive “chunks” (Islam et al., 2021).

3. Theoretical Frameworks and Formal Properties

Recent works formalize explanations as comprising two components: objective evidence extracted from the model (such as gradients, attention scores, or feature attributions) and the interpretation function that assigns semantic meaning to this evidence (Rizzo et al., 2022). The framework introduces two foundational properties:

• Faithfulness: The degree to which the explanation reflects the model’s actual reasoning and internal causal processes, evaluated through the correspondence between model internals and external mutation tests or ablation studies.
• Plausibility: The subjective credibility or acceptability of an explanation to a human user, often measured by user studies but potentially at odds with faithfulness.

The overall faithfulness $\Phi$ of an explanation can be formalized as:

$\Phi = \sum_{i \in I} \epsilon_i \varphi_i$

where $\epsilon_i$ is the explanatory potential of evidence at transformation step $i$, and $\varphi_i$ is the faithfulness of interpretation at that step.

Logic-based methods further advance explainability rigor by defining abductive explanations (AXps) and contrastive explanations (CXps) as minimal, sufficient (and, for contrast, minimally different) feature subsets that guarantee the outcome (Marques-Silva, 4 Jun 2024):

$$\forall x \in \mathcal{F}.\left[\bigwedge_{i\in X}(x_i = v_i)\right] \to (\kappa(x) = \kappa(v))$$

These approaches, combined with SAT/SMT encoding, enable efficient enumeration and verification of explanations for classes of models such as decision trees and rule-based systems, and motivate a shift away from potentially misleading feature importance scores (as revealed in recent critiques of SHAP values (Marques-Silva et al., 2023, Marques-Silva, 4 Jun 2024)).

4. Evaluation, Standardization, and Stakeholder Alignment

The measurement and benchmarking of explainability remain open challenges. Quantitative metrics include:

• Model complexity measures: Rule depth, number of nonzero coefficients, operation counts.
• Fidelity metrics: Local accuracy in surrogate methods, consistency between explanation and model perturbations, and objective faithfulness tests.
• Human-centered evaluations: Simulatability, operation count required for human reproduction of model output, and subjective comprehensibility scores.
• Correlation with regulatory and practical needs: Assessments of auditability, compliance, and risk calibration.

Frameworks such as the one in (Langer et al., 2021) decompose the XAI process into: (i) the explainability approach, (ii) explanatory information, (iii) stakeholder understanding, and (iv) satisfaction of stakeholders’ desiderata, with context acting as a moderator throughout. Stakeholder requirements (developers, deployers, affected parties, policy-makers) necessitate mapping explanations to varying degrees of epistemic (can I know/evaluate the system?) and substantial (is the system actually fair/robust?) satisfaction.

Efforts to standardize the field include calls for taxonomies with clear operational phases and metrics (e.g., “explainability-to-complexity” ratio), as well as for inclusion in regulatory frameworks such as the EU Artificial Intelligence Act, which mandates XAI for high-risk applications with enforceable technical documentation and oversight (Pavlidis, 24 Jan 2025).

5. Applied Domains and Human-Centric Design

Explainability’s practical value is highlighted across critical domains:

• Legal and policy AI: Adaptations of vision techniques (such as Grad-CAM) to legal text processing pipelines, using token-level heatmaps and metrics like attention spread ($\mathcal{F}$) and intersection over union ($\mathcal{I}$) to elucidate model behavior on high-stakes legal judgements (Gorski et al., 2020).
• Medical diagnostics: Human-centric, expert-driven concept bottleneck models (CBM), as in lung cancer detection, split decision-making into concept extraction (reflecting radiologically meaningful terms) and explicit label prediction, validated both quantitatively and via expert review (Rafferty et al., 14 May 2025).
• Engineering applications: In autonomous vehicles, layered explanations tie together sensor input fusion, object detection, visual attention (via heatmaps), and action selection, with explicit vehicle dynamic models for simulation and traceability (Hussain et al., 2021).
• LLMs: Current XAI must address scale, generative diversity, and emergent properties. Recent frameworks leverage LLMs both as explanation generators (e.g., translating feature attributions into natural language) and as explanation consumers or critics, employing chain-of-thought protocols, knowledge-augmented prompting, influence functions, and attribution-based diagnostics (Wu et al., 13 Mar 2024, Paraschou et al., 13 Jun 2025, Mumuni et al., 17 Jan 2025).
• Responsible AI: Explainability is argued to underpin all pillars of Responsible AI, including fairness (via detection of unwanted bias), robustness (via adversarial and counterfactual tests), privacy (alignment with federated/differential privacy), and accountability (regulatory documentation, auditability) (Baker et al., 2023).

6. Current Limitations, Critiques, and Future Directions

Multiple sources highlight significant challenges and controversies:

• Lack of Standardization: Inconsistent definitions, ad hoc approaches, and the absence of rigorous, universally accepted frameworks diminish the reliability of XAI in practice (Benchekroun et al., 2020).
• Trade-offs and Performance: Efforts to enhance explainability—by restricting model complexity or imposing transparency—may decrease predictive accuracy; domain-specific trade-offs must be judiciously managed (Islam et al., 2021, Hussain et al., 2021).
• Flawed Attribution Methods: Game-theoretic attributions (e.g., SHAP) can fail to identify truly relevant features or may attribute nonzero importance to irrelevant features due to their aggregation over all possible coalitions, challenging their use in high-stakes settings (Marques-Silva et al., 2023, Marques-Silva, 4 Jun 2024).
• Faithfulness vs Plausibility: Highly plausible (subjectively convincing) explanations are not guaranteed to be technically faithful, and faithfulness alone may fail to capture explanatory needs or cognitive costs (Rizzo et al., 2022).
• Domain and User Adaptation: The necessity for context-sensitive, audience-adaptive explanations is articulated in both regulatory contexts (e.g., EU AI Act) and artistic/performance domains, mandating iterative, feedback-driven explanation design (Privato et al., 2023, Pavlidis, 24 Jan 2025).
• Toward Rigorous, Certified XAI: The field is moving toward formally verified and certified approaches, scalable logic-based methods, human-centered and usability-focused XAI, standardized evaluation protocols, and hybrid techniques incorporating domain knowledge (Rizzo et al., 2022, Marques-Silva, 4 Jun 2024, Paraschou et al., 13 Jun 2025).

7. Implications for Research and Practice

Explainability is now an operational and regulatory imperative for deploying AI in most high-stakes settings. The field’s trajectory includes:

• Standard-setting via interdisciplinary engagement (logic, psychology, legal studies, engineering).
• Objective measurement and benchmarking frameworks balancing faithfulness, plausibility, and user needs.
• Integration of domain knowledge and user context in explanation generation, with advances in leveraging LLMs and multi-modal models for human-centered explanation delivery.
• Rigorous, logic-based, and formally certified explanation methods for trust and accountability, especially in safety-critical and adversarial environments.

A plausible implication is that future XAI research will further align technical explanations with both the epistemic needs of diverse stakeholder groups and the legal, social, and economic imperatives imposed by governance frameworks. This is a shift away from ad hoc, model-agnostic attribution and toward structured, context-sensitive, and certified explanation pipelines.