---
title: Explainable AI (xAI) Models
url: https://www.emergentmind.com/topics/explainable-ai-xai-models
type: topic
---

# Explainable AI (xAI) Models

Explainable Artificial Intelligence (xAI) models are systems and methodologies designed to render the decision processes of complex AI models intelligible to human stakeholders. These approaches separate the technical inner logic of model computation from the human-facing act of rendering explanations, operationalize dual criteria—faithfulness to the model’s operations and plausibility to end users—and ground the comparison and evaluation of explanation methods in formal properties. While originally motivated by safety and regulatory demands in domains such as biomedicine, finance, and aerospace, contemporary xAI also faces calls for democratized, human-centered explanation that serves both technical experts and lay audiences [2212.14447], [2501.09967], [2412.17440].

## 1. Formal Definition and Structural Framework

A principled xAI model decomposes every explanation into two core components [2212.14447]:

- **Evidence** ($e$): Objective information extracted from the trained model $M$, query input $x$, and/or the model’s output $\hat{y}$, via an extractor $\xi$, i.e., $e = \xi(x, \hat{y}, M)$.
- **Interpretation** ($g$): A function that maps evidence (and possibly $x$, $\hat{y}$, $M$) to a human-understandable explanation, $E = g(e, x, \hat{y}, M)$.

This separation exposes the dual “atomic” building blocks of any explanation, distinguishing raw model artifacts from the mapping that contextualizes them for human understanding.

Central to this account is **explanatory potential** ($\varepsilon(e)$), which quantifies the proportion of the model’s decision chain or internal transformations that the evidence can, in principle, illuminate. Breadth (number of covered steps) and depth (extent to which each step is constrained) jointly compose $\varepsilon$.

## 2. Faithfulness and Plausibility: Dual Quality Criteria

The two classical desiderata—faithfulness and plausibility—are operationalized in the evidence–interpretation architecture [2212.14447]:

- **Faithfulness** ($\varphi_i(g, e)$): For each internal transformation step $s_i$ that $g(e)$ claims to explain, faithfulness is the scalar $\varphi_i(g, e) \in [0, 1]$ measuring how accurately $g$ reflects the true function $f_{i, m_i}$ learned at step $s_i$. It is only nonzero if the evidence possesses nonzero explanatory potential for $s_i$. Overall faithfulness aggregates as
  $$
  \Phi = \sum_{i \in I} \varepsilon_i \cdot \varphi_i(g, e)
  $$
  and is measured via requirement-based testing (e.g., feature occlusion, sensitivity analysis).

- **Plausibility** ($\pi(E)$): A user-dependent assessment of whether the explanation is convincing/intelligible. It comprises:
  1. Human-understandability: Can the user parse the explanation?
  2. Informativeness (depth): Does the explanation delve into the most relevant steps?
  3. Completeness (width): Does the explanation cover sufficient aspects of the model to be useful for generalization or anticipation?

Plausibility is typically evaluated by user studies or expert surveys, especially in high-stakes domains.

## 3. Principal XAI Model Classes and Case Study Mapping

xAI methods can be categorized by properties including ante-hoc versus post-hoc operation, model-specificity, and the granularity (global vs local) of explanations [2206.04800], [2312.06082], [2501.09967].

**Intrinsic (ante-hoc) models** are interpretable by design:
- **Linear Regression**: $f_{\text{lin}}(x) = \beta_0 + \sum_i \beta_i x_i$, with evidence $e = \{\beta_i\}_i$, full explanatory potential $\varepsilon = 1$, and faithfulness $\varphi = 1$ by construction.
- **Rule-Based/Fuzzy Systems**: Human-readable rules with $\varepsilon = 1$, $\varphi \approx 1$; typically high plausibility among practitioners.

**Post-hoc XAI methods** provide explanations for black-box models. Notable techniques include:
- **Attention Mechanisms**: Expose learned token or region importances. Empirical studies have found low faithfulness to real causal pathways despite high plausibility [2212.14447].
- **Grad-CAM and Saliency Maps**: Visual highlight maps from gradients or convolution outputs; faithfulness is measured by occlusion tests and correlation with output changes.
- **SHAP (Shapley Additive Explanations)**: Computes feature attributions $\phi_i$ via the Shapley value formula,
  $$
  \phi_i(x) = \sum_{S \subseteq F \setminus \{i\}} \frac{|S|!(|F|-|S|-1)!}{|F|!} [f_{S \cup \{i\}}(x) - f_{S}(x)]
  $$
  providing high faithfulness by satisfying local accuracy, consistency, and other axioms, though sometimes low plausibility if too many features are nonzero.

- **LIME (Local Interpretable Model-Agnostic Explanations)**: Fits a local surrogate (typically sparse linear) around an instance; faithfulness limited to the locality of the fit.

- **Layer-wise Relevance Propagation (LRP)**: Backpropagates relevance through each layer, yielding fine-grained attribution heatmaps; faithfulness and potential depend on propagation rules.

**Table 1: Mapping of Representative XAI Methods to the Evidence–Interpretation Framework**

| Method     | Evidence ($e$)           | Explanatory Potential ($\varepsilon$) | Faithfulness ($\varphi$)         |
|------------|--------------------------|----------------------------------------|----------------------------------|
| Linear     | Regression coefficients  | $1$ (full model)                       | $1$ (algebraic identity)         |
| SHAP       | Marginal outputs over    | Fraction of coalitions sampled         | High (axiomatic guarantee)       |
|            | feature coalitions       |                                        |                                  |
| LIME       | Local surrogate weights  | Fraction of neighborhood covered       | Moderate; local only             |
| Grad-CAM   | Feature map gradients    | Fraction of conv. layer params         | Varies; tested by occlusion      |
| Attention  | Attention weights        | Single layer; limited global coverage  | Often low by intervention tests  |

## 4. Design, Evaluation, and Selection in Practice

The evidence–interpretation formalism allows systematic design, quantitative evaluation, and rational selection of xAI approaches [2212.14447]. Specifically:

- **Design**: Select evidence with high explanatory potential relevant to decision points of interest. Architect models to facilitate tractable evidence extraction and high-fidelity explanatory mapping.
- **Evaluation**: Faithfulness can be quantitatively tested (e.g., feature occlusion drops), while plausibility requires user studies. Trade-offs are observed—maximizing explanatory coverage ($\varepsilon$) often increases cognitive load, possibly lowering plausibility ($\pi$).
- **Comparison**: Methods are compared as tuples $(\varepsilon, \varphi, \pi)$. For instance, SHAP may offer high $\varepsilon$, $\varphi$ but lower $\pi$ in data-rich domains; Grad-CAM may be moderate in $\varepsilon, \varphi$, and high in $\pi$ for image specialists.

In domain-specific deployments such as biomedicine, explainability should be defined and enforced as a non-functional system requirement, with explicit tests for explanatory potential and faithfulness, piloted in situ with real end users.

## 5. Audience Adaptation and Human-Centered Explanations

Recent work highlights the necessity of tailoring explanations for experts, generalists, and non-expert stakeholders. Approaches such as x-[plAIn] use large language models conditioned on audience profiles to adapt the style, detail, and mathematical depth of explanations [2401.13110]. These models introduce “audience embeddings” selecting the degree of formulaic transparency and automate translation of raw XAI outputs (SHAP, LIME, Grad-CAM) into audience-specific, consumable rationales. Empirical studies demonstrate improved comprehension accuracy and reduced interpretation time, especially for non-expert users.

Parallel developments in “human-centered” xAI frameworks leverage structured dual-output templates: a machine-readable (“expert”) explanation (feature importances, rules) and a narrative (“non-expert”) explanation. This stratification, validated by correlation with ground-truth XAI methods and user studies, bridges the gap between algorithmic transparency and practical interpretability [2506.12240].

## 6. Limitations, Best Practices, and Open Research Challenges

Despite substantial progress, critical challenges remain:

- **Evaluation Deficit**: A scoping review found that 81% of published applications that label themselves “explainable” do not evaluate the quality or suitability of their chosen XAI method [2307.09673]. Explanation fidelity—how well the surrogate or attribution matches the underlying model—should be explicitly measured.
- **Faithfulness vs. Plausibility Tension**: High plausibility (user satisfaction, surface intuitiveness) can occur without faithfulness; empirical failures of attention as an attribution method exemplify this pitfall [2212.14447].
- **Contextualization and Human-in-the-Loop**: Explanation needs and appropriate metrics differ by user, domain, and task. Best practices demand contextual validation, mixed quantitative–qualitative evaluation, and rigorous, reproducible reporting aligned with community standards [2307.09673].
- **Hybrid and Adaptive Explanations**: Combining multiple methodologies—model-specific and model-agnostic, local and global—can yield more complete, trust-calibrated explanations [2206.04800], [2312.06082].
- **Open Problems**: There is no consensus on universally applicable faithfulness metrics or semantic ground truths for explanation, and further research is needed on adversarial robustness and user-centered explanation interfaces [2206.04800], [2307.09673].

## 7. Application Domains and Impact

In safety-critical and regulated sectors, xAI frameworks have been applied to:

- **Biomedicine**: For diagnosis support, risk scoring, and biomarker discovery, with explicit balancing of faithfulness, explanatory potential, and user plausibility [2212.14447].
- **IoT and Smart Infrastructure**: To pinpoint influential sensors or control signals, using SHAP/LIME for feature attribution and Grad-CAM for spatial localization [2206.04800].
- **Bioinformatics**: Combining model-agnostic and introspective methods to interpret deep sequence, structure, and image models [2312.06082].
- **Aerospace**: Facilitating error analysis, trust, and control in air traffic management and predictive maintenance via hybrid transparent and post-hoc explanation methods [2412.17440].

Successful deployment requires domain-specific benchmarking, continuous evaluation against evolving user needs, and a layered portfolio of interpretable model forms and post-hoc explainers.

---

References:  
- [2212.14447]  
- [2401.13110]  
- [2307.09673]  
- [2206.04800]  
- [2312.06082]  
- [2412.17440]  
- [2506.12240]

Source: https://www.emergentmind.com/topics/explainable-ai-xai-models