Self-Explainability in AI Systems

• Self-Explainability (SX) is the capacity of AI systems to generate intrinsic explanations of their own predictions and internal states.
• SX methodologies embed explanation into the model architecture using strategies like attribution maps, concept bottlenecks, and natural language rationales.
• Evaluation of SX focuses on balancing human interpretability with model fidelity, though standardized benchmarks and metrics remain an open research challenge.

Self-Explainability (SX) denotes the capacity of a system to provide explanations of its own predictions, states, or behavior as an intrinsic part of its operation. In work on self-adaptive and self-organising systems, SX is presented as a goal beyond conventional Explainable AI: rather than explaining an AI component externally, the system itself is endowed with explanatory ability in response to growing complexity and the trust demands created by AI-enabled autonomy (Beyer et al., 8 Jun 2026). Across recent machine learning literature, closely related terms include self-explainable models and self-explaining models, covering architectures that emit predictions together with patch attributions in a single forward pass, expose concept or prototype bottlenecks, generate structured subgraphs or concise sufficient subsets, or produce natural-language rationales and counterfactuals as part of the learned model itself (Yoshikawa et al., 6 Jul 2025, Bassan et al., 5 Feb 2025).

1. Conceptual foundations

A central distinction in the SX literature is between post-hoc explanation and intrinsic explanation. Post-hoc methods explain a trained black box after the fact, whereas SX incorporates explanation into the model, training objective, or system architecture. The 2026 systematic literature review on self-adaptive and self-organising systems reports that it develops a unified definition and taxonomy of SX and introduces Levels of Self-Explainability as a framework for positioning current and future research; it also reports that most existing SX approaches remain conceptual, with few practical implementations, and that there is currently no formal or de facto standard for evaluating SX (Beyer et al., 8 Jun 2026).

A representative formalization appears in medical image analysis, where a self-explainable model $f_\theta$ is trained end-to-end with both a prediction loss and an explanation loss:

$$L(\theta)=L_{\text{task}}(f_\theta(x),y)+\lambda\cdot L_{\text{expl}}(E_\theta(x),e).$$

Here $E_\theta(x)$ denotes the explanation module, and the explanation object $e$ may be an attention map, concept vector, prototype similarity pattern, textual sentence, or counterfactual. This formulation makes the explanation part of the trained system rather than an external diagnostic layer (Hou et al., 2024).

The driving literature emphasizes a complementary conceptual split between interpretability and completeness. In that framing, interpretability concerns the degree to which a human can understand the cause of a decision, while completeness or fidelity concerns how accurately the explanation reflects the true computation of the model. Explainability is then treated as a trade-off between human comprehensibility and faithfulness to the internal mechanism (Zablocki et al., 2021). This suggests that SX is not defined by any single explanation format; it is defined by the degree to which explanation is constitutive of the model’s own computation.

2. Explanation objects and architectural forms

Recent SX systems differ primarily in what they treat as the model’s explanatory substrate. Some use human-interpretable intermediate representations such as concepts, prototypes, or semantic maps; others expose locally sufficient feature subsets, class-discriminative subgraphs, or natural-language statements grounded in the model’s internal state.

Paradigm	Explanation object	Representative mechanism
Concept-based	Phrase concepts or concept bottlenecks	Local and global concept layers
Prototype-based	Class prototypes in embedding space	Nearest-prototype decision rule
Attribution-based	Patch or pixel attributions	Single-pass explanation outputs
Subgraph-based	Class-discriminative subgraphs	Dependency weights over subgraphs
Sufficiency-based	Minimal sufficient feature subsets	Explanation head with sparsity penalty
Causal self-model	Semantically grounded latent message $z_t$	Intervenable self-model across time

In medical imaging, a three-layer view is common: input explainability through explainable feature engineering or knowledge-graph integration, model explainability through attention-based, concept-based, or prototype-based learning, and output explainability through textual or counterfactual explanations (Hou et al., 2024). Text classification provides a particularly explicit decomposition. SelfExplain augments a neural classifier with a Local Interpretability Layer that scores the relevance of phrase-level concepts and a Global Interpretability Layer that retrieves influential concepts from a training-time concept store; both operate over non-terminal phrases extracted by a constituency parser (Rajagopal et al., 2021).

Prototype-based SX appears in multiple forms. KMEx converts a frozen encoder into a prototypical self-explainable model without retraining by clustering class-specific latent embeddings with K-means and replacing the original classifier with a transparent nearest-prototype rule based on the similarity

$$s(z,p)=\log\bigl((\|z-p\|^2+1)/(\|z-p\|^2+\epsilon)\bigr).$$

Global explanations visualize prototypes via nearest training images, while local explanations identify the winning prototype for a test instance (Gautam et al., 2023). In visual classification with vision-language backbones, one-pass SX can be obtained by using the same fixed text-embedding head to map the $[\mathrm{CLS}]$ token to class logits and each patch token to per-class patch attributions, so that a single forward pass yields both $\ell$ and $\tilde\Phi$ (Yoshikawa et al., 6 Jul 2025).

Graph and decision-tree settings extend the architectural repertoire. GraphOracle learns a GNN classifier jointly with a set of sparse subgraphs whose class dependencies are encoded in a weight matrix $W\in\mathbb{R}^{|\mathcal{C}|\times M}$, with an RBF kernel quantifying graph–subgraph dependency (Liu et al., 15 Aug 2025). SX-GeoTree embeds explanation robustness into tree induction itself by jointly optimizing weighted MSE reduction, residual spatial autocorrelation via global Moran’s $$L(\theta)=L_{\text{task}}(f_\theta(x),y)+\lambda\cdot L_{\text{expl}}(E_\theta(x),e).$$0, and a modularity objective defined on a consensus network that combines geographically weighted regression similarity with SHAP attribution similarity (Kang et al., 25 Nov 2025). Sufficient Subset Training (SST), by contrast, makes the explanation object a selected subset of input features, produced by an explanation head $$L(\theta)=L_{\text{task}}(f_\theta(x),y)+\lambda\cdot L_{\text{expl}}(E_\theta(x),e).$$1 whose thresholded support is explicitly trained to be a concise sufficient reason for the model’s prediction (Bassan et al., 5 Feb 2025).

3. Learning paradigms for self-explanation

The dominant learning pattern in SX is joint optimization of predictive performance and explanatory structure, but the literature now includes several distinct paradigms.

One family uses explicit explanation supervision. In the task-arithmetic framework for transferring visual explainability, a model is fine-tuned on a source domain either for prediction only or for joint prediction and explanation, yielding parameters $$L(\theta)=L_{\text{task}}(f_\theta(x),y)+\lambda\cdot L_{\text{expl}}(E_\theta(x),e).$$2 and $$L(\theta)=L_{\text{task}}(f_\theta(x),y)+\lambda\cdot L_{\text{expl}}(E_\theta(x),e).$$3. Their difference,

$$L(\theta)=L_{\text{task}}(f_\theta(x),y)+\lambda\cdot L_{\text{expl}}(E_\theta(x),e).$$4

is treated as an explainability vector. Explainability is then transferred to a target-domain classifier by adding this vector to target parameters according to

$$L(\theta)=L_{\text{task}}(f_\theta(x),y)+\lambda\cdot L_{\text{expl}}(E_\theta(x),e).$$5

The method thus separates explanation skill from task-specific classification skill at the parameter level (Yoshikawa et al., 6 Jul 2025).

A second family uses self-supervision or self-refinement. SST trains a model $$L(\theta)=L_{\text{task}}(f_\theta(x),y)+\lambda\cdot L_{\text{expl}}(E_\theta(x),e).$$6 with a standard prediction branch and an explanation branch, combining prediction loss, a faithfulness loss on masked inputs, and a cardinality penalty:

$$L(\theta)=L_{\text{task}}(f_\theta(x),y)+\lambda\cdot L_{\text{expl}}(E_\theta(x),e).$$7

The thresholded support of $$L(\theta)=L_{\text{task}}(f_\theta(x),y)+\lambda\cdot L_{\text{expl}}(E_\theta(x),e).$$8 is trained to be sufficient for the prediction while remaining small, thereby internalizing the search for minimal sufficient reasons during training rather than as an external combinatorial procedure (Bassan et al., 5 Feb 2025). Learning by Self-Explaining (LSX) introduces an internal critic $$L(\theta)=L_{\text{task}}(f_\theta(x),y)+\lambda\cdot L_{\text{expl}}(E_\theta(x),e).$$9 that attempts to solve the original task from the learner’s explanations alone. The learner is optimized with

$E_\theta(x)$0

so explanations are rewarded to the extent that they are useful for the critic (Stammer et al., 2023).

A third family relies on synthetic or latent explanatory channels. In multimodal foundation models, visual rejection sampling uses self-synthesized, concept-grounded explanations as training targets. The model samples multiple candidate answers, scores them against image-specific concepts selected by an InfoNCE-style criterion, filters them with a reward-model-free rule, and fine-tunes iteratively on the retained answers (Shi et al., 19 Feb 2025). In reinforcement learning, Causal Self-Talk trains an agent to communicate with itself across time through a semantically grounded latent variable $E_\theta(x)$1. Explanations become causally meaningful because interventions on $E_\theta(x)$2 are inserted back into the agent’s recurrent state update and evaluated through return maximization, memory reconstruction, or policy-distillation objectives (Roy et al., 2022).

A fourth paradigm minimizes modification of the base model. KMEx is notable because it converts an existing pretrained model into a prototypical self-explainable model by freezing the encoder, discarding the original classifier, clustering embeddings into class-specific prototypes, and using transparent nearest-prototype inference; no retraining of the backbone is required (Gautam et al., 2023). This suggests that the SX design space includes both deeply integrated end-to-end methods and conversion procedures that retrofit intrinsic transparency onto already trained systems.

4. Evaluation: fidelity, stability, usefulness, and the problem of standards

Evaluation remains the least standardized part of SX. The 2026 review explicitly identifies the absence of a formal or de facto evaluation standard as a major research gap (Beyer et al., 8 Jun 2026). Existing work therefore assembles evaluation protocols from neighboring literatures and from model-specific desiderata.

A recurring set of desiderata includes faithfulness, consistency, understandability, plausibility, computational efficiency, and portability across architectures. In medical imaging, functionality-grounded evaluation includes deletion and insertion tests for attention maps, ROAR for feature importance, Completeness Score for concept-based methods, non-representativeness and diversity for prototypes, BLEU, ROUGE-L, and CIDEr for textual explanations, and validity, proximity, sparsity, and realism for counterfactuals (Hou et al., 2024). Autonomous driving uses related metrics, including neuron coverage, path coverage, concept activation coverage, realism of counterfactuals via FID, and human-centered notions such as simulatability and time to understand, while also emphasizing that natural-language metrics measure plausibility rather than faithfulness (Zablocki et al., 2021).

Recent SX papers add specialized metrics tied to their architectural commitments. KMEx evaluates predictive faithfulness through top-1 accuracy, local explanation robustness through Average Drop, Average Increase, and Relevance Ordering curves, transparency through the prototype ghosting metric

$E_\theta(x)$3

faithfulness to the black box through normalized KL divergence $E_\theta(x)$4 between relevance maps, and prototype diversity through an entropy-based measure $E_\theta(x)$5 (Gautam et al., 2023). The explainability-transfer framework evaluates explanation root-mean-square error, IoU@K for top-attributed patches, and Infidelity, and compares one-pass SX explanations against Kernel SHAP under different perturbation budgets (Yoshikawa et al., 6 Jul 2025). LSX measures explanation consolidation through Inter- vs Intra-class Explanation Similarity and ridge-regression accuracy on explanation representations, and measures faithfulness through sufficiency and comprehensiveness (Stammer et al., 2023).

Graph and spatial systems exemplify still more specialized evaluation. GraphOracle proposes masking-based class-level faithfulness scores $E_\theta(x)$6, $E_\theta(x)$7, and $E_\theta(x)$8 on dependency-based subgraph features (Liu et al., 15 Aug 2025). SX-GeoTree evaluates not only predictive $E_\theta(x)$9 but also residual spatial evenness through Moran’s $e$0 and attribution consensus through graph modularity $e$1, directly aligning evaluation with its definition of spatially coherent, locally stable explanations (Kang et al., 25 Nov 2025). The heterogeneity of these measures is itself a substantive feature of the field: different SX systems operationalize “self-explanation” through different explanatory objects, and each object induces a different notion of success.

5. Representative systems and empirical findings

Practical SX implementations now span vision, language, graphs, geospatial modeling, reinforcement learning, and multimodal reasoning, although the broader review literature still characterizes implementation maturity as limited (Beyer et al., 8 Jun 2026).

In image and multimodal classification, one influential trajectory prioritizes explanation generation in a single inference. The vision-language task-arithmetic approach reports that visual explainability can often be transferred from source to target domains without sacrificing classification accuracy, that an explainability vector learned on ImageNet+X improves explanation quality on nine out of ten target datasets, and that the explanation quality of a single SX forward pass is comparable to Kernel SHAP using approximately $e$2 perturbation-based model inferences (Yoshikawa et al., 6 Jul 2025). A different line of work improves multimodal foundation models through self-synthesized, concept-grounded training data: on CUB-200, performance across iterations rises from $e$3 to $e$4, explanation existence reaches $e$5 across datasets, concept-selection precision reaches up to $e$6 with $e$7 descriptions, and removing the rejection-sampling filter causes accuracy on CUB-200 to stall near $e$8 rather than climbing to $e$9 (Shi et al., 19 Feb 2025).

Prototype-based and concept-based systems show that intrinsic explainability need not require large accuracy losses. KMEx matches the frozen ResNet34 baseline within $z_t$0 top-1 accuracy, achieves $z_t$1 on CIFAR-10, and attains the lowest $z_t$2 across all evaluated datasets, with example values in the range $z_t$3–$z_t$4 (Gautam et al., 2023). In text classification, SelfExplain reports that phrase-based explanations are sufficient for model predictions under the FRESH protocol, with local concepts reaching $z_t$5 accuracy and global concepts $z_t$6 on SST-2, while end-task accuracy remains comparable to or slightly above baselines on SST-2, SST-5, TREC-6, TREC-50, and SUBJ; human judges also prefer its explanations in adequacy, understandability, and trust (Rajagopal et al., 2021).

Spatial and graph-structured domains demonstrate that SX can be tailored to domain geometry. SX-GeoTree preserves predictive performance within $z_t$7 $z_t$8 of decision trees while improving residual spatial evenness and roughly doubling attribution consensus: on Fujian, modularity improves from $z_t$9 to $$s(z,p)=\log\bigl((\|z-p\|^2+1)/(\|z-p\|^2+\epsilon)\bigr).$$0, and on Seattle from $$s(z,p)=\log\bigl((\|z-p\|^2+1)/(\|z-p\|^2+\epsilon)\bigr).$$1 to $$s(z,p)=\log\bigl((\|z-p\|^2+1)/(\|z-p\|^2+\epsilon)\bigr).$$2; ablations show that removing either Moran’s $$s(z,p)=\log\bigl((\|z-p\|^2+1)/(\|z-p\|^2+\epsilon)\bigr).$$3 or modularity degrades both spatial residual structure and explanation stability (Kang et al., 25 Nov 2025). GraphOracle addresses class-level self-explainability in GNNs and reports a $$s(z,p)=\log\bigl((\|z-p\|^2+1)/(\|z-p\|^2+\epsilon)\bigr).$$4 of $$s(z,p)=\log\bigl((\|z-p\|^2+1)/(\|z-p\|^2+\epsilon)\bigr).$$5 on MUTAG at $$s(z,p)=\log\bigl((\|z-p\|^2+1)/(\|z-p\|^2+\epsilon)\bigr).$$6 sparsity, compared with $$s(z,p)=\log\bigl((\|z-p\|^2+1)/(\|z-p\|^2+\epsilon)\bigr).$$7 for ProtGNN and $$s(z,p)=\log\bigl((\|z-p\|^2+1)/(\|z-p\|^2+\epsilon)\bigr).$$8 for PGIB, while also training at least $$s(z,p)=\log\bigl((\|z-p\|^2+1)/(\|z-p\|^2+\epsilon)\bigr).$$9 faster than ProtGNN and PGIB and up to $[\mathrm{CLS}]$0 faster than GLGExplainer (Liu et al., 15 Aug 2025).

Learning-time self-explanation can also act as a regularizer. LSX improves few-shot and full-data generalization on several datasets, including ChestMNIST, where accuracy rises from $[\mathrm{CLS}]$1 to $[\mathrm{CLS}]$2 at the full training size, and CUB-10, where accuracy rises from $[\mathrm{CLS}]$3 to $[\mathrm{CLS}]$4 at $[\mathrm{CLS}]$5 training samples. It also mitigates confounding effects, for example increasing DecoyMNIST test accuracy from $[\mathrm{CLS}]$6 to $[\mathrm{CLS}]$7 under confounded training, while producing markedly more class-consolidated explanations as measured by IIES and ridge-regression separability (Stammer et al., 2023). In reinforcement learning, Causal Self-Talk shows that ordinary decoders can have high correlational faithfulness yet near-chance causal faithfulness, whereas CST variants retain correlational performance while substantially increasing causal faithfulness; semantic interventions on the learned self-model also recover much of the return achieved when the environment itself provides full task information (Roy et al., 2022).

6. Open problems, tensions, and research directions

The most explicit field-wide conclusion is that SX lacks shared standards. The 2026 review identifies both a conceptual need—its unified definition, taxonomy, and Levels of Self-Explainability are introduced precisely because the area is fragmented—and an empirical need, namely the absence of standard evaluation practice (Beyer et al., 8 Jun 2026). Survey work in medical imaging and autonomous driving adds that standardized benchmarks with concept labels, segmentation masks, eye-tracking, counterfactual ground truth, or human explanations are scarce, making fair comparison difficult (Hou et al., 2024, Zablocki et al., 2021).

A second persistent issue is the relation between explanation quality and task performance. Medical imaging and driving surveys both identify the performance–interpretability trade-off as unresolved, even where recent concept bottleneck and prototype approaches narrow the gap (Hou et al., 2024, Zablocki et al., 2021). Concrete methods expose the same tension in different forms: explainability transfer can fail when source and target domains are too dissimilar; KMEx inherits the stability of the frozen encoder and does not introduce its own explanation-robustness regularizer; SST can slightly degrade accuracy under aggressive sparsity or large-scale settings; and GraphOracle remains competitive rather than uniformly best on all datasets despite its explainability constraints (Yoshikawa et al., 6 Jul 2025, Gautam et al., 2023, Bassan et al., 5 Feb 2025, Liu et al., 15 Aug 2025).

A third issue concerns the meaning of faithfulness. Some systems optimize agreement with external explanations, such as SHAP-derived patch attributions; others optimize sufficiency, causal intervention effects, class-level dependency masking, or domain-specific stability properties such as spatial coherence. This plurality indicates that “faithfulness” is not operationally uniform across SX. A plausible implication is that future standards will need to distinguish between at least causal faithfulness, predictive sufficiency, representation-level alignment, and human plausibility, rather than collapsing them into a single score.

Finally, several papers point to broader directions: integrating SX priors into larger foundation models, enabling human-in-the-loop correction of concepts or prototypes, improving portability across architectures, scaling explanation-supervised sources, extending class-level structure learning to more GNN families, and moving from correlational to causal criteria in high-stakes settings (Hou et al., 2024, Zablocki et al., 2021, Yoshikawa et al., 6 Jul 2025, Liu et al., 15 Aug 2025, Roy et al., 2022). Taken together, these directions portray SX not as a single method family but as an organizing principle for system design: explanations are most credible when they are part of the computation that produces the decision, and most useful when their faithfulness, stability, and domain relevance can be established with rigor.