Explanatory AI Overview

• Explanatory AI is a paradigm that treats explanation as a primary problem, using narrative and interactive methods to align with human understanding.
• It integrates contextual reasoning, hybrid architectures, and adaptive communication to serve both technical insiders and societal stakeholders.
• It supports trust, accountability, and effective decision-making by combining algorithmic transparency with conversational, human-centered interfaces.

Explanatory AI denotes a family of approaches that treat explanation as a first-class problem of human understanding rather than as a narrow by-product of model inspection. In the literature, the term is used in several closely related senses: as a demand for “outside explanations” that answer the questions asked by society rather than only the debugging questions asked by AI experts (Gilpin et al., 2019); as a move beyond low-level “narrow” explanations toward human-aligned conversational explanations that include beliefs, motivations, hypotheses of other agents’ intentions, interpretation of external cultural expectations, and the processes used to generate the explanation itself (Dazeley et al., 2021); and as a complementary paradigm in which hybrid or generative systems function as explanatory partners for human decision-making rather than as providers of algorithmic transparency alone (Nirenburg et al., 2024, Meske et al., 8 Aug 2025). Across these uses, Explanatory AI is concerned with faithfulness, inspectability, context sensitivity, user adaptation, and the ability to support trust, accountability, and action.

1. Conceptual scope and boundaries

A central distinction in this literature is between explanations aimed at technical insiders and explanations aimed at affected outsiders. “Explaining Explanations to Society” distinguishes inside explanations, which support debugging, reliability, and validation, from outside explanations, which must answer why a decision occurred and how it could have been different in a form that is both faithful to the model and understandable to humans (Gilpin et al., 2019). This distinction matters because a saliency map, local surrogate, or neuron-level attribution may satisfy expert needs while failing to answer the questions asked by a loan applicant, regulator, clinician, or accident investigator.

Several papers argue that explanation cannot be reduced to opening a black box. “Explaining Explaining” states that machine-learning systems “can’t explain because they are usually black boxes,” and proposes instead a hybrid approach in which a knowledge-based infrastructure is supplemented by machine learning when applicable, with humans retaining ultimate responsibility for decisions and actions (Nirenburg et al., 2024). In a different but related register, “From Explainable to Explanatory Artificial Intelligence” presents Explanatory AI as a complementary paradigm to classical XAI: XAI reveals algorithmic decision processes, whereas Explanatory AI addresses contextual reasoning through narrative communication, adaptive personalization, and progressive disclosure (Meske et al., 8 Aug 2025).

The boundaries of the term are themselves contested. One paper proposes a strict hierarchy in which $$\mathrm{XAI}\subseteq\mathrm{IAI}\subseteq\mathrm{EAI}\subseteq\mathrm{TAI}$$, defining Explanatory AI as the union of objective explanation and subjective interpretation, and placing it above explainable and interpretable AI in a broader trustworthy-AI stack (Wu et al., 2024). That formulation is not universal, but it captures an important recurrent claim: explanation is not exhausted by post-hoc factual disclosure, because intelligibility, acceptability, and ethical judgment enter at the level of interpretation.

2. Taxonomies of explanatory content

Yiheng Yao’s “Explanatory Pluralism in Explainable AI” argues that the demand for explanation is ambiguous because there are many types of explanation with different evaluative criteria. The paper introduces a 2×2 taxonomy, organized by Mechanistic vs. Social and Particular vs. General, yielding four explanation types (Yao, 2021).

Type	Core characterization	Primary use
Diagnostic-explanations	Inner mechanisms of a specific model decision	Debugging, error attribution
Expectation-explanations	Global regularities or guarantees of a model class	Safety certification, assurance
Explication-explanations	A single output rendered understandable to humans	Trust calibration, transparency
Role-explanations	Justification of a system’s place in social context	Accountability, regulation

This taxonomy is important because it prevents direct comparison of explanation methods that pursue different goals. A high-fidelity diagnostic explainer and a socially legible role-explanation are not competing answers to the same question. They intervene at different loci in the AI system and are judged by different standards.

A second taxonomy comes from Jacovi et al., who draw on folk psychology and Theory-Theory to identify four “folk concepts of behavior” that humans use to understand agents: Representation Causes, Internal Representation, External Causes, and Contrast Cases (Jacovi et al., 2022). Their blueprint for a complete explanatory narrative makes explicit the chain

$$(RC \rightarrow R)\;\land\;(R \rightarrow \mathrm{Outcome})\;\land\;(EC \rightarrow \mathrm{Outcome}),$$

with each clause presented contrastively. This framework shifts attention from the information contained in an explanatory artifact to the information that a human explainee is likely to understand from it.

The concern that current XAI is too narrow is also explicit in “Levels of explainable artificial intelligence for human-aligned conversational explanations.” Its abstract argues that most XAI/IML applications provide low-level explanations of how an individual decision was reached from a particular datum, but rarely provide insight into beliefs and motivations, hypotheses about others’ intentions, interpretation of cultural expectations, or the process used to generate the explanation itself (Dazeley et al., 2021). This situates Explanatory AI as a search for explanatory depth rather than only local attribution.

3. Methodological families

A major methodological divide concerns what counts as an instance-wise explanation. Camburu et al. formalize two prevalent perspectives. In the feature-additivity perspective, an explanation is a set of contributions $w_i^x(f)$ satisfying

$\sum_{i=1}^d w_i^x(f)\approx f(x).$

In the feature-selection perspective, an explanation is a subset $S(x)\subseteq\{1,\dots,d\}$ such that

$f(x_{S(x)})\approx f(x).$

The paper shows formally that these two views disagree on single examples and therefore should not be directly compared as if they answered the same explanatory question (Camburu et al., 2019). This distinction is foundational for Explanatory AI because it clarifies that “why this output?” can mean contribution decomposition, sufficient evidence, contrastive recourse, or something else entirely.

Within the post-hoc tradition, saliency maps, LIME, SHAP, and related feature-based methods remain central. “Explaining Explanations to Society” describes gradient-based saliency as

$S_c(x)=\frac{\partial f_c(x)}{\partial x},$

and counterfactual explanation as the search for a minimally perturbed input

$$\min_{x'} \|x'-x\|_p \quad \text{s.t.}\quad f(x')=y'.$$

The same paper discusses Concept Activation Vectors as a way to translate hidden activations into human-meaningful concepts through directional derivatives in representation space (Gilpin et al., 2019). These methods are often useful, but the paper emphasizes that they do not automatically yield outside explanations that are precise and understandable to non-experts.

A separate line of work builds explanatory structure into the model itself. “ExplAIn” for diabetic retinopathy diagnosis learns to segment and categorize lesions in fundus images, and the final image-level classification directly derives from these multivariate lesion segmentations; the architecture is trained end to end with image supervision only, so lesion concepts and lesion categories emerge without manual pixel annotation (Quellec et al., 2020). The reported explanation is simple by design: an image and/or a few sentences identifying the lesion categories and their contribution to the diabetic-retinopathy grade.

Other work treats explanation as learned interpretation rather than post-hoc approximation. “Explanatory Learning” defines an unknown interpreter $I^\*\!:\Sigma\times U\to\{0,1\}$ over symbolic explanations and observations, and introduces Critical Rationalist Networks with a learned interpreter, a conjecture generator, adjustable test-time inference budget, and the ability to abstain when no fully consistent conjecture is found (Norelli et al., 2022). In a different direction, “Knowledge-intensive Language Understanding for Explainable AI” argues that low-level features and activations must be translated into explicit domain knowledge that humans already understand and use, including peer-validated guidelines and domain knowledge graphs (Sheth et al., 2021). Together, these approaches move explanatory work from after-the-fact visualization toward interpretable symbolic mediation.

4. Interactive, conversational, and human-in-the-loop systems

Explanatory AI is frequently implemented as an interface, dialogue, or steering loop rather than a static artifact. Sovrano and Vitali adapt Achinstein’s view that to explain $x$ is to provide a pragmatically appropriate, content-giving answer to a question about $$(RC \rightarrow R)\;\land\;(R \rightarrow \mathrm{Outcome})\;\land\;(EC \rightarrow \mathrm{Outcome}),$$0, and operationalize it with seven archetypal questions: Why, What for, How, When, Where, What, Who (Sovrano et al., 2021). Their system structures documents into knowledge graphs, generates archetype-based overviews, and presents explanation as interactive exploration rather than one-shot answer generation. In a user study with more than 100 participants, the interactive extension produced a statistically relevant improvement on effectiveness over the baseline, with $$(RC \rightarrow R)\;\land\;(R \rightarrow \mathrm{Outcome})\;\land\;(EC \rightarrow \mathrm{Outcome}),$$1 and $$(RC \rightarrow R)\;\land\;(R \rightarrow \mathrm{Outcome})\;\land\;(EC \rightarrow \mathrm{Outcome}),$$2 (Sovrano et al., 2021).

IXAII exemplifies a more explicitly multi-method interface. It provides explanations from four XAI methods—LIME, SHAP, Anchors, and DiCE—offers tailored views for five user groups, and gives users agency over explanation content and format (Speckmann et al., 26 Jun 2025). The design premise is that static post-hoc explanations neglect the user perspective; accordingly, IXAII includes a perspective selector, format selector, and explanation guide mapping user questions such as “Why,” “Why Not,” “What If,” and “When” to appropriate methods. Its evaluation through interviews with experts and lay users found that the combination of multiple explanation methods, visualization options, and interactivity was perceived as helpful for transparency (Speckmann et al., 26 Jun 2025).

In high-stakes settings, interactive explanation is often linked to control. Bhattacharya et al.’s Explanatory Model Steering system combines data-centric and model-centric explanations in a dashboard that lets healthcare experts steer prediction models through manual and automated data-configuration approaches (Bhattacharya et al., 2024). The steering loop is explicit: users inspect explanations, modify data configuration, trigger retraining, and receive updated explanations. An industrial quality-control system follows a related pattern by combining CNN-based defect localization with inductive logic programming rules, then allowing experts to accept, reject, or partially correct the justification atoms that supported a defect classification (Müller et al., 2022). In both cases, explanation is part of a collaborative workflow that preserves user agency and accountability rather than merely annotating a final prediction.

Hybrid cognitive architectures push this idea further. Nirenburg et al. describe a strategic symbolic layer and a tactical data-driven layer, with autogenerated “under-the-hood” panels for text-meaning representation, thoughts, agenda, and visual-meaning representation (Nirenburg et al., 2024). Because the reasoning trace is symbolic and inspectable, explanation is not a post-hoc gloss but a direct view into the actual inference process. This suggests a version of Explanatory AI in which interaction design and system architecture are inseparable.

5. Evaluation, verification, and explanatory effectiveness

Evaluation in Explanatory AI is not limited to plausibility or user preference; a recurrent concern is whether explanations are faithful to the actual causal structure of the system. Camburu et al. propose a verification framework for feature-selection explainers using a rationalizing RCNN on BeerAdvocate reviews, where the model’s architecture and pruning conditions make it possible to certify which tokens have zero contribution and to identify at least one clearly relevant token per retained instance (Camburu et al., 2019). They define three error metrics—$$(RC \rightarrow R)\;\land\;(R \rightarrow \mathrm{Outcome})\;\land\;(EC \rightarrow \mathrm{Outcome}),$$3, $$(RC \rightarrow R)\;\land\;(R \rightarrow \mathrm{Outcome})\;\land\;(EC \rightarrow \mathrm{Outcome}),$$4, and $$(RC \rightarrow R)\;\land\;(R \rightarrow \mathrm{Outcome})\;\land\;(EC \rightarrow \mathrm{Outcome}),$$5—and show that all tested explainers commit non-negligible failures. On $$(RC \rightarrow R)\;\land\;(R \rightarrow \mathrm{Outcome})\;\land\;(EC \rightarrow \mathrm{Outcome}),$$6, zero-contribution tokens are selected as most important up to 14.8% of the time for LIME on the aroma aspect and 12.95% for L2X; SHAP is strongest overall but still misranks at least one irrelevant token in 9–16% of instances (Camburu et al., 2019). The importance of this result is methodological: post-hoc explanation cannot be assumed correct merely because the predictive model is accurate.

A second evaluation strand measures explanation relative to the explainee. Jacovi et al. define a successful explanation as one for which the explainee’s mental model is coherent with additional observations of the AI system’s behavior (Jacovi et al., 2022). Formally, if $$(RC \rightarrow R)\;\land\;(R \rightarrow \mathrm{Outcome})\;\land\;(EC \rightarrow \mathrm{Outcome}),$$7 is the explainee’s mental model and $$(RC \rightarrow R)\;\land\;(R \rightarrow \mathrm{Outcome})\;\land\;(EC \rightarrow \mathrm{Outcome}),$$8 is a set of contrast cases, success requires $$(RC \rightarrow R)\;\land\;(R \rightarrow \mathrm{Outcome})\;\land\;(EC \rightarrow \mathrm{Outcome}),$$9 for all $w_i^x(f)$0. Cope and McBurney go further by casting explanation as a two-player cooperative game and defining explanatory effectiveness as

$w_i^x(f)$1

where $w_i^x(f)$2 combines compression, utilization, integration, mutual information with the explanandum, and the explainee’s self-report (Cope et al., 2023). Although the measure is theoretical and its components are not directly computable, it provides a formal target for explainee-centered assessment rather than model-centered evaluation alone.

Empirical system-level evaluations reinforce this shift. The Explanatory Model Steering system was evaluated with 174 healthcare experts across three user studies, including a repeated-measures ANOVA showing a significant main effect of explanation type on bias-detection accuracy, $w_i^x(f)$3, and post-hoc Tukey tests showing that manual steering improved accuracy more for high-experience users, $w_i^x(f)$4 (Bhattacharya et al., 2024). Such results do not eliminate the need for fidelity testing, but they show that explanatory quality is also a property of collaborative performance, trust calibration, and task improvement.

6. Domains, limitations, and research directions

Medical imaging provides one of the clearest demonstrations of built-in explanatory modeling. In diabetic-retinopathy diagnosis, ExplAIn reported image-level AUCs of 0.9682 for “$w_i^x(f)$5 mild NPDR,” 0.9939 for “$w_i^x(f)$6 moderate NPDR,” 0.9978 for “$w_i^x(f)$7 severe NPDR,” and 0.9945 for “PDR” on OPHDIAT$w_i^x(f)$8, while also producing lesion maps for hemorrhages, exudates, microaneurysms, and advanced DR signs (Quellec et al., 2020). Industrial quality control offers a contrasting case in which explanatory rules, superpixel descriptors, and direct expert correction are combined in a human-in-the-loop classification system (Müller et al., 2022). Search-and-retrieve robotics, credit approval, diabetes-risk prediction, and general decision-support interfaces extend the same pattern into domains where explanation must support oversight, correction, and responsibility rather than only retrospective interpretation (Nirenburg et al., 2024, Bhattacharya et al., 2024, Sovrano et al., 2021).

The literature also identifies persistent limitations. “Explaining Explanations to Society” emphasizes the trade-off between precision and understandability, the tension between local and global scope, and the incompleteness of current tools for sensitivity analysis or composite system failures (Gilpin et al., 2019). Jacovi et al. note that nearly no method explicitly distinguishes External Causes from factors that shaped the system’s internal representation, leaving a systematic gap in explanatory narratives (Jacovi et al., 2022). “Explaining Explaining” argues that post-hoc rationales are “neither the actual causal reasons nor guaranteed faithful to the model’s inner workings” when the underlying system remains a black box (Nirenburg et al., 2024). More recent work on Explanatory AI through generative AI adds a different concern: the “transparency paradox,” in which excessive technical detail harms comprehension, motivating progressive disclosure and adaptive communication instead of uniform disclosure (Meske et al., 8 Aug 2025).

Current research directions increasingly extend explanation beyond final predictions. Holistic Explainable AI (HXAI) embeds explanation into six components of the machine-learning workflow—data, analysis set-up, learning process, model output, model quality, and communication channel—and organizes stakeholder needs through a 112-item question bank (Paterakis et al., 15 Aug 2025). In parallel, generative Explanatory AI emphasizes narrative communication, adaptive personalization, and universal accessibility across languages, cultures, and impairment profiles (Meske et al., 8 Aug 2025). Hybrid knowledge-based agents retain the strongest claim to faithful explanation because they can expose the actual symbolic trace of reasoning, but they face scaling and knowledge-engineering challenges (Nirenburg et al., 2024). A plausible implication is that Explanatory AI is evolving from a subtopic of local post-hoc interpretability into a broader architecture-and-interaction paradigm concerned with how explanations are generated, delivered, contested, and used across the full sociotechnical lifecycle of AI systems.