Self-Explaining Neural Networks

• Self-Explaining Neural Networks (SENNs) are inherently interpretable models that decompose inputs into concepts and relevance scores to provide both predictions and explanations.
• They enforce design constraints such as local linearity, stability, and sparsity to ensure that explanations reliably reflect the impact of input features.
• Extensions include quantized, contrastive, and domain-specific adaptations, enhancing interpretability in areas like time-series analysis, survival analysis, and reinforcement learning.

Self-Explaining Neural Networks (SENNs) are intrinsically interpretable predictive models in which explanation is embedded in the predictor’s functional form rather than attached post hoc. In the canonical construction, a model maps an input into interpretable concepts, assigns input-dependent relevance scores to those concepts, and aggregates the resulting concept contributions into the prediction, so that each forward pass yields both an output and a structured explanation of that output. The framework was introduced to reconcile model complexity with explicitness, faithfulness, and stability of explanations (Alvarez-Melis et al., 2018). Subsequent work has broadened the SENN paradigm toward quantized concept-class relations, contrastive concept learning, uncertainty-aware concept bottlenecks, sufficient-subset explanations, phrase-based text explanations, time-series causal inference, survival analysis, business process monitoring, and reinforcement learning (Norrenbrock et al., 2023, Sawada et al., 2022, Qian et al., 2024, Bassan et al., 5 Feb 2025, Rajagopal et al., 2021, Marcinkevičs et al., 2021, Alabdallah et al., 2024, Bassan et al., 23 Mar 2025, Nowosadko et al., 18 Sep 2025).

1. Conceptual basis and departure from post-hoc interpretability

The original SENN program was formulated against the dominant paradigm of post-hoc explanation, in which a predictor is trained first and an explanatory method is applied afterward. The central objection is that post-hoc methods such as saliency maps, LIME, SHAP, integrated gradients, and related attribution schemes do not constrain the predictor itself to remain interpretable. SENNs therefore shift interpretability into model design: the explanation is part of the forward computation, not an auxiliary approximation of an opaque mechanism (Alvarez-Melis et al., 2018).

This architectural position was sharpened by later arguments for “self-explaining AI” as an alternative to interpretable AI in the usual post-hoc sense. That work contends that deep networks trained on complex real-world data are better viewed as flexible interpolators than as systems that extract a small number of stable, human-readable rules; on that view, post-hoc explanations can become plausible narratives without mechanism-level fidelity. It therefore defines self-explaining systems as models that output both a prediction and an explanation, with the additional requirement that the explanation be genuinely related to the decision mechanism rather than merely co-produced by an unrelated explanation branch. To operationalize that requirement, it proposes mutual-information-based relatedness measures between latent units, diagnoses, and explanation attributes, including

$R(A_j) = \sum\limits_i^N \mbox{MI}(L_i, D)\mbox{MI}(L_i, A_j),$

and argues that self-explaining systems should also expose confidence levels and an applicability-domain warning for extrapolative inputs (Elton, 2020).

Within this broader intrinsic-interpretability agenda, SENNs occupy a specific formal niche. They do not merely demand that explanations accompany predictions; they require that the prediction itself decompose into interpretable intermediate quantities whose semantics can be regularized, interrogated, and, in some variants, grounded to prototypes, phrases, or concept vocabularies.

2. Canonical formulation, constraints, and learning objective

The canonical SENN starts from a staged generalization of linear models. A standard linear predictor

$f(x) = \sum_i^n \theta_i x_i + \theta_0$

is first generalized to an input-dependent linear form, then to an interpretable concept space. The resulting predictor takes the form

$$f(x) = g\bigl(\theta_1(x) h_1(x), \dots, \theta_k(x) h_k(x)\bigr),$$

with the simplest case

$$f(x) = \theta(x)^T h(x) = \sum_{i=1}^K \theta_i(x)\, h_i(x).$$

Here, $h(x)$ is a concept encoder, $\theta(x)$ is a parametrizer producing relevance scores, and $g$ is an aggregation function (Alvarez-Melis et al., 2018).

The explanation associated with a prediction is the set

$$\mathcal{E}_f(x) = \{(h_i(x), \theta_i(x))\}_{i=1}^k,$$

so explicitness is obtained by exposing both concept activations and their local importances. In the foundational formulation, interpretability places restrictions on $g$: it should remain monotone, completely additively separable, and non-interacting in a way that preserves the sign and scale of individual concept contributions. Addition is the most direct case, but affine variants with nonnegative partial derivatives are also admissible (Alvarez-Melis et al., 2018).

Faithfulness and stability are enforced through a local linearity requirement in concept space. The core regularizer is

$$\mathcal{L}_{\theta}(f(x)) := \left\| \nabla_x f(x) - \theta(x)^\top J_x^h(x) \right\|,$$

which encourages the relevance vector to behave like the local derivative of the predictor with respect to concepts. This is coupled with the notion of being difference-bounded by the concept map $f(x) = \sum_i^n \theta_i x_i + \theta_0$0: a function $f(x) = \sum_i^n \theta_i x_i + \theta_0$1 is difference-bounded by $f(x) = \sum_i^n \theta_i x_i + \theta_0$2 if

$f(x) = \sum_i^n \theta_i x_i + \theta_0$3

A local version imposes the same condition in a neighborhood of each input. The full training objective is

$f(x) = \sum_i^n \theta_i x_i + \theta_0$4

where $f(x) = \sum_i^n \theta_i x_i + \theta_0$5 is the task loss, $f(x) = \sum_i^n \theta_i x_i + \theta_0$6 is the stability regularizer, and $f(x) = \sum_i^n \theta_i x_i + \theta_0$7 is a reconstruction loss for the concept encoder-decoder pair (Alvarez-Melis et al., 2018).

When concepts are learned rather than prescribed, the original framework adds three auxiliary desiderata. First, fidelity is encouraged by autoencoding, so that the concept bottleneck preserves information required for prediction. Second, diversity is encouraged by sparsity, so that only a small number of concepts are active for a given sample. Third, grounding is supplied by prototypes, typically examples that maximally activate a concept. These mechanisms already foreshadow later SENN variants in which concept quality, rather than merely coefficient stability, becomes the principal research problem (Alvarez-Melis et al., 2018).

3. Concept quality: quantization, contrastive learning, and cross-domain grounding

A major line of later work keeps the canonical “concepts plus relevance” template but modifies the structure of the concept layer to make explanations more compact, human-readable, or transferable. Q-SENN is a particularly explicit example. It retains the SENN philosophy that predictions should be linear combinations of interpretable concepts, but constrains the final concept-to-class matrix to ternary values,

$f(x) = \sum_i^n \theta_i x_i + \theta_0$8

so each class-feature relation is positive, negative, or neutral. Its final classifier is

$f(x) = \sum_i^n \theta_i x_i + \theta_0$9

The model is trained through an iterative dense-train, sparse-fit, quantize, and fine-tune loop, using glm-saga to obtain a sparse decision layer before quantization. Q-SENN reiterates the SENN desiderata of fidelity, diversity, and grounding, reports about 5 interpretable features per class on average, and adds a CLIP-based alignment procedure to map learned features to language prompts without additional supervision (Norrenbrock et al., 2023).

Contrastive learning has also been used to improve the readability of learned concepts. C-SENN replaces decoder- or discriminator-style concept pressure with supervised contrastive learning and Barlow Twins regularization. The model keeps the standard self-explaining decomposition

$$f(x) = g\bigl(\theta_1(x) h_1(x), \dots, \theta_k(x) h_k(x)\bigr),$$0

but uses object-region masking derived from Faster R-CNN detections to create augmented views, supervised contrastive learning to pull same-label concept representations together, and Barlow Twins to reduce redundancy across concept dimensions. On the BDD-OIA driving dataset, C-SENN reported an mF1 of 0.618, compared with 0.539 for M-SENN and 0.610 for a concept bottleneck model, while qualitative analyses showed more localized concept saliency maps (Sawada et al., 2022).

A related but distinct development treats concept learning itself as a domain-generalization problem. The Representative Concept Extraction framework preserves the SENN pattern of concept encoder, relevance network, and concept-based aggregation, but augments it with a salient concept selection network $$f(x) = g\bigl(\theta_1(x) h_1(x), \dots, \theta_k(x) h_k(x)\bigr),$$1, contrastive concept learning, and prototype-based concept grounding. Its prediction combines a relevance-weighted path and a saliency-driven path:

$$f(x) = g\bigl(\theta_1(x) h_1(x), \dots, \theta_k(x) h_k(x)\bigr),$$2

The model is trained with reconstruction, prediction, self-supervised contrastive, grounding, and same-class fidelity objectives, and evaluates concept fidelity through concept overlap and concept interoperability through domain adaptation performance. This extends the original SENN concern with local interpretability toward class-consistent and domain-invariant concept semantics (Sinha et al., 2024).

Taken together, these variants indicate that the concept map $$f(x) = g\bigl(\theta_1(x) h_1(x), \dots, \theta_k(x) h_k(x)\bigr),$$3 is no longer treated as a neutral intermediate representation. It is a design object whose sparsity pattern, sign structure, contrastive geometry, and linguistic grounding materially determine whether a SENN is interpretable in practice.

4. Alternative explanation objects: sufficient subsets and phrase-level concepts

Not all self-explaining architectures preserve the canonical additive concept-weight decomposition. A substantial branch of later work redefines the explanation object itself. In Sufficient Subset Training (SST), the model outputs both a class prediction and a feature-selection mask,

$$f(x) = g\bigl(\theta_1(x) h_1(x), \dots, \theta_k(x) h_k(x)\bigr),$$4

from which a subset is extracted by thresholding:

$$f(x) = g\bigl(\theta_1(x) h_1(x), \dots, \theta_k(x) h_k(x)\bigr),$$5

The explanation is therefore a sufficient reason rather than an attribution vector. The training objective

$$f(x) = g\bigl(\theta_1(x) h_1(x), \dots, \theta_k(x) h_k(x)\bigr),$$6

combines predictive accuracy, invariance of the predicted label under masking outside $$f(x) = g\bigl(\theta_1(x) h_1(x), \dots, \theta_k(x) h_k(x)\bigr),$$7, and cardinality control via an $$f(x) = g\bigl(\theta_1(x) h_1(x), \dots, \theta_k(x) h_k(x)\bigr),$$8 penalty. SST formalizes baseline, robust, and probabilistic sufficient reasons, proves hardness results for post-hoc minimal sufficient reasons, and argues that explanation generation should therefore be learned rather than searched for after training (Bassan et al., 5 Feb 2025).

A closely related instantiation appears in predictive business process monitoring. There, a three-layer LSTM next-activity predictor is augmented with an explanation output $$f(x) = g\bigl(\theta_1(x) h_1(x), \dots, \theta_k(x) h_k(x)\bigr),$$9 having the same dimensionality as the input trace tensor. Scores are squashed to $$f(x) = \theta(x)^T h(x) = \sum_{i=1}^K \theta_i(x)\, h_i(x).$$0, thresholded at $$f(x) = \theta(x)^T h(x) = \sum_{i=1}^K \theta_i(x)\, h_i(x).$$1, and interpreted as a sparse sufficient subset of trace features:

$$f(x) = \theta(x)^T h(x) = \sum_{i=1}^K \theta_i(x)\, h_i(x).$$2

Training again combines standard predictive loss with a masking-based faithfulness term and a cardinality penalty,

$$f(x) = \theta(x)^T h(x) = \sum_{i=1}^K \theta_i(x)\, h_i(x).$$3

On real-world event logs, this self-explaining LSTM produced explanations far faster than Anchors and with substantially higher sufficient-explanation percentages, while maintaining predictive performance close to the non-explanatory baseline (Bassan et al., 23 Mar 2025).

Text classification has produced another departure from the classical SENN template. SelfExplain uses phrase-based concepts extracted from constituency parse trees, together with a Local Interpretability Layer and a Global Interpretability Layer. Phrases are represented by the mean of their constituent token vectors; local relevance is computed relative to the predicted class by comparing the full-sentence label distribution to a phrase-specific distribution, while global explanations retrieve influential training phrases via cosine similarity and maximum inner product search. The total objective jointly optimizes the main classifier and both explanation layers,

$$f(x) = \theta(x)^T h(x) = \sum_{i=1}^K \theta_i(x)\, h_i(x).$$4

Across five text-classification datasets, the model maintained or slightly improved performance, and a human evaluation with 14 graduate student judges on 50 SST-2 validation samples reported higher adequate justification, understandability, and trust than saliency and influence-function baselines (Rajagopal et al., 2021).

These models are self-explaining in the architectural sense, but they diversify the meaning of explanation. In classical SENNs, the explanation is a signed local decomposition over concepts. In sufficient-reason and phrase-retrieval models, the explanation is instead a sparse subset or a ranked set of human-readable textual units. This suggests that SENNs are best understood as a family of intrinsic explanation designs rather than a single immutable equation.

5. Domain-specific adaptations

The SENN template has been specialized across several prediction regimes by redefining what counts as a concept and what explanatory object should be emitted.

Domain	SENN-style model	Explanation object
Multivariate time series	GVAR	Lag-specific generalized coefficient matrices
Survival analysis	CoxSE / CoxSENAM	Input-dependent local feature weights for log-risk
Reinforcement learning	SENN actor in PPO	Relevance scores and feature-wise effects

For nonlinear Granger causality, GVAR recasts the next-step forecast as a SENN-like autoregression:

$$f(x) = \theta(x)^T h(x) = \sum_{i=1}^K \theta_i(x)\, h_i(x).$$5

Each lag network $$f(x) = \theta(x)^T h(x) = \sum_{i=1}^K \theta_i(x)\, h_i(x).$$6 outputs a $$f(x) = \theta(x)^T h(x) = \sum_{i=1}^K \theta_i(x)\, h_i(x).$$7 generalized coefficient matrix, so the model explains every forecast in terms of source-target-lag interactions. Training uses prediction error, an elastic-net sparsity penalty, and a temporal smoothing penalty on consecutive coefficient matrices. Unlike sparse-input neural Granger models, GVAR yields not only causal structure but also effect sign and temporal variation directly from the learned coefficients. On simulated data it performed competitively for causal discovery and substantially better for sign recovery, with especially strong results on Lotka–Volterra sign detection (Marcinkevičs et al., 2021).

In survival analysis, CoxSE and CoxSENAM adapt SENN local linearity to the Cox proportional hazards setting by replacing the linear log-risk with an input-dependent local linear model. CoxSE defines

$$f(x) = \theta(x)^T h(x) = \sum_{i=1}^K \theta_i(x)\, h_i(x).$$8

so each local weight depends on the entire feature vector and can encode interactions. CoxSENAM modifies this to

$$f(x) = \theta(x)^T h(x) = \sum_{i=1}^K \theta_i(x)\, h_i(x).$$9

recovering a NAM-like additive structure while retaining SENN-style regularization. Both models add a gradient-consistency penalty

$h(x)$0

and an $h(x)$1 sparsity term on local weights. On synthetic and real survival datasets, CoxSE matched DeepSurv when interactions mattered, while CoxSENAM improved robustness to non-informative features; the hybrid model was the most robust among the SENN-based variants (Alabdallah et al., 2024).

In reinforcement learning, SENNs have been used as the policy actor inside PPO for mobile-network resource allocation. Because the state is low-dimensional, the model adopts an identity conceptizer and a biased local-linear policy,

$h(x)$2

The explanation consists of relevance scores and feature-wise effect scores

$h(x)$3

A SENN robustness penalty analogous to the canonical Jacobian-matching loss constrains local explanation stability, while global explanations are extracted from local relevance vectors through effect distributions and K-means clustering. In this low-dimensional setting, biased SENNs performed only slightly worse than regular DNNs, outperformed regular SENNs, and produced lower estimated Lipschitz values than the unbiased version (Nowosadko et al., 18 Sep 2025).

These domain adaptations preserve the central SENN thesis: the explanatory parameters should be part of the predictive function. What changes from domain to domain is the ontology of concepts—lags, covariates, or state features—and the most useful level of explanation granularity.

6. Uncertainty, evaluation, and unresolved issues

A limitation of early SENNs was that they typically produced point explanations without calibrated uncertainty. The unSENN framework addresses this by applying conformal prediction to both the interpretation layer and the label layer. For concept-supervised SENNs, it calibrates a concept nonconformity score over the top-$h(x)$4 concept probabilities and constructs concept prediction sets with marginal coverage guarantee under exchangeability. It then transfers concept uncertainty to the label space by searching over latent concept vectors within the calibrated radius, yielding label prediction sets with the same distribution-free guarantee. The same idea is extended to prototype-based SENNs by defining nonconformity in latent prototype space (Qian et al., 2024).

The evaluation of SENNs has therefore become multi-criterial. The original framework emphasized explicitness, faithfulness, and stability, with stability estimated by a local Lipschitz-style robustness score over explanation maps and faithfulness assessed by concept or feature removal (Alvarez-Melis et al., 2018). Later vision work added diversity@5, dependence, binary-feature behavior, and concept-language alignment as grounding metrics, while showing that quantization and sparsity can improve both diversity and grounding relative to dense baselines (Norrenbrock et al., 2023). Survival-analysis variants emphasized robustness to non-informative features and agreement with SHAP rankings, revealing that additive architectures can be robust without necessarily yielding faithful local explanations (Alabdallah et al., 2024). Text and business-process models instead evaluated sufficiency, explanation size, and computation time, reflecting their interpretation of explanation as a sufficient subset or phrase set rather than a local linear decomposition (Rajagopal et al., 2021, Bassan et al., 23 Mar 2025).

Several controversies remain open. First, explanation quality is still bottlenecked by concept quality: if the learned concepts are semantically weak, the relevance scores can be locally stable yet not especially meaningful. Second, faithfulness is not guaranteed merely by adding an explanation head; the explanation must remain causally or informationally tied to the predictive pathway, a concern that has motivated mutual-information diagnostics and explicit calls for confidence estimates and applicability-domain warnings (Elton, 2020). Third, different SENN variants trade off distinct desiderata. Q-SENN shows that ternary class-feature relations can improve readability, but at the cost of an iterative rather than fully end-to-end quantized decision layer. C-SENN improves concept localization, yet also notes that the ideal form of a human-readable concept is not formally settled and that the number of concepts remains a design choice. GVAR still requires thresholding because generalized coefficients are not exactly sparse. CoxSE gains expressiveness through interaction-sensitive local weights but is less robust to noisy features than NAM-style models. RL adaptations remain most persuasive in low-dimensional settings, and even there the parametrizer is acknowledged to remain somewhat opaque (Norrenbrock et al., 2023, Sawada et al., 2022, Marcinkevičs et al., 2021, Alabdallah et al., 2024, Nowosadko et al., 18 Sep 2025).

The current literature therefore supports a precise but nontrivial view of SENNs. They are not simply “interpretable neural networks,” nor are they reducible to one architecture. They are a design principle: prediction should proceed through explicit explanatory variables whose semantics, local validity, uncertainty, and stability can be engineered and tested. Subsequent work suggests that the strength of the paradigm lies less in any single decomposition than in the insistence that explanation be structurally internal to inference itself.