---
title: Unified Formulation of Interpretable Inference
url: https://www.emergentmind.com/topics/unified-formulation-of-interpretable-inference
type: topic
---

# Unified Formulation of Interpretable Inference

A unified formulation of interpretable inference seeks to subsume diverse interpretability methodologies within a precise, formal framework—allowing systematic comparison, principled design, and guarantees on stability, fidelity, and validity. Recent advances provide concrete mathematical structures for achieving this unification, spanning model-agnostic explanation pipelines, decision-theoretic Bayesian surrogates, additive feature attribution, probabilistic latent-variable approaches, inferential models with non-additive beliefs, symmetry-driven axioms, and frameworks for property-based scientific enquiry. This article surveys and synthesizes these main lines of research.

## 1. Model-Agnostic Unified Pipelines: The SIPA Framework

The SIPA (Sampling, Intervention, Prediction, Aggregation) framework provides a universal abstraction for model-agnostic interpretability techniques. In this paradigm, any post-hoc interpretation procedure is decomposed into four canonical stages [1904.03959]:

1. **Sampling:** Draw K “perturbation indicators” $z^{(k)}$ from a distribution $\mathcal{S}(z\,|\,x,S)$, each encoding which features to intervene upon and how.
2. **Intervention:** Apply an operator $I(x,z,S)$ that perturbs features in $S$ according to $z$.
3. **Prediction:** Evaluate the black-box model at each intervened input, yielding $\hat y^{(k)} = f(\tilde x^{(k)})$.
4. **Aggregation:** Use an operator $A$ to combine the tuples $(z^{(k)}, \tilde x^{(k)}, \hat y^{(k)}, w^{(k)})$ into a final interpretation, such as attributions or importance.

Many canonical methods fall under SIPA by instantiating each stage differently:

| Method              | Sampling                  | Intervention               | Aggregation                       |
|---------------------|--------------------------|----------------------------|-----------------------------------|
| LIME                | Binary local perturbation | Reference-value imputation | Weighted least squares surrogate  |
| Shapley (SHAP)      | Feature orderings         | Set-to-reference          | Combinatorial weighted averaging  |
| Permutation FI      | Permuted feature values   | Shuffle column             | Loss difference                   |
| Variance-based FI   | Grid over feature         | Set single feature value   | Empirical variance                |

Extrapolation risks arise from marginal-based sampling; combinatorial cost (notably in Shapley-type methods, $\mathcal{O}(p2^p)$) is a computational limitation [1904.03959].


## 2. Additive Feature Attribution and Axiomatic Unification

A major alternative unification is the additive feature attribution framework, which postulates that faithful explanations must be locally additive in a binary interpretable space [1611.07478]. For an interpretable vector $x' \in \{0,1\}^M$, any explanation is modeled as
$$
g(x') = \phi_0 + \sum_{i=1}^M \phi_i x'_i
$$
The uniquely justified set of attributions $\phi$ satisfying local accuracy, missingness, and consistency are the Shapley values:
$$
\phi_i = \sum_{S \subseteq N\setminus \{i\}} \frac{|S|!(M - |S| - 1)!}{M!} [f_x(S \cup \{i\}) - f_x(S)]
$$
Here $f_x(S)$ represents the conditional expectation of $f$ given the presence of features in $S$.

This result connects LIME, DeepLIFT, LRP, and other explanation methods as either exact or approximate Shapley value estimators under particular kernels and reference choices. KernelSHAP’s regression formulation, for example, is provably equivalent when using the Shapley kernel [1611.07478].


## 3. Decision-Theoretic and Bayesian Unification

A unified decision-theoretic Bayesian approach defines interpretability as a utility function balancing fidelity and simplicity [1910.09358]. Given a reference predictive distribution $p_\text{ref}(y\,|\,x,\mathcal{D})$, an interpretable model $\psi$ is chosen to maximize:
$$
U(\psi) = \mathbb E_{x \sim \pi}\left[\mathbb E_{y \sim p_\text{ref}(\cdot\,|\,x,\mathcal{D})} [\log p_\psi(y\,|\,x)]\right] - \lambda I(\psi)
$$
where $I(\psi)$ quantifies interpretability (tree size, sparsity, etc.), and $\pi$ assigns importance to regions of input space (global or local). The method is model-agnostic and admits arbitrary reference and proxy families. Maximizing $U(\psi)$ is equivalent to minimizing expected KL divergence plus an interpretability penalty.

Stability is quantified by measuring the mean dissimilarity between interpretable proxies across bootstrapped datasets (Briand–Emonet distance). Experimental results confirm that utility-based surrogates dominate prior-restricted proxies in fidelity-for-complexity tradeoff and stability [1910.09358].


## 4. Unified Probabilistic and Latent-Variable Models

The LEX (Latent EXplanation) framework casts interpretable inference as a latent variable model: $z \sim p_\theta(z|x)$, $x̃ \sim p(x̃|x, z)$, $y \sim p_\phi(y|x̃)$, where $z$ is a mask encoding feature relevance [2212.03131]. The joint and marginal likelihoods allow learning interpretable selectors and imputers by maximum likelihood (or regularized variants), unifying L2X, INVASE, REAL-X, rationale selection, LIME, SHAP, and occlusion within a single probabilistic structure. Inference and explanation are amortized; multiple imputation strategies yield robust, less artifact-prone masks.

Popular instance-wise feature selection methods emerge as specific choices of regularization and imputation. Multiple imputation (e.g., via VAEAC or mixtures) is essential for high true positive rate (TPR) and low false discovery rate (FDR) when ground-truth masks exist [2212.03131].


## 5. Symmetry-Based and Bayesian-Inversion Principles

An emerging perspective defines interpretable inference in terms of four symmetries: inference-equivariance, information-invariance, concept-closure invariance, and structural invariance [2601.12913]. These symmetries, formalized as group actions or functorial constraints, enforce that explanations are simulatable, compressive, semantically aligned, and user-structural.

Every interpretable model is forced to factor as $X \to Z \to C \to Y$, where $Z$ compresses to task-relevant information, $C$ expresses human concepts, and all Bayes-inversion queries (alignment, intervention, counterfactual) are special cases of
$$
p(z|x) \propto p(x|z)p(z)
$$
subject to the imposed symmetries.

This framework provides a categorical characterization of interpretability, subsuming concept-bottleneck models and making actionable interpretability a property of the model's symmetry structure rather than a set of informal desiderata [2601.12913].


## 6. Scientific Inference With Interpretable Machine Learning

A property-descriptor framework formalizes the process of interpreting machine learning models to draw conclusions about the scientific data-generating process [2206.05487]. The property descriptor $g_K: M \to \mathcal Q$ maps a model $m$ to a quantity $Q$ (e.g., partial dependence curve, feature importance) such that:

- **Identification:** $g_K(m^*) = Q$, recovering the true scientific property from the optimal predictor.
- **Continuity:** If $m \approx m^*$ in risk, $g_K(m) \approx Q$.
- **Estimability:** There exists an unbiased estimator $\hat g_\mathcal{D^*}$ for $g_K(m)$.
- **Uncertainty Quantification:** Both model error and estimation error are quantifiable in a bias–variance decomposition.

PDPs, PFI, Shapley values for risk, and local ICE/SHAP explanations are all expressible as property descriptors within this formalism, ensuring that post-hoc model interpretation is grounded in principles of statistical validity [2206.05487].


## 7. Broader Implications and Limitations

By reducing diverse interpretability techniques to unified mathematical forms—be it SIPA pipelines, additive-attribution axioms, decision-theoretic Bayesian projections, probabilistic latent-variable models, or symmetry-based categories—the field achieves the following:

- **Method integration:** Methods such as LIME, SHAP, LEX, and post-hoc surrogates become specific instantiations within general frameworks.
- **Comparison and extension:** One can analyze fidelity, stability, error control, and computational cost in a principled way.
- **Limitations:** Extrapolation arises in marginal-based samplings; combinatorial methods may be computationally intractable without approximations; additive models may not encompass every class of explanations; and not all desiderata are always mutually realizable (per symmetry constraints).

At its core, a unified formulation enables technically rigorous, cross-method comparison, and principled design for interpretable inference in complex models [1904.03959, 1611.07478, 1910.09358, 2212.03131, 2206.05487, 2601.12913].

Source: https://www.emergentmind.com/topics/unified-formulation-of-interpretable-inference