Model-Centered Interpretability

• Model-Centered Interpretability is a framework that defines understanding in terms of a model’s internal structure, predictor functions, and input-output relations.
• It formalizes interpretability as an optimization problem, balancing fidelity and complexity via surrogate extraction, active querying, and path constraints.
• The approach also integrates concept-based methods and targeted interventions to enhance transparency and improve explanation quality with measurable performance benefits.

Searching arXiv for recent and foundational papers on model-centered interpretability and related frameworks. Model-centered interpretability denotes a family of approaches that seeks understanding by centering the model itself: the predictor function it computes, the internal mechanisms that realize that function, or an interpretable proxy constrained to mimic its behavior. In one contemporary formulation, it comprises methods that extract insights about an internal mechanism of the system; in broader philosophical and methodological accounts, it concerns understanding the relation between inputs and outputs, reasoning about how changes in $X$ affect $Y$, and distinguishing transparent models from post-hoc explanatory artifacts (Calderon et al., 2024, Räz, 2022, Lipton, 2016).

1. Conceptual scope and taxonomic structure

Model-centered interpretability is not a monolithic property. A recurrent distinction separates transparency from post-hoc explanation. Transparency concerns “how the model works,” and is commonly decomposed into simulatability, decomposability, and algorithmic transparency. Post-hoc explanation concerns “what else the model can tell,” including natural-language rationales, visualizations, local surrogates, and example-based explanations (Lipton, 2016). A related semantic account argues that interpretability is not purely mathematical but semantic: internal representations, parameters, rules, or dimensions must link to concepts understandable by end users (Silva et al., 2019).

A second line of work defines interpretability in terms of the predictor function $f:X\to Y$. Under the notion of functional interpretability, understanding means grasping the relation between inputs and outputs so that one can reason about how changes in $X$ affect $Y$ and anticipate qualitative or quantitative consequences of inputs without re-training the model. This perspective places linear models and small decision trees at the high-interpretability end of an “interpretability spectrum,” followed by MARS and GAMs, and then highly expressive deep architectures (Räz, 2022).

The same literature also distinguishes where interpretability is introduced. Intra-model interpretability is embedded in the model’s data or algorithmic structure and favors transparency; extra-model interpretability adds a separate interpretability layer on top of an existing model and favors user-friendly abstractions at the expense of full transparency. On this basis, interpretable systems may be grouped as data models, algorithmic models, and hybrid models (Silva et al., 2019).

Within NLP and LLM analysis, model-centered methods are further organized by the mechanism they target. The main categories are feature-attribution methods for local input$\to$output mechanisms, probing and clustering for input$\to$internal mechanisms, and mechanistic interpretability for internal$\to$internal mechanisms such as neurons, heads, and circuits (Calderon et al., 2024). This taxonomy makes explicit that “model-centered” can refer either to transparent model classes or to white-box analysis of otherwise opaque systems.

2. Formalizations of fidelity, complexity, and interpretability cost

A central formal pattern is to pose interpretability as an optimization problem over a simpler hypothesis class. In surrogate extraction, one begins with a fixed trained classifier $f$ and seeks an interpretable model $g$ that mimics it. For decision-set surrogates, the quality of a decision set $Y$0 is written as

$Y$1

where $Y$2 is expected fidelity with respect to $Y$3 under the true input distribution and $Y$4 is the number of rules (Lu et al., 2019). In a more general extraction formulation, the target is

$Y$5

with $Y$6 an interpretable model family and $Y$7 a sampling distribution over inputs (Bastani et al., 2017).

A different formalization treats interpretability as a property of a path by which a model is constructed. Here a model space $Y$8 is equipped with an interpretable step operator $Y$9, and a model is judged by the best interpretable path leading to it. Standard proxies such as sparsity in linear models, number of splits in trees, and number of clusters in clustering appear as special cases of path-length minimization. The resulting family of losses

$f:X\to Y$0

yields a one-parameter account of path complexity and path quality, and enables the price of interpretability to be studied as a Pareto problem between predictive cost $f:X\to Y$1 and interpretability $f:X\to Y$2 (Bertsimas et al., 2019).

A decision-theoretic formulation makes the trade-off explicit through a utility on interpretable models. Given a predictive reference model $f:X\to Y$3 and an interpretable proxy $f:X\to Y$4, the utility is

$f:X\to Y$5

or equivalently expected log-likelihood under the reference minus an interpretability penalty $f:X\to Y$6. This separates data modeling from user preferences: first fit an accurate reference model, then optimize within an interpretable family to best mimic it under a fidelity–simplicity trade-off (Afrabandpey et al., 2019).

These formulations share a common structure. Fidelity is optimized against either a black-box predictor or a stepwise construction process, while interpretability enters through complexity penalties, path constraints, or explicit utility terms. This suggests that model-centered interpretability is often less a fixed model attribute than a constrained optimization criterion imposed on model construction or model approximation.

3. Surrogate extraction and active-query interpretation

A large branch of model-centered interpretability treats explanation as model extraction. The extracted model is not merely descriptive; as long as approximation quality is good, statistical properties of the complex model are reflected in the interpretable surrogate (Bastani et al., 2017). In this setting, a decision tree or rule set becomes a global explanatory object for a random forest, neural network, or control policy.

A key refinement is the observation that interpretation differs from standard supervised learning because the explainer can generate synthetic inputs and query the target model for labels. This is the active-query property. Rather than learning a surrogate passively from a fixed archive of $f:X\to Y$7 pairs, one can adaptively concentrate queries in regions of the input space that are most informative for improving the surrogate (Lu et al., 2019).

The canonical example is Active Decision Set Induction (ADS). ADS performs local search over the space of all decision sets. At each iteration it considers local edits—adding a rule, removing a rule, or modifying a rule—and uses LUCB-style confidence bounds to decide which action is best. Synthetic instances are generated by counterfactual pool-based sampling in the coverage regions of the most promising actions, and the black box is queried only where uncertainty about competing edits is greatest (Lu et al., 2019).

Empirically, this active design changes the fidelity–complexity frontier. On the UCI Adult dataset, with all methods forced to approximately 24 rules unless stated otherwise, ADS (active) attained accuracy $f:X\to Y$8, F1 $f:X\to Y$9, precision $X$0, recall $X$1, and 24 rules with average rule length $X$2; ADS non-active attained accuracy $X$3, F1 $X$4, and 29 rules; BRS+ attained accuracy $X$5 and required 57 rules (Lu et al., 2019). The reported trade-off curves show that higher $X$6, corresponding to more active sampling, pushes the F1-versus-number-of-rules curve up-and-left.

The extraction literature also provides formal guarantees. For tree extraction with active sampling from a fitted mixture of axis-aligned Gaussians, the extracted tree converges to the oracle greedy tree $X$7 as the per-node sample size grows. In the stated theorem, for any $X$8, sufficiently large $X$9 yields

$Y$0

with probability at least $Y$1 (Bastani et al., 2017). This does not solve the global interpretability problem, but it gives a precise asymptotic notion of fidelity for a widely used surrogate family.

4. Statistical-inference and functional-information viewpoints

Another formal strand casts interpretability as statistical inference. In LEX, the explanatory object is a binary selector $Y$2 over input features, produced by a selector network $Y$3. Selected features are retained, unselected ones are imputed through $Y$4, and prediction is made by $Y$5. The full model factorizes as

$Y$6

Training proceeds by maximum likelihood with regularization to avoid the trivial “select all” solution, yielding a unified account in which LIME, L2X, INVASE, REAL-X, and rationale selection appear as special cases (Senetaire et al., 2022).

In this framework, imputation is not an implementation detail but a constitutive part of interpretability. Constant-fill or 0-imputation can create spurious discontinuities that let the predictor “peek” at the fill value and let the selector encode labels rather than true feature relevance. Multiple imputation schemes that approximate the true conditional distribution produce smoother masks, lower false-discovery rates, and more stable feature-importance heat-maps. Across the reported tasks, LEX with multiple imputation reduced FDR by 10–30% while preserving accuracy approximately equal to the full model (Senetaire et al., 2022).

A separate information-theoretic account measures the contribution of features to the functional entropy of the decision function. For a class-specific decision function $Y$7 under a Gaussian measure $Y$8, functional entropy is bounded by covariance-weighted functional Fisher information through a log–Sobolev inequality: $Y$9 This yields per-feature and per-subset contribution scores that explicitly account for feature correlations through $\to$0 (Gat et al., 2022).

The practical consequence is a model-centered attribution method that requires only covariance estimation, Monte Carlo sampling, backpropagation, and averaging. On Google Speech Commands, the reported post-hoc perturbation metric improves accuracy-AUC from approximately $\to$1 to approximately $\to$2 and consistency-AUC from approximately $\to$3 to approximately $\to$4; on CIFAR-10 and IMDB, the covariance-aware method yields sharper or more focused explanations than SmoothGrad-style baselines (Gat et al., 2022). These formulations attempt to replace heuristic attribution with inference-theoretic and functional-analytic objects.

5. Concept-structured representations and intervention

Recent work increasingly embeds interpretability into the architecture or training objective. In a Concept Bottleneck framework for time-series Transformers, a single Autoformer layer is modified so that a selected attention head or feed-forward slice aligns with predefined concepts. The concepts used are an autoregressive surrogate model and time features such as hour-of-day, and the training loss combines forecasting MSE with a Centered Kernel Alignment term: $\to$5 The reported result is that forecasting performance remains mostly unaffected while interpretability is much improved, the concepts become local, and a single-component intervention under a time shift restores accuracy close to the unshifted baseline without retraining (Sprang et al., 2024).

A related design, Concept-Centric Transformers, uses concept slots in a shared global workspace. Object-centric concept learning and cross-attention yield concept relevance scores $\to$6, and by design the final explanation is linear in these relevance scores. The paper states that this guarantees faithful explanations in the sense that removing a concept correspondingly removes its contribution. The model also reports better classification accuracy than baseline concept-based methods on CIFAR-100, CUB-200-2011, and ImageNet, while generating more consistent concept-based explanations (Hong et al., 2023).

Concept Distillation moves concept analysis from post-hoc diagnosis to ante-hoc training. Concept Activation Vectors are generalized to intermediate layers using class prototypes, and a Concept Loss

$\to$7

is added to the training objective to sensitize or desensitize the model toward a user-supplied concept. A teacher model can be used to distill richer concept representations into a student. Reported applications include debiasing ColorMNIST from $\to$8 to $\to$9 on the reversed-color test set, with a further increase to $\to$0 when a local saliency loss is added, and improving DecoyMNIST from $\to$1 to $\to$2 (Gupta et al., 2023).

Intervention has also become an explicit evaluation target. An abstract encoder–decoder framework unifies sparse autoencoders, logit lens, tuned lens, and probing by representing an interpretability method as an encoder $\to$3 and decoder $\to$4, allowing a feature vector $\to$5 to be edited and mapped back into latent space for control (Bhalla et al., 2024). Two metrics are introduced: intervention success rate and the coherence–intervention tradeoff. The reported findings are that lens-based methods outperform SAEs and probes for simple, concrete interventions, current mechanistic interventions often compromise model coherence, and a simple prompting baseline often outperforms mechanistic methods on the coherence–intervention tradeoff (Bhalla et al., 2024). This suggests that model-centered interpretability is increasingly evaluated not only by whether it describes model behavior, but by whether it supports controlled and coherent interventions.

6. Evaluation, limits, and contested assumptions

A persistent theme is that interpretability claims require evaluation against explicit tasks. A human-subject study defines simulatability as the user’s ability to compute $\to$6 from a human-readable model representation and “what-if” local explainability as the ability to predict $\to$7 under a local input change given $\to$8. On this basis, decision trees and logistic regression were locally interpretable, while a neural network was not: for 930 confident respondents, the what-if task yielded $\to$9 accuracy for decision trees, $\to$0 for logistic regression, and $\to$1 for the neural network, with the neural network failing the what-if test (Slack et al., 2019). The same study uses runtime operation count as a proxy for cognitive effort and finds evidence that as the number of operations increases, participant accuracy decreases.

A complexity-theoretic approach reaches a structurally similar conclusion. Interpretability of a model class is defined by the worst-case complexity of answering local post-hoc queries such as MinimumChangeRequired, MinimumSufficientReason, and CountCompletions. Under this criterion, both linear and tree-based models are strictly more $\to$2-interpretable than multilayer neural networks for key queries, but there is no single clear-cut ordering between linear and tree-based models across all queries (Barceló et al., 2020). This result formalizes a common folk belief while also showing that the notion depends on the explanation problem being asked.

Stakeholder studies complicate the picture further. A trend analysis over NLP interpretability papers reports that explanations of internal model components are rarely used outside the NLP field. Non-developer stakeholders rarely use mechanistic methods and instead favor local feature attributions, surrogates such as LIME and SHAP, and clustering-based analyses (Calderon et al., 2024). In medicine and psychology, local explanations dominate to support individual decisions, whereas neuroscience and social science more often use global methods such as probing and clustering (Calderon et al., 2024). A plausible implication is that model-centered methods may have strong diagnostic value for developers without being the explanation format most useful to non-developer users.

Finally, the “model” in model-centered interpretability is not always a single predictor. Composite decision systems may combine multiple ML models with an explicit rule layer, and established model-agnostic methods can produce poor explanations in this setting. SMACE addresses this by combining geometric analysis of axis-aligned rule surfaces with local feature-importance vectors from component models to generate a final feature ranking over the original inputs (Lopardo et al., 2021). This is consistent with the broader caution that interpretability is application-specific, semantically mediated, and rarely captured by a single metric or technique (Silva et al., 2019, Lipton, 2016).