Variable Importance Measures (VIMs)

• Variable Importance Measures are quantitative metrics that determine each predictor's contribution to model fit and predictive accuracy.
• They encompass model-specific scores, permutation-based methods, decomposition approaches, and causal estimators tailored to different analytical objectives.
• VIMs provide actionable insights for machine learning interpretability, regression analysis, and treatment-effect evaluation.

Variable importance measures (VIMs) are statistical or algorithmic quantities that assign each predictor a contribution to model fit, predictive performance, or a closely related target, and they occupy a central place in interpretable machine learning, statistical modeling, and causal analysis. Across the literature, the term covers model-specific scores such as impurity-based random-forest importance, model-agnostic perturbation scores such as permutation importance, decomposition-based quantities such as Shapley-style allocations, and causal summaries defined on treatment-effect heterogeneity. What unifies these objects is the attempt to formalize “which variables matter,” but the answer depends on the estimand, the perturbation or decomposition used, the dependence structure among predictors, and the model class or causal target under study (Liang et al., 2024, Coleman, 2022, Reyero-Lobo et al., 23 Jul 2025).

1. Foundational definitions and estimands

A common predictive formulation starts from observed data $O=(Y,X)$, a fitted prediction rule $f$, and a loss function $L$. In that setup, a perturbation-based VIM is a loss contrast of the form

$$\text{Importance} = L(f,\text{perturbed features}) - L(f,\text{original features}).$$

For marginal permutation importance, if $\ddot X^j$ denotes the feature matrix with variable $X_j$ shuffled across observations, then

$VI_j^{P}=L(f,\ddot X^j)-L(f,X).$

For Conditional Predictive Impact (CPI), if $\tilde X^j$ replaces $X_j$ by a knockoff variable, then

$VI_j^{CPI}=L(f,\tilde X^j)-L(f,X).$

A positive value means that destroying or replacing the information in $f$0 worsens prediction, so the variable is important in the sense induced by the perturbation (Liang et al., 2024).

A second tradition defines importance as a decomposition of model fit. In linear models, using $f$1 as the fit criterion leads to the Shapley value (LMG) and proportionate value (PMVD) as variable importance measures. In that formulation, VIMs are “a measure of each regressor’s contribution to model fit,” and the central question is how the explained variance should be allocated when regressors are correlated (Coleman, 2022).

Recent work makes the target explicit through a minimal axiom: $f$2 This axiom formalizes the principle that features providing no additional information should not be assigned importance. It also connects VIMs to variable selection, because the null set of the importance functional coincides with the conditionally irrelevant variables (Reyero-Lobo et al., 23 Jul 2025).

2. Main methodological families

The modern literature contains several major VIM families that differ in their estimands rather than only in implementation.

Family	Representative quantity	Emphasis
Marginal permutation	$f$3	Marginal predictive degradation
Conditional perturbation	$f$4	Dependence-aware predictive degradation
Impurity-based RF importance	MDI / local MDI	Split-based global or instance-wise relevance
PDP-based importance	$f$5 as spread of $f$6	Flatness or variability of partial dependence
Model-class importance	$f$7	Range over well-performing models
Treatment-effect importance	$f$8	Heterogeneity in causal effects

Permutation-style methods perturb a feature and measure degradation in predictive performance. CPI modifies this idea by replacing the feature with a knockoff or conditionally valid surrogate rather than an unconditional shuffle. By contrast, impurity-based random-forest importance accumulates decreases in node impurity. Under totally randomized asymptotic entropy-based trees, global Mean Decrease of Impurity (MDI) is exactly a Shapley value for a game whose characteristic function is mutual information, and local MDI is a Shapley value for a local game based on conditional entropy reduction (Sutera et al., 2021).

A distinct model-based construction uses partial dependence. For a fitted model $f$9, the partial dependence function for predictor $L$0 is estimated by averaging predictions after fixing $L$1 across the observed values of the other variables. The proposed VIM is then the spread of that partial dependence function: for continuous predictors, the sample standard deviation of the partial dependence values; for categorical predictors, the range divided by 4. This yields a standardized, model-based importance notion across supervised learning algorithms (Greenwell et al., 2018).

Model uncertainty motivates a further extension. Model Reliance (MR) measures how much a fixed model’s performance worsens when a variable is switched, while Model Class Reliance (MCR) is the range of MR values over all models in a prespecified class whose expected loss is within $L$2 of a reference model. MCR therefore treats variable importance as a property of a Rashomon set rather than of a single fitted predictor (Fisher et al., 2018).

3. Correlation, redundancy, and support violations

A central result across the literature is that between-feature correlation is a major confounder for variable importance estimation. In standard permutation importance, shuffling one predictor breaks the joint distribution and creates implausible covariate combinations. This can induce extrapolation bias and distort rankings: correlated variables may appear spuriously more important, less important, or have their importance redistributed in ways that do not match the scientific interpretation one wants (Liang et al., 2024).

Conditional perturbation was introduced to reduce that problem, but it is not a free lunch. In the bivariate Gaussian knockoff construction used for CPI, the correlation between a variable and its own knockoff obeys

$L$3

Thus knockoffs are uncorrelated with the original variable only up to a predictor-correlation threshold of $L$4; beyond that threshold, the knockoff must itself become increasingly correlated with the original variable in order for the joint covariance matrix to remain positive semidefinite. The empirical consequence is a turning point: CPI can avoid inflation at low to moderate correlations, but under strong dependence it tends to attenuate or deflate importance (Liang et al., 2024).

A related population analysis appears in the marginal and conditional importance metrics MVIM and CVIM. MVIM replaces the feature of interest by an independent draw from its marginal distribution and measures the increase in expected squared loss under the true conditional expectation $L$5. The paper’s bias–variance decomposition shows that the estimator can be seriously biased when predictors are highly correlated because the switched covariate vectors fall in low-probability regions where black-box models extrapolate poorly. CVIM instead replaces the feature by a draw from its conditional distribution given the remaining covariates, reducing this extrapolation effect (Khan et al., 28 Jan 2025).

The same concern underlies the recent axiomatic critique of Shapley-style global importance. If the objective is conditional relevance, then a method should assign zero importance exactly when a variable provides no additional information beyond the others. This is why the minimal axiom above is proposed as a criterion for comparing VIMs, and why several Shapley-style global scores are criticized for producing false positives under spurious correlation (Reyero-Lobo et al., 23 Jul 2025).

4. Global, local, and task-specific importance

Global VIMs summarize relevance across an entire dataset, but many applications require observation-specific or task-specific importance. In tree ensembles, local MDI is defined by summing impurity changes along the path traversed by a given observation. Unlike global MDI, local MDI may be negative because the impurity of the child node can exceed that of the parent for that observation. A key identity is that global MDI is the average of local MDI over instances, so local importance is a decomposition of the global score across observations (Sutera et al., 2021).

Model-agnostic local importance can be defined by localized loss perturbation. CLIQUE computes, for observation $L$6 and feature $L$7,

$L$8

where the altered values $L$9 come from a quantile grid and the predictions are computed by cross-validation models that excluded observation $$\text{Importance} = L(f,\text{perturbed features}) - L(f,\text{original features}).$$0. The intended behavior is that $$\text{Importance} = L(f,\text{perturbed features}) - L(f,\text{original features}).$$1 is near zero in regions where a variable has no local effect and positive where changing that variable materially worsens predictive accuracy for that observation. The method is also presented as directly applicable to multi-category classification problems (Bladen et al., 2024).

Multi-class outcomes motivate still another specialization. Conventional random-forest VIMs such as permutation and Gini importance reward variables that improve overall discrimination or node purity, but they do not distinguish class-associated covariates from covariates that only separate broad groups of classes. Multi forests (MuFs) address this by combining class-assigned multi-way splits with ordinary binary splits and defining two separate VIMs: a multi-class VIM, intended to rank class-associated covariates highly, and a discriminatory VIM, intended to measure general class discrimination irrespective of class-associatedness (Hornung et al., 2024).

These developments indicate that “local,” “global,” and “class-specific” importance are not interchangeable. A plausible implication is that the choice among them should be driven by the scientific question: instance-wise explanation, class-associated discovery, or overall predictive contribution.

5. Causal and treatment-effect importance

VIMs need not be restricted to outcome prediction. In heterogeneous treatment-effect analysis, treatment effect variable importance measures (TE-VIMs) are defined on the conditional average treatment effect

$$\text{Importance} = L(f,\text{perturbed features}) - L(f,\text{original features}).$$2

For a subset $$\text{Importance} = L(f,\text{perturbed features}) - L(f,\text{original features}).$$3, the reduced treatment-effect predictor is

$$\text{Importance} = L(f,\text{perturbed features}) - L(f,\text{original features}).$$4

and the proposed importance measure is

$$\text{Importance} = L(f,\text{perturbed features}) - L(f,\text{original features}).$$5

This quantity is the additional treatment-effect heterogeneity explained by $$\text{Importance} = L(f,\text{perturbed features}) - L(f,\text{original features}).$$6 beyond $$\text{Importance} = L(f,\text{perturbed features}) - L(f,\text{original features}).$$7. A scaled version,

$$\text{Importance} = L(f,\text{perturbed features}) - L(f,\text{original features}).$$8

plays the role of a proportion of explained heterogeneity (Hines et al., 2022).

A related line defines switching-based importance for black-box models through the true conditional expectation $$\text{Importance} = L(f,\text{perturbed features}) - L(f,\text{original features}).$$9. The Generalized Variable Importance Metric (GVIM) measures the increase in expected squared prediction loss when a predictor is replaced by an independent replicate from its marginal distribution, and the paper shows that GVIM can be represented as a function of Conditional Average Treatment Effects for binary, multinomial, and continuous predictors. Its role is predictive first, but under positivity, conditional ignorability, and consistency it admits a causal interpretation through squared treatment contrasts (Khan et al., 2022).

The newer MVIM and CVIM sharpen this connection. MVIM is the increase in expected squared loss when $\ddot X^j$0 is replaced by an independent draw from its marginal distribution; CVIM replaces $\ddot X^j$1 by a draw from its conditional distribution given $\ddot X^j$2. Both are shown to have quadratic relationships with CATE. For binary treatment, CVIM takes the form

$\ddot X^j$3

so it quantifies conditional treatment-effect variation weighted by residual treatment variation within covariate strata (Khan et al., 28 Jan 2025).

These causal VIMs differ from prediction-oriented VIMs in a substantive way. They ask not which variables improve prediction of $\ddot X^j$4, but which variables explain variation in $\ddot X^j$5. The distinction is central in domains where treatment recommendation is the inferential target rather than outcome prediction alone.

6. Estimation, inference, and interpretation

Once the estimand is fixed, estimation and inference become separate problems. A recent organizing proposal is a three-step pipeline: define the theoretical index, estimate it, and add statistical guarantees for selected variables. This is intended to clarify that perturbation, marginalization, and refitting are estimation strategies, not by themselves definitions of importance (Reyero-Lobo et al., 23 Jul 2025).

Inference is especially delicate because VIMs depend on estimated nuisance functions and often on complex perturbations. For conditional permutation importance, targeted learning has been proposed as an alternative to one-step efficient-influence-function corrections. The targeted estimator iteratively fluctuates the estimated distribution in the efficient influence direction and, under pathwise differentiability and remainder control assumptions, is asymptotically linear with the efficient influence function and asymptotically normal, while the numerical experiments indicate improved finite-sample robustness relative to one-step correction (Wang et al., 2024).

For random-forest permutation importance, nonparametric permutation tests provide distribution-free inference but can be computationally prohibitive. Sequential permutation testing addresses this by stopping early when the final decision is already sufficiently clear. In the reported simulation studies, the sequential tests control type-I error at a nominal level and maintain high power with considerably fewer permutations than conventional permutation testing (Hapfelmeier et al., 2022).

In large neural networks, retraining-based VIMs are often the bottleneck. LazyVI approximates the reduced model by linearizing the network at the full-model parameters and solving a ridge-penalized convex problem in the induced neural tangent kernel feature space. When the ridge penalty is sufficiently large, the paper proves $\ddot X^j$6 error for the VIM estimator and asymptotic normality, enabling confidence bounds while avoiding full retraining for each deleted variable (Gao et al., 2022).

Model uncertainty introduces a further layer of ambiguity. Model Class Reliance replaces a single model-specific importance by the interval of importances over all models in a prespecified class whose expected loss is within $\ddot X^j$7 of a reference model. With omitted or unobserved variables, recent work extends this interval view by using Rashomon sets together with explicit sensitivity parameters for unobserved predictive heterogeneity and VI drift, yielding bounds on the importance of a variable for the true conditional mean function rather than a single observed-feature model (Fisher et al., 2018, Donnelly et al., 14 Oct 2025).

Taken together, these results support a broad conclusion already explicit in several of the papers: there is no universally correct VIM. Some VIMs decompose model fit, some quantify predictive degradation under perturbation, some target class-associated discrimination, and some summarize heterogeneity in causal effects. Under strong feature correlation, ranking individual variables is intrinsically fragile; under model multiplicity, a single fitted model can be misleading; and under causal questions, predictive importance and treatment-effect importance answer different questions. The principled use of VIMs therefore depends less on choosing a universally best method than on aligning the estimand, perturbation, and inferential machinery with the scientific target.