---
title: Variable Importance Measures (VIMs)
url: https://www.emergentmind.com/topics/variable-importance-measures-vims
type: topic
---

# Variable Importance Measures (VIMs)

Variable importance measures (VIMs) are statistical or algorithmic quantities that assign each predictor a contribution to model fit, predictive performance, or a closely related target, and they occupy a central place in interpretable machine learning, statistical modeling, and causal analysis. Across the literature, the term covers model-specific scores such as impurity-based random-forest importance, model-agnostic perturbation scores such as permutation importance, decomposition-based quantities such as Shapley-style allocations, and causal summaries defined on treatment-effect heterogeneity. What unifies these objects is the attempt to formalize “which variables matter,” but the answer depends on the estimand, the perturbation or decomposition used, the dependence structure among predictors, and the model class or causal target under study [2402.03447] [2212.03289] [2507.17306].

## 1. Foundational definitions and estimands

A common predictive formulation starts from observed data \(O=(Y,X)\), a fitted prediction rule \(f\), and a loss function \(L\). In that setup, a perturbation-based VIM is a loss contrast of the form
\[
\text{Importance} = L(f,\text{perturbed features}) - L(f,\text{original features}).
\]
For marginal permutation importance, if \(\ddot X^j\) denotes the feature matrix with variable \(X_j\) shuffled across observations, then
\[
VI_j^{P}=L(f,\ddot X^j)-L(f,X).
\]
For Conditional Predictive Impact (CPI), if \(\tilde X^j\) replaces \(X_j\) by a knockoff variable, then
\[
VI_j^{CPI}=L(f,\tilde X^j)-L(f,X).
\]
A positive value means that destroying or replacing the information in \(X_j\) worsens prediction, so the variable is important in the sense induced by the perturbation [2402.03447].

A second tradition defines importance as a decomposition of model fit. In linear models, using \(R^2\) as the fit criterion leads to the Shapley value (LMG) and proportionate value (PMVD) as variable importance measures. In that formulation, VIMs are “a measure of each regressor’s contribution to model fit,” and the central question is how the explained variance should be allocated when regressors are correlated [2212.03289].

Recent work makes the target explicit through a minimal axiom:
\[
\psi(j,P)=0 \iff X^j \perp\!\!\!\perp y \mid X^{-j}.
\]
This axiom formalizes the principle that features providing no additional information should not be assigned importance. It also connects VIMs to variable selection, because the null set of the importance functional coincides with the conditionally irrelevant variables [2507.17306].

## 2. Main methodological families

The modern literature contains several major VIM families that differ in their estimands rather than only in implementation.

| Family | Representative quantity | Emphasis |
|---|---|---|
| Marginal permutation | \(VI_j^{P}=L(f,\ddot X^j)-L(f,X)\) | Marginal predictive degradation |
| Conditional perturbation | \(VI_j^{CPI}=L(f,\tilde X^j)-L(f,X)\) | Dependence-aware predictive degradation |
| Impurity-based RF importance | MDI / local MDI | Split-based global or instance-wise relevance |
| PDP-based importance | \(i(x_j)\) as spread of \(\bar f_j(x_j)\) | Flatness or variability of partial dependence |
| Model-class importance | \([MCR_-(\epsilon),MCR_+(\epsilon)]\) | Range over well-performing models |
| Treatment-effect importance | \(\Theta_s=\|\tau(X)-\tau_s(X)\|^2\) | Heterogeneity in causal effects |

Permutation-style methods perturb a feature and measure degradation in predictive performance. CPI modifies this idea by replacing the feature with a knockoff or conditionally valid surrogate rather than an unconditional shuffle. By contrast, impurity-based random-forest importance accumulates decreases in node impurity. Under totally randomized asymptotic entropy-based trees, global Mean Decrease of Impurity (MDI) is exactly a Shapley value for a game whose characteristic function is mutual information, and local MDI is a Shapley value for a local game based on conditional entropy reduction [2111.02218].

A distinct model-based construction uses partial dependence. For a fitted model \(\hat f\), the partial dependence function for predictor \(x_j\) is estimated by averaging predictions after fixing \(x_j\) across the observed values of the other variables. The proposed VIM is then the spread of that partial dependence function: for continuous predictors, the sample standard deviation of the partial dependence values; for categorical predictors, the range divided by 4. This yields a standardized, model-based importance notion across supervised learning algorithms [1805.04755].

Model uncertainty motivates a further extension. Model Reliance (MR) measures how much a fixed model’s performance worsens when a variable is switched, while Model Class Reliance (MCR) is the range of MR values over all models in a prespecified class whose expected loss is within \(\epsilon\) of a reference model. MCR therefore treats variable importance as a property of a Rashomon set rather than of a single fitted predictor [1801.01489].

## 3. Correlation, redundancy, and support violations

A central result across the literature is that between-feature correlation is a major confounder for variable importance estimation. In standard permutation importance, shuffling one predictor breaks the joint distribution and creates implausible covariate combinations. This can induce extrapolation bias and distort rankings: correlated variables may appear spuriously more important, less important, or have their importance redistributed in ways that do not match the scientific interpretation one wants [2402.03447].

Conditional perturbation was introduced to reduce that problem, but it is not a free lunch. In the bivariate Gaussian knockoff construction used for CPI, the correlation between a variable and its own knockoff obeys
\[
\mathrm{corr}(X_1,\tilde X_1)=
\begin{cases}
0, & \text{if } \mathrm{corr}(X_1,X_2)\le 0.5,\\
2\,\mathrm{corr}(X_1,X_2)-1, & \text{otherwise}.
\end{cases}
\]
Thus knockoffs are uncorrelated with the original variable only up to a predictor-correlation threshold of \(0.5\); beyond that threshold, the knockoff must itself become increasingly correlated with the original variable in order for the joint covariance matrix to remain positive semidefinite. The empirical consequence is a turning point: CPI can avoid inflation at low to moderate correlations, but under strong dependence it tends to attenuate or deflate importance [2402.03447].

A related population analysis appears in the marginal and conditional importance metrics MVIM and CVIM. MVIM replaces the feature of interest by an independent draw from its marginal distribution and measures the increase in expected squared loss under the true conditional expectation \(f_0\). The paper’s bias–variance decomposition shows that the estimator can be seriously biased when predictors are highly correlated because the switched covariate vectors fall in low-probability regions where black-box models extrapolate poorly. CVIM instead replaces the feature by a draw from its conditional distribution given the remaining covariates, reducing this extrapolation effect [2501.16988].

The same concern underlies the recent axiomatic critique of Shapley-style global importance. If the objective is conditional relevance, then a method should assign zero importance exactly when a variable provides no additional information beyond the others. This is why the minimal axiom above is proposed as a criterion for comparing VIMs, and why several Shapley-style global scores are criticized for producing false positives under spurious correlation [2507.17306].

## 4. Global, local, and task-specific importance

Global VIMs summarize relevance across an entire dataset, but many applications require observation-specific or task-specific importance. In tree ensembles, local MDI is defined by summing impurity changes along the path traversed by a given observation. Unlike global MDI, local MDI may be negative because the impurity of the child node can exceed that of the parent for that observation. A key identity is that global MDI is the average of local MDI over instances, so local importance is a decomposition of the global score across observations [2111.02218].

Model-agnostic local importance can be defined by localized loss perturbation. CLIQUE computes, for observation \(i\) and feature \(j\),
\[
V_{ij}=\frac{1}{M}\sum_{m=1}^M \mathcal L\!\left(\hat f(\tilde{\mathbf x}_i(j,m)),y_i\right)-\mathcal L\!\left(\hat f(\mathbf x_i),y_i\right),
\]
where the altered values \(\tilde{\mathbf x}_i(j,m)\) come from a quantile grid and the predictions are computed by cross-validation models that excluded observation \(i\). The intended behavior is that \(V_{ij}\) is near zero in regions where a variable has no local effect and positive where changing that variable materially worsens predictive accuracy for that observation. The method is also presented as directly applicable to multi-category classification problems [2411.08821].

Multi-class outcomes motivate still another specialization. Conventional random-forest VIMs such as permutation and Gini importance reward variables that improve overall discrimination or node purity, but they do not distinguish class-associated covariates from covariates that only separate broad groups of classes. Multi forests (MuFs) address this by combining class-assigned multi-way splits with ordinary binary splits and defining two separate VIMs: a **multi-class VIM**, intended to rank class-associated covariates highly, and a **discriminatory VIM**, intended to measure general class discrimination irrespective of class-associatedness [2409.08925].

These developments indicate that “local,” “global,” and “class-specific” importance are not interchangeable. A plausible implication is that the choice among them should be driven by the scientific question: instance-wise explanation, class-associated discovery, or overall predictive contribution.

## 5. Causal and treatment-effect importance

VIMs need not be restricted to outcome prediction. In heterogeneous treatment-effect analysis, treatment effect variable importance measures (TE-VIMs) are defined on the conditional average treatment effect
\[
\tau(x)=E(Y^1-Y^0\mid X=x).
\]
For a subset \(s\), the reduced treatment-effect predictor is
\[
\tau_s(x)=E(Y^1-Y^0\mid X_{-s}=x_{-s}),
\]
and the proposed importance measure is
\[
\Theta_s=L\{\tau_s\}-L\{\tau\}
      =\|\tau(X)-\tau_s(X)\|^2
      =E\!\left[\operatorname{var}\{\tau(X)\mid X_{-s}\}\right].
\]
This quantity is the additional treatment-effect heterogeneity explained by \(X_s\) beyond \(X_{-s}\). A scaled version,
\[
\Psi_s=\Theta_s/\Theta_p,
\]
plays the role of a proportion of explained heterogeneity [2204.06030].

A related line defines switching-based importance for black-box models through the true conditional expectation \(f_0\). The Generalized Variable Importance Metric (GVIM) measures the increase in expected squared prediction loss when a predictor is replaced by an independent replicate from its marginal distribution, and the paper shows that GVIM can be represented as a function of Conditional Average Treatment Effects for binary, multinomial, and continuous predictors. Its role is predictive first, but under positivity, conditional ignorability, and consistency it admits a causal interpretation through squared treatment contrasts [2212.09931].

The newer MVIM and CVIM sharpen this connection. MVIM is the increase in expected squared loss when \(X\) is replaced by an independent draw from its marginal distribution; CVIM replaces \(X\) by a draw from its conditional distribution given \(\boldsymbol Z\). Both are shown to have quadratic relationships with CATE. For binary treatment, CVIM takes the form
\[
\mathcal{CI}_X
=
2E_{\boldsymbol Z}\!\left[
\operatorname{Var}(X\mid \boldsymbol Z)\,
\big\{E(Y_1\mid \boldsymbol Z)-E(Y_0\mid \boldsymbol Z)\big\}^2
\right],
\]
so it quantifies conditional treatment-effect variation weighted by residual treatment variation within covariate strata [2501.16988].

These causal VIMs differ from prediction-oriented VIMs in a substantive way. They ask not which variables improve prediction of \(Y\), but which variables explain variation in \(\tau(X)\). The distinction is central in domains where treatment recommendation is the inferential target rather than outcome prediction alone.

## 6. Estimation, inference, and interpretation

Once the estimand is fixed, estimation and inference become separate problems. A recent organizing proposal is a three-step pipeline: define the theoretical index, estimate it, and add statistical guarantees for selected variables. This is intended to clarify that perturbation, marginalization, and refitting are estimation strategies, not by themselves definitions of importance [2507.17306].

Inference is especially delicate because VIMs depend on estimated nuisance functions and often on complex perturbations. For conditional permutation importance, targeted learning has been proposed as an alternative to one-step efficient-influence-function corrections. The targeted estimator iteratively fluctuates the estimated distribution in the efficient influence direction and, under pathwise differentiability and remainder control assumptions, is asymptotically linear with the efficient influence function and asymptotically normal, while the numerical experiments indicate improved finite-sample robustness relative to one-step correction [2411.02221].

For random-forest permutation importance, nonparametric permutation tests provide distribution-free inference but can be computationally prohibitive. Sequential permutation testing addresses this by stopping early when the final decision is already sufficiently clear. In the reported simulation studies, the sequential tests control type-I error at a nominal level and maintain high power with considerably fewer permutations than conventional permutation testing [2206.01284].

In large neural networks, retraining-based VIMs are often the bottleneck. LazyVI approximates the reduced model by linearizing the network at the full-model parameters and solving a ridge-penalized convex problem in the induced neural tangent kernel feature space. When the ridge penalty is sufficiently large, the paper proves \(O(1/\sqrt n)\) error for the VIM estimator and asymptotic normality, enabling confidence bounds while avoiding full retraining for each deleted variable [2207.09097].

Model uncertainty introduces a further layer of ambiguity. Model Class Reliance replaces a single model-specific importance by the interval of importances over all models in a prespecified class whose expected loss is within \(\epsilon\) of a reference model. With omitted or unobserved variables, recent work extends this interval view by using Rashomon sets together with explicit sensitivity parameters for unobserved predictive heterogeneity and VI drift, yielding bounds on the importance of a variable for the true conditional mean function rather than a single observed-feature model [1801.01489] [2510.12734].

Taken together, these results support a broad conclusion already explicit in several of the papers: there is no universally correct VIM. Some VIMs decompose model fit, some quantify predictive degradation under perturbation, some target class-associated discrimination, and some summarize heterogeneity in causal effects. Under strong feature correlation, ranking individual variables is intrinsically fragile; under model multiplicity, a single fitted model can be misleading; and under causal questions, predictive importance and treatment-effect importance answer different questions. The principled use of VIMs therefore depends less on choosing a universally best method than on aligning the estimand, perturbation, and inferential machinery with the scientific target.

Source: https://www.emergentmind.com/topics/variable-importance-measures-vims