Permutation Feature Importance

• PFI is a perturbation-based, model-agnostic method that quantifies a feature’s importance by comparing model performance before and after random shuffling.
• It leverages metrics such as accuracy and RMSE to evaluate the increase in prediction error, extending naturally to feature sets and conditional contexts.
• Recent advancements include adaptations for streaming data, fairness analysis, and various data types, enhancing its utility across diverse applications.

Permutation Feature Importance (PFI) is a perturbation-based, model-agnostic feature importance method that measures how much a trained predictor relies on a feature by breaking that feature’s relationship to the rest of the data and then reevaluating predictive performance. In its standard form, a feature is permuted across samples, which preserves the feature’s marginal distribution but breaks its association with the response and with other predictors; the importance score is the resulting increase in loss or prediction error, or equivalently the decrease in performance. PFI is commonly used as a global explanation method because it is efficient, intuitively interpretable, and applicable to fixed trained models across model classes, but its interpretation depends strongly on the perturbation regime and on the dependence structure among predictors (König et al., 2020, Molnar et al., 2021, Foote et al., 9 Jul 2025).

1. Definition and basic computation

A standard formulation compares the original model risk with the risk after perturbing a feature $X_j$. With loss $L$, target $Y$, trained model $f$, and remaining features $X_R$, the risk on original data is

$R = \mathbb{E}\big[L(Y, f(X_j, X_R))\big],$

and feature importance is written as

$$FI_j = \tilde{R} - R \qquad \text{or} \qquad FI_j = \frac{\tilde{R}}{R},$$

where $\tilde{R}$ is the risk after replacing $X_j$ by a perturbed version (König et al., 2020). In the usual empirical workflow, one trains a model once, evaluates baseline performance on an evaluation set $E$, randomly shuffles one feature’s values across samples, reevaluates the model, and uses the performance drop as the importance score. This procedure is summarized as: compute baseline error or performance on $L$0, shuffle a feature’s values in $L$1, compute performance again, and take the difference (Espinosa et al., 24 Jan 2025).

The metric used in the score depends on task and implementation. The subset-based high-dimensional feature-selection formulation uses accuracy for classification and RMSE for regression (Espinosa et al., 24 Jan 2025). A neural-network baseline implementation averages the drop in accuracy over repeated permutations,

$L$2

with $L$3 in the reported experiments (K, 2020). In related XAI comparisons, PFI is described as calculating the increase or decrease of the model’s prediction error after permuting, i.e. removing, a feature, with larger increases indicating greater importance (Ghosh et al., 12 May 2026).

PFI is also naturally defined for feature sets rather than only for single variables. For a feature set $L$4, one formulation writes

$L$5

where $L$6 is a permuted or resampled version of the feature set and $L$7 is the complement of $L$8 (Molnar et al., 2021). This makes explicit that PFI is a property of a trained model under a specified perturbation mechanism rather than an intrinsic property of the raw data alone.

2. Statistical estimands, theory, and inference

A central theoretical development is the treatment of PFI as an estimator of a quantity defined at the level of the data generating process (DGP). In that framework, empirical PFI is not merely an ad hoc diagnostic: it targets a DGP-level estimand obtained by replacing the learned predictor with the true regression function $L$9. The same work distinguishes three sources of discrepancy between empirical PFI and the DGP target: model bias, model variance, and Monte Carlo approximation error (Molnar et al., 2021).

This perspective yields a distinction between model-PFI and learner-PFI. Model-PFI is the usual PFI computed for one fixed fitted model. Learner-PFI averages PFI over multiple refitted models and is intended to answer a different question: not how important a feature is in one particular model fit, but how important it is on average for the learning procedure. The variance of model-PFI reflects variability across evaluation cases, whereas learner-PFI requires corrected variance estimation because refitted models are correlated when they reuse overlapping data; the correction

$Y$0

is adopted from Nadeau and Bengio for real-world resampling settings (Molnar et al., 2021).

For random forest regression, stronger asymptotic results are available under explicit assumptions. When covariates are pairwise independent, the regression function is bounded, and the infinite random forest is $Y$1-consistent, the out-of-bag random-forest permutation importance is exactly unbiased for non-informative variables and asymptotically unbiased for informative ones (Ramosaj et al., 2019). The corresponding theoretical population quantity is

$Y$2

where $Y$3 replaces the $Y$4-th coordinate with an independent copy (Ramosaj et al., 2019).

Inference for permutation importance has also been developed for incomplete data. In random forests with missing values, the delete-$Y$5 jackknife variance estimator used by Ishwaran et al. is reported to work reasonably well on complete data, but single imputation can produce severe undercoverage. A proposed remedy applies Rubin’s rule to combine permutation-importance estimates and their variances over several imputed datasets, with pooled variance

$Y$6

where $Y$7 is the within-imputation variance and $Y$8 is the between-imputation variance (Föge et al., 18 Jul 2025). This suggests that statistical interpretation of PFI is inseparable from the uncertainty induced by resampling, model instability, and data preprocessing.

3. Dependence, redundancy, and conditional interpretations

The principal limitation of classical PFI is its treatment of the feature of interest in isolation. Marginal permutation preserves the marginal distribution of $Y$9 but breaks dependence with both the target and the remaining features, and this can create unrealistic or out-of-distribution inputs when predictors are correlated (König et al., 2020, Foote et al., 9 Jul 2025). In tree-based models, such unrealistic samples can move observations into leaf communities far from the training data; TRIP formalizes this extrapolation problem through the average leaf community distance (ALCD) and uses a paired permutation test against an independent baseline feature to detect when vanilla PFI is likely unreliable (Foote et al., 9 Jul 2025).

The effect of correlation on importance values can be derived explicitly in additive and Gaussian settings. For the additive regression model

$f$0

one result is

$f$1

In Gaussian settings with $f$2 equally correlated variables and common correlation $f$3, the paper gives

$f$4

These formulas show that as positive correlation $f$5 increases, or as the size of a correlated group grows, each individual variable’s permutation importance decreases; predictive information is diluted across correlated predictors (Gregorutti et al., 2013).

This dilution motivates conditional and relative formulations. Relative Feature Importance (RFI) generalizes both marginal PFI and Conditional Feature Importance (CFI) by conditioning on an arbitrary set $f$6: $f$7 with $f$8 and $f$9. PFI is recovered when $X_R$0, and CFI is recovered when $X_R$1 (König et al., 2020). The associated interpretive shift is precise: marginal PFI asks how important $X_R$2 is on its own, whereas conditional importance asks how much additional information $X_R$3 provides given the remaining features.

A model-agnostic conditional alternative uses conditional subgroups. Decision trees or transformation trees partition the data based on $X_R$4 into groups in which $X_R$5 is approximately independent of $X_R$6. Permuting within subgroup rather than globally yields a conditional subgroup PFI estimator,

$X_R$7

which produces both a global conditional importance and local subgroup importances (Molnar et al., 2020). Related correction strategies include pairwise permutation with correlation-weighted aggregation, introduced to mitigate correlation bias in high-dimensional biological data (Maaslandand et al., 2021), and Covered Information Disentanglement (CID), which models the part of a feature’s information that is “covered” by covariates and uses multivariate entropy together with a Markov random field approximation to adjust standard PFI (Pereira et al., 2021). A recurrent caution across these approaches is that low conditional importance does not mean irrelevance; it can indicate that the feature’s information is already available through dependent variables (Molnar et al., 2020).

4. Computational regimes and algorithmic variants

PFI has been extended well beyond the one-feature-at-a-time, repeated-random-permutation setting. These variants preserve the core perturbation logic but change the unit of perturbation, the optimization objective, or the computational regime.

Variant	Defining change	arXiv id
Subset-based PFI	Permutes selected feature subsets jointly and optimizes performance degradation versus subset cardinality with NSGA-II	(Espinosa et al., 24 Jan 2025)
Incremental PFI (iPFI)	Replaces batch permutations by online sampled replacements and exponential smoothing on data streams	(Fumagalli et al., 2022)
Deterministic one-permutation importance	Replaces repeated random permutations with one deterministic cyclic rank shift	(Dorador, 15 Dec 2025)

In high-dimensional feature selection, subset-based PFI evaluates a binary mask $X_R$8 by jointly shuffling all selected features and measuring

$X_R$9

where $R = \mathbb{E}\big[L(Y, f(X_j, X_R))\big],$0 is baseline performance and $R = \mathbb{E}\big[L(Y, f(X_j, X_R))\big],$1 is performance after permutation. This turns PFI into a two-objective combinatorial optimization problem: maximize the degradation in model performance when the selected features are shuffled, and minimize subset cardinality. The search is carried out with NSGA-II using a binary chromosome, half uniform crossover, and bit-flip mutation (Espinosa et al., 24 Jan 2025).

For streaming and concept-drift settings, iPFI replaces the batch permutation mechanism by an online sampled replacement and updates importance via exponential smoothing,

$R = \mathbb{E}\big[L(Y, f(X_j, X_R))\big],$2

Uniform and geometric sampling schemes are proposed, and the method is designed to provide anytime explanations with constant-time model evaluations and low memory requirements (Fumagalli et al., 2022).

A different line of work argues that classical repeated random permutations introduce unnecessary stochastic instability. A deterministic alternative replaces multiple random permutations with a single cyclic rank shift

$R = \mathbb{E}\big[L(Y, f(X_j, X_R))\big],$3

chosen because it is max-min optimal under a circular displacement criterion. In that formulation, Direct Variable Importance (DVI) measures the prediction disruption induced by isolated perturbation, while Systemic Variable Importance (SVI) propagates perturbations through empirical correlations for model stress-testing (Dorador, 15 Dec 2025). This suggests that the computational identity of PFI is not fixed: it can function as a Monte Carlo estimator, a streaming statistic, or a deterministic stress-test primitive, depending on the perturbation design.

5. Adaptations beyond ordinary tabular predictors

A substantial body of work adapts PFI to data types for which naive column-wise permutation is not directly meaningful. In functional data, raw inputs are correlated over time, so direct permutation of time points can create unrealistic samples. One solution applies functional principal component analysis (fPCA) first, trains the predictive model on uncorrelated functional principal component (fPC) scores, and then computes PFI on those scores. In an explosion-signature case study, the first fPC explains 94% of the variability and the first three fPCs together explain 99%; PFI then identifies which modes of functional variation matter for prediction and visualizations map those components back to interpretable signal characteristics such as early intensity, timing of peaks, number of peaks, and overall intensity (Goode et al., 2020).

A related functional-data refinement targets augmented functional random forests built on scores from original curves and successive derivatives. Because those scores are strongly correlated across derivative orders, a grouped conditional permutation scheme is introduced: $R = \mathbb{E}\big[L(Y, f(X_j, X_R))\big],$4 Permutation is performed conditionally within each component group, and importance is defined as the average increase in loss after conditional permutation (MAturo et al., 2024).

For image classification, direct pixel permutation is structurally destructive and difficult to interpret. DEPICT moves the permutation operation to a semantic concept space $R = \mathbb{E}\big[L(Y, f(X_j, X_R))\big],$5, such as captions or concept annotations, and uses a text-conditioned diffusion model $R = \mathbb{E}\big[L(Y, f(X_j, X_R))\big],$6 to generate images from permuted concepts. If $R = \mathbb{E}\big[L(Y, f(X_j, X_R))\big],$7 denotes performance on generated images from unpermuted concepts and $R = \mathbb{E}\big[L(Y, f(X_j, X_R))\big],$8 performance after permuting concept $R = \mathbb{E}\big[L(Y, f(X_j, X_R))\big],$9, importance is the performance drop

$$FI_j = \tilde{R} - R \qquad \text{or} \qquad FI_j = \frac{\tilde{R}}{R},$$0

This produces dataset-level rankings of concept importance rather than local pixel-space attributions (Jabbour et al., 2024).

Context-aware adaptations make a similar move for structured scenario data. In pedestrian intention prediction, Context-aware Permutation Feature Importance (CAPFI) restricts shuffling to context-defined subsets of the PIE dataset rather than permuting across all scenes. The reported global ranking over the full validation and test set places pedestrian bounding box first, ego-vehicle speed second, local context next, and pose last, while context-specific analysis shows that the relative importance of these features changes across roadway type, traffic-light state, crosswalk state, proximity level, and ego-vehicle speed (Azarmi et al., 2024). A plausible implication is that PFI remains most interpretable when the permutation mechanism respects the semantic structure of the domain.

6. Fairness-oriented formulations, applications, and common misconceptions

PFI has also been generalized from predictive performance to fairness analysis. A fairness-oriented permutation score compares a bias or fairness metric before and after feature permutation: $$FI_j = \tilde{R} - R \qquad \text{or} \qquad FI_j = \frac{\tilde{R}}{R},$$1 where $$FI_j = \tilde{R} - R \qquad \text{or} \qquad FI_j = \frac{\tilde{R}}{R},$$2 is a sensitive attribute and $$FI_j = \tilde{R} - R \qquad \text{or} \qquad FI_j = \frac{\tilde{R}}{R},$$3 is a chosen fairness metric. Here the question is no longer how much a feature contributes to prediction error, but how much it contributes to model bias or fairness (Little et al., 9 Feb 2026). The same work contrasts this with an occlusion-based score,

$$FI_j = \tilde{R} - R \qquad \text{or} \qquad FI_j = \frac{\tilde{R}}{R},$$4

and notes that the occlusion score can be computed efficiently through minipatch learning, whereas the permutation score is conceptually simpler but may require training $$FI_j = \tilde{R} - R \qquad \text{or} \qquad FI_j = \frac{\tilde{R}}{R},$$5 models (Little et al., 9 Feb 2026).

PFI is also regularly used as a baseline in comparative XAI and feature-selection studies. In one explainability comparison, PFI and SHAP are baselines for a graph-theoretic feature-ranking method, and the evaluation protocol judges ranking quality by training classifiers on the top 30% and bottom 30% of features selected by each method (Ghosh et al., 12 May 2026). In neural-network feature selection, PFI is used as a post hoc, model-agnostic ranking baseline, but the reported top-versus-bottom 10% accuracy gaps are small on correlated, high-dimensional datasets such as MNIST, ISOLET, and HAR, which the paper attributes to PFI’s tendency to share importance across correlated features (K, 2020).

Several misconceptions recur in the literature. First, PFI is not a measure of intrinsic causal relevance; it measures performance degradation under a specified perturbation regime for a trained predictor (Molnar et al., 2021). Second, low or near-zero importance under a conditional method does not imply that a feature is irrelevant; it can mean that its information is redundant with other variables (Molnar et al., 2020). Third, negative PFI values can occur: in the functional-data formulation, negative PFI means that the permuted feature oddly leads to better predictions, so the original feature may not be useful (Goode et al., 2020). Fourth, the semantics of the score depend on what is preserved during perturbation. Marginal PFI breaks all dependencies, CFI preserves dependence on the remaining features, RFI interpolates between those extremes, and fairness-aware PFI replaces prediction loss by a fairness metric (König et al., 2020, Little et al., 9 Feb 2026).

Taken together, these developments position PFI less as a single algorithm than as a family of perturbation-based importance estimators. Its classical form remains the increase in loss or error after random permutation of a feature, but contemporary work treats the choice of perturbation distribution, evaluation metric, refitting regime, dependence correction, and sampling strategy as defining the precise scientific meaning of the resulting importance score.