Feature-Importance Analysis Methods

• Feature-importance analysis is defined as quantifying each input feature's contribution to model predictions using statistical, perturbation, and game-theoretic methods.
• It distinguishes between global approaches that provide overall impact scores and local methods offering instance-specific additive decompositions.
• Practical guidelines recommend method selection based on model structure, computational efficiency, and feature dependency to enhance interpretability.

Feature-Importance Analysis

Feature-importance analysis refers to the quantification and ranking of the relative contributions of individual input features to the output or behavior of a machine learning model. It is a central tool in statistical learning, interpretability, and scientific modeling for diagnosing, auditing, and understanding both the inner workings and external associations of predictive models. Feature-importance methods are highly diverse; they differ in scope (global vs. local), estimand, theoretical guarantees, computational strategy, and assumptions about model structure and data generating process.

1. Foundations and Mathematical Formulation

Let $L = \{(X_1, Y_1), \ldots, (X_n, Y_n)\}$ denote a dataset with $X_i \in \mathbb{R}^p$ and target $Y_i$ (categorical or continuous), and let $f$ denote a fitted model. Feature-importance analysis aims to assign each feature $j$ in $1,\ldots,p$ a score $w_j$ (or, more generally, a vector-valued or local score $w_j(x)$) reflecting its quantitative impact on model output or predictive performance.

1.1 Global Importance Scores

Global methods seek a summary $w_j$ for the impact of $X_j$ over the whole data distribution. Classical approaches include:

• Permutation Feature Importance (PFI): The expected loss increment from shuffling $X_j$ independently from all other features:

$$\mathrm{PFI}_j = \mathbb{E}\bigl[L\bigl(Y, f(\tilde{X}_j, X_{-j})\bigr)\bigr] - \mathbb{E}\bigl[L(Y, f(X))\bigr]$$

where $\tilde X_j$ is $X_j$ permuted independently (Ewald et al., 2024).

• Relative and Conditional Importance (RFI/CFI): Generalizes the above by permuting $X_j$ conditionally given a subset $G$ of other features, preserving dependencies (König et al., 2020). CFI sets $G = X_{-j}$, PFI sets $G = \emptyset$, and RFI interpolates via arbitrary $G$.
• Shapley/Additive Methods (SAGE, SHAP): Assigns credit to each feature as its average marginal contribution across all possible feature subsets $S$, i.e.,

$$\phi_j^{\mathrm{SAGE}} = \sum_{S \subseteq [p] \setminus \{j\}} \frac{|S|!(p - |S| - 1)!}{p!} [v(S \cup \{j\}) - v(S)]$$

with $v(S)$ defined as the expected predictive gain or loss when only $S$ is observed (Johnsen et al., 2021, Ewald et al., 2024).

• Partial Dependence/Sobol Indices: Measures based on the variance or influence of $X_j$ in the partial dependence function $PD_j(x_j) = \mathbb{E}_{X_{-j}}[f(x_j, X_{-j})]$ (Ewald et al., 2024).
• Berkelmans-Pries Method: Model-agnostic, axiomatically grounded global importance via a normalized $\ell_1$-shift dependence function and Shapley allocation, yielding analytically correct results on canonical interactions (Pries et al., 2023).
• Nonparametric Impact/Importance: Based purely on stratified partial dependence and averaging, avoiding any model fit to measure true data-level effect (Parr et al., 2020).

1.2 Local/Instance-wise Importance

Local approaches provide per-sample decompositions, attributing model output $f(x)$ at $x$ across the input dimensions:

• Additive Decomposition: $f(x) = \phi_0 + \sum_{j=1}^p \phi_j(x)$, with $\phi_j(x)$ produced via methods such as SHAP, LIME, TreeInterpreter, SSFI, or MFI (Ayad et al., 11 Feb 2025, Gatto et al., 2019, Vidovic et al., 2016).

1.3 Model Class and Data Structure

• Tree-Specific Methods: Methods tailored to trees/forests exploit the prediction path or logic, e.g., SSFI (Single-Sample Feature Importance for random forests), MUC-driven importance (Minimal Unsatisfiable Core extraction), and DT-Sampler (SAT-based sampling over tree space) (Gatto et al., 2019, Ma et al., 2022, Huang et al., 2023).
• Kernel/Deep Models: For non-linear models, dependence can be measured through covariance in high-dimensional RKHS or kernelized summary statistics (Vidovic et al., 2016).

2. Methodological Variants

A spectrum of methodologies with distinct computational and statistical properties are prominent:

Approach Class	Method/Example	Mechanism
Perturbation-based	PFI, RFI, CFI, Randomized Ablation	Loss increase from shuffling or replacing features
Game-theoretic	SHAP, SAGE, BP-FI	Shapley-value allocation based on marginal effects
Local Surrogate	LIME, LSurro	Local linear model over perturbations
Path/Logic-based	SSFI, MUC, DT-Sampler	Attribution via tree traversal/logic/SAT sampling
Model-free/Nonparametric	cmr (Partial dependence), BP-FI	Data-stratified, no model fit
Confidence/Uncertainty	Bootstrap sub-SAGE, Minipatch LOCO	Statistical confidence intervals/wide-sense error

Key distinctions include whether the method is model-agnostic, the extent to which it accounts for feature dependencies, interpretability, computational scalability, and availability of uncertainty quantification (Ewald et al., 2024, Johnsen et al., 2021, Huang et al., 2023).

3. Interpretation, Assumptions, and Inference

Feature-importance results depend critically on the estimand and perturbation mechanism. For example:

• PFI and mSAGE detect unconditional association, but can be confounded by correlated features unless feature independence is guaranteed.
• CFI, LOCO, cSAGE target the unique, direct (conditional) association of $X_j$ beyond all others, equivalent to asking whether $X_j \not\!\perp\!\!\!\!\perp Y | X_{-j}$ (Ewald et al., 2024).
• RFI, SAGE surplus, WVIM, LOCI allow for analysis conditional on an arbitrary subset, exposing direct and indirect paths, shielding or exposing confounders as desired (König et al., 2020).
• BP-FI is uniquely axiomatic (efficiency, symmetry, null-player, clone-invariance) and exactly recovers ground-truth values on synthetic constructs such as XOR, mixtures, and is model-free (Pries et al., 2023).

Proper inference requires attention to:

• Dependence Structure: PFI is valid only when feature independence holds; model refitting or conditional permutations are needed for direct associations.
• Model Misspecification: If $f$ is not risk-optimal, importance can be shifted from true drivers to proxies or spurious correlates.
• Statistical Uncertainty: Bootstrap, permutation, or sampling-based variance estimation is necessary for confidence in variable rankings, especially in $p \gg n$ regimes or when reported importance values are close (Johnsen et al., 2021, Neuhof et al., 2023).

4. Computation and Complexity

Practical aspects vary:

• Model-Agnostic Methods (e.g., PFI, LOCO, minipatch LOCO): $O(np)$ for $n$ samples, $p$ features, but can be expensive with large $p$ due to repeated perturbation.
• Tree-specific Methods (DT-Sampler, SSFI): DT-Sampler encodes all trees of fixed size and accuracy as CNF and uses SAT-sampling; complexity depends on tree size $N$ and feature count $K$ (hard above $K \approx 50, N \approx 20$) (Huang et al., 2023). SSFI is linear in tree size and data (Gatto et al., 2019).
• Game-theoretic Shapley estimates: Full SAGE has $O(2^p)$ terms; sub-SAGE or Sampling-SHAP take polynomial time at the expense of some Shapley axioms (Johnsen et al., 2021, Ayad et al., 11 Feb 2025).
• Confidence Intervals: Approaches leveraging minipatch ensembles or paired bootstrapping (as in sub-SAGE) provide valid asymptotic or finite-sample CIs without the cost of refitting per feature (Gan et al., 2022, Johnsen et al., 2021).

5. Empirical Results and Practical Guidelines

Extensive benchmarking has revealed empirical principles:

• Tree-based Ranking: For regression trees/random forests, feature importance computed by mean decrease in impurity, permutation tests, or SSFI consistently isolates top algorithmic drivers—validated in process control and chemical engineering applications (Garcia-Ceja et al., 2020).
• Local Explanation Stability: Tree-SHAP and Sampling-SHAP yield remarkably consistent local importances, while LIME, local surrogates, and TreeInterpreter may bias towards irrelevant or noisy features, especially under correlation or high noise (Ayad et al., 11 Feb 2025).
• Nonparametric Impact: cmr-based partial dependence delivers model-free, interpretable feature impact, robust to model misspecification and suitable for correlated predictors in scientific and medical domains (Parr et al., 2020).
• Ranking Uncertainty: Confident Feature Ranking (pairwise CIs on feature ranks) is necessary in applications with low sample support or when feature rankings are used for decision making/variable selection (Neuhof et al., 2023).
• DT-Sampler Stability: Decision tree sampling with SAT encoding over fixed-size, high-accuracy trees reduces variance in importance scores and yields concise, interpretable models for high-stakes contexts (Huang et al., 2023).
• Speed/efficiency: Coefficient of Variation (CoV) of path contributions allows for rapid model-selection in trees, matching cross-validation accuracy at much lower computational cost (Fang et al., 2020).

Quantitative findings include sub-SAGE yielding valid 95% confidence intervals and recovery of known scientific loci in genetics (Johnsen et al., 2021), or symbolic regression of closure models in turbulence with 6–7x speedup using permutation importance as a feature filter (Wu et al., 9 May 2025).

6. Limitations, Open Challenges, and Future Directions

Feature-importance analysis is subject to many recognized limitations:

• Interaction Attribution: Most methods cannot sharply decompose interaction effects; Shapley values average over coalition contributions, often diluting unique importance for strongly synergistic variables (Ewald et al., 2024, Pries et al., 2023).
• Dependence and Causal Structure: Without explicit modeling of dependencies or causal graph structure, global importance scores can confound direct and indirect effects (König et al., 2020, Främling, 2023).
• No single gold standard: Large-scale evaluations show that most popular procedures fail basic axioms or ground-truth recovery, with only BP-FI passing all 18 canonical property-tests across diverse synthetic setups (Pries et al., 2023).
• Complexity and Sample Size: Model-agnostic or game-theoretic methods are exponential in $p$ unless approximated; DT-Sampler and MUC-driven approaches scale poorly with very large feature sets (Huang et al., 2023, Ma et al., 2022).
• Faithfulness and Sufficiency: Faithfulness metrics (comprehensiveness, sufficiency) reveal that many explainers do not robustly identify true driver features, especially under high noise or in presence of engineered/inactive predictors (Conti et al., 19 Nov 2025, Ayad et al., 11 Feb 2025).
• Out-of-distribution explanations: Any perturbation-based explanation can produce infeasible feature combinations unless care is taken to match the intervention distribution to data support (Ewald et al., 2024, Främling, 2023).

Proposed future directions include principled decomposition of interaction effects, integration with causal inference frameworks, automated selection of the appropriate FI method for a given scientific or regulatory objective, and further empirical benchmarking on datasets with known ground-truth (Ewald et al., 2024).

7. Implementation Guidance and Practical Recommendations

• Match method to goal: For fast per-instance explanations in random forests, use SSFI; for rigorous conditional attribution and uncertainty, employ sub-SAGE with bootstrap; for global model-agnostic impact, nonparametric partial dependence (cmr) or BP-FI is preferred (Gatto et al., 2019, Johnsen et al., 2021, Parr et al., 2020, Pries et al., 2023).
• Account for correlation and causality: Default to refitting-based or conditional (CFI/LOCO/relative) importances when dependencies or possible confounders are present, especially in scientific or policy contexts (Ewald et al., 2024, König et al., 2020).
• Check stability: Compute confidence intervals/rank intervals on importance estimates, and perform subsampling or bootstrapping to verify robustness (Johnsen et al., 2021, Neuhof et al., 2023).
• Handle high dimensions: For $p \gg n$, employ screening, dimension reduction or sampling-based Shapley/replacement to keep computational cost and variance in check (Johnsen et al., 2021, Pries et al., 2023).
• Use domain knowledge for discretization: For BP-FI and nonparametric impact, careful discretization or stratification is needed for continuous variables to obtain reliable scores (Pries et al., 2023).
• Beware of out-of-distribution sampling: Masking and perturbation should be performed such that synthetic or edge-case samples do not distort model evaluations (Främling, 2023, Ewald et al., 2024).
• Interpret results in context: Zero importance may imply true irrelevance, absence in the training support, or insufficient model fit; only under strong optimality/correct distributional assumptions do null importance results guarantee independence (Ewald et al., 2024).

Feature-importance analysis thus constitutes a multifaceted set of methods, each with rigorously defined limits and explicit statistical assumptions, and with a mathematical taxonomy unified by notions of perturbation, marginal gain, conditional association, and information attribution, all of which must be grounded in the data and the modeling context for valid scientific or operational use.