MDI-oob: Unbiased Feature Importance in RF

• MDI-oob is a debiased feature importance estimator that uses out-of-bag samples to accurately assess variable influence in Random Forests.
• It recalculates impurity decreases on unseen data to mitigate overrating noisy features and improve feature ranking.
• Empirical results in genomic and simulated datasets show that MDI-oob outperforms standard MDI while maintaining robust interpretability.

MDI-oob (Mean Decrease Impurity out-of-bag) is a debiased feature importance estimator for Random Forests. Building on the canonical Mean Decrease Impurity (MDI) introduced by Breiman, MDI-oob corrects the systematic bias that plagues standard impurity-based importances, especially the tendency to overvalue irrelevant (noisy) features. The core innovation is the application of out-of-bag (oob) samples—data not used in the construction of a tree—to estimate impurity decreases, thereby removing the bias induced by in-sample impurity reduction calculations. MDI-oob achieves state-of-the-art feature selection accuracy in both simulated and real-world datasets, including genomic ChIP data, and is applicable to both shallow and deep tree ensembles (Li et al., 2019).

1. Theoretical Foundations and Analytical Debiasing

Traditional MDI calculates the importance of feature $j$ in a Random Forest as the sum, over all splits involving $j$, of the decrease in impurity (such as Gini or entropy) weighted by the proportion of training samples reaching the split. However, as shown analytically, this in-sample evaluation is systematically biased: random (uninformative) features can receive spuriously high importance scores merely due to randomness, and this effect worsens for deeper trees (Li et al., 2019). The paper provides a tight non-asymptotic bound for the expected bias of MDI, rigorously quantifying that deeper trees exhibit higher expected feature selection bias.

To address this, the authors derive a new analytical expression for MDI based on the original Breiman definition, which supports unbiased estimation when impurity decrease is calculated on samples that were not used to train the corresponding tree structure. This insight is central to the MDI-oob construction.

2. Methodology and Algorithmic Framework

MDI-oob modifies the calculation pipeline for feature importance as follows:

1. Tree Training: Train each decision tree in the Random Forest on a bootstrap sample from the dataset.
2. Out-of-bag Identification: For every tree $T$, maintain the set of oob samples $Z^{oob}_T$, i.e., data not used in training $T$.
3. Split Traversal and Impurity Computation: For each internal split $s$ of $T$ on feature $j$, recompute the impurity decrease by applying the split to the oob samples that reach the corresponding node.
4. MDI-oob Aggregation: Feature importance for $j$ is accumulated over all splits across all trees, using the out-of-bag-based impurity decrease at each split.

The key technical step is the replacement of in-sample impurity estimates with oob-based estimates at the split level, producing an unbiased estimator of the true feature importance expected over fresh data (Li et al., 2019).

3. Comparative Performance and Empirical Evaluation

MDI-oob demonstrates superior empirical performance to traditional MDI in both controlled simulations and real genomic datasets (specifically, a ChIP-seq dataset). In both settings, MDI-oob provides more accurate ranking of truly informative features and avoids overrating noise variables. This is evident for both deep and shallow forests, confirming that bias correction is robust across tree depths (Li et al., 2019).

Results show that where traditional MDI assigns high importances even to randomly permuted features, MDI-oob collapses their scores towards zero (as desired), thus minimizing feature selection bias without sacrificing detection power for genuinely predictive covariates.

4. Mathematical Formalism and Notation

For a feature $j$, let $j$0 denote the set of all splits on $j$1 across all trees. For split $j$2 in tree $j$3, let $j$4 denote the impurity decrease, computed on oob samples. Then the MDI-oob importance score for $j$5 is formalized as:

$j$6

where $j$7 is the impurity function, $j$8 and $j$9 are the sets of oob samples reaching split $T$0 and its $T$1-th child respectively, and $T$2 weights by the proportion of oob samples at node $T$3 (Li et al., 2019).

5. Practical Implications and Use Cases

MDI-oob is valuable in any application of Random Forests where reliable feature ranking is critical, especially in high-dimensional domains such as genomics, bioinformatics, and variable selection in scientific modeling. Its ability to deliver reliable importances without over-fitting to noise makes it preferable over standard impurity-based metrics.

A plausible implication is that workflows relying on MDI importances for feature selection, causal inference, or interpretability can benefit from substituting MDI-oob as a direct drop-in replacement, especially in regimes with many weak or irrelevant features (Li et al., 2019).

6. Limitations and Further Directions

While MDI-oob addresses bias in impurity-based importance estimation, the method inherits computational cost from the necessity to rerun impurity calculations on out-of-bag samples at every split. In extremely large datasets or forests with very large oob sets, this may increase computational demands. The authors do not report pathologies regarding variance inflation, but this remains an open area for further empirical study.

A plausible direction for extension is the adaptation of MDI-oob estimators to alternative tree-based ensembles or the generalization to other impurity-based metrics, provided proper definition of out-of-sample estimates at each split.

7. Summary Table: Standard MDI vs. MDI-oob

Aspect	Standard MDI	MDI-oob
Impurity samples	In-sample (training data)	Out-of-bag samples
Bias to noise vars	Systematic, worsens with tree depth	Eliminated by construction
Empirical accuracy	Spurious importances possible	State-of-the-art feature selection
Computational cost	Lower	Slightly higher (oob impurity evals)

The use of MDI-oob is motivated by its ability to provide debiased feature importance estimates in Random Forests, significantly improving the interpretability and reliability of variable selection outputs (Li et al., 2019).