Random-Forest Teacher: Guided Feature Selection

• Random-Forest Teacher is a methodology that leverages external feature importance to guide tree splits, enhancing interpretability and promoting sparsity.
• It utilizes parallel tree construction to achieve high computational efficiency while maintaining decorrelated errors in ensemble models.
• Empirical evaluations show that guided feature selection improves accuracy and reduces the feature set size in domains like genomics, text mining, and image analysis.

A Random-Forest Teacher is a system, algorithm, or methodological contribution that teaches, selects, guides, or interprets random forest models either for improved feature selection, better interpretability, enhanced computational efficiency, or as a means of providing insight into the behavior and structure of ensemble-based learners. The notion encompasses both algorithmic innovations (such as the Guided Random Forest for feature selection) and educational tools (such as visualization packages for demystifying random forest predictions), as well as theoretical perspectives (such as kernel and density analogies for forest proximity), all aiming to convey or leverage knowledge about random forests for practical or didactic purposes.

1. Guided Random Forest: Feature Selection by Supervision

Guided Random Forest (GRF) formalizes the use of external guidance—typically in the form of feature importance weights—to steer the variable selection process during random forest tree construction (Deng, 2013). In GRF, each candidate feature split at a tree node is evaluated by a weighted gain function: $$\text{gain}_G(X_i) = \lambda_i \cdot \text{gain}(X_i)$$ where $\text{gain}(X_i)$ is the standard impurity-based gain (e.g., Gini), and $\lambda_i$ is a feature-specific weight determined as: $$\lambda_i = 1 - \gamma + \gamma \cdot (\text{Imp}_i/\text{Imp}^*)$$ with $\text{Imp}_i$ the importance score (e.g., MeanDecreaseGini) from a preceding RF and $\gamma \in [0,1]$ a penalty parameter controlling the influence of importance scores. When $\gamma=1$, splitting is maximally guided by prior feature importance; when $\gamma=0$, the GRF collapses to a standard RF.

This construction enables the model to automatically down-weight low-importance features during tree growth, thereby encouraging sparsity and interpretability while preserving the parallelizable structure of the ensemble (unlike sequential methods such as GRRF, which promote feature sparsity at the expense of parallelism).

2. Parallel Tree Construction and Optimization

The independence of tree construction in GRF provides a critical computational advantage, especially on high-dimensional datasets (Deng, 2013). Unlike the Guided Regularized Random Forest (GRRF), which builds trees sequentially (each new tree potentially conditioned on features selected earlier), all GRF trees leverage the same externally provided guidance, allowing the entire ensemble to be constructed in parallel.

This architectural shift means that GRF achieves both high computational throughput (benefiting from multi-core and distributed environments) and decorrelated tree errors, enhancing ensemble strength and robustness compared to sequentially built correlated ensembles.

3. Empirical Evaluation and Performance Analysis

In the evaluation on ten high-dimensional gene expression datasets, GRF-based feature selection demonstrates that using RF on the subset of features selected by GRF (termed GRF-RF) consistently outperforms RF trained on all features, both in terms of accuracy and statistical significance: on 9/10 datasets, the improvement is documented, with 7/10 showing statistically significant gains ($p < 0.05$) (Deng, 2013).

These experiments further highlight a key trade-off: although GRF sometimes selects more features than methods such as GRRF, its classification accuracy is higher. For example, in a simulation with 500 features, GRF selects 196 features, a drastic reduction, and the resultant classifier is stronger. This bolsters the case for using guided forests in high-stakes, high-dimensional classification tasks where both interpretability and accuracy are required.

4. Parameterization and Tuning

The only essential tuning parameter in GRF is $\gamma$, which aligns the strength of external guidance with the gain function. The range of $\gamma$ values controls the sparsity-accuracy trade-off:

• $\gamma = 0$: No guidance, fully random forest.
• $\gamma = 1$: Maximum use of importance; aggressive feature penalization.
• $0 < \gamma < 1$: Intermediate guidance; balance between inclusion and parsimony.

Empirical results indicate that fixed, non-tuned $\gamma=1$ can already produce feature sets that yield highly accurate and interpretable models, but further adjustment is possible to meet application-specific needs.

5. Implementation specifics in RRF Package

GRF is available from version 1.4 onwards in the RRF (Regularized Random Forest) R package. Implementation proceeds as follows:

library(RRF) set.seed(1) X <- matrix(runif(500*500, min=-1, max=1), ncol=500) Y <- (X[,1]) + (X[,21]) ix <- which(Y > quantile(Y, 1/2)) Y <- rep(-1, length(Y)); Y[ix] <- 1 trainX <- X[1:250, ] trainY <- Y[1:250] testX <- X[251:500, ] testY <- Y[251:500] RF <- RRF(trainX, flagReg = 0, as.factor(trainY)) imp <- RF$importance[,"MeanDecreaseGini"] impRF <- imp / max(imp) gamma <- 1 coefReg <- (1 - gamma) + gamma * impRF GRF <- RRF(trainX, as.factor(trainY), flagReg = 0, coefReg = coefReg)

Key processing steps:

• Feature importances are extracted from a standard RF and normalized.
• The $\lambda_i$ coefficients (coefReg) are computed as per the $\gamma$ value.
• The GRF is constructed using these coefficients, guiding all trees without interdependency.

The approach is fully compatible with further application of "RF on selected features" (the GRF-RF pipeline), which is empirically shown to enhance predictive accuracy (Deng, 2013).

6. Interpretability and Application Domains

GRF directly addresses the interpretability challenge common in ensemble methods by explicit feature selection and reduction. The explanatory source of sparsity, $\lambda_i$, can also be set not only from data-driven RF importances but extended (e.g., by user-specified or domain-driven weights), reflecting, for example, human insight or prior scientific knowledge (Deng, 2013). This flexibility highlights GRF’s suitability for domains requiring both data adaptivity and domain transparency.

Applications extend well beyond bioinformatics (genomics), such as:

• Text mining: identifying significant terms from high-dimensional document-term matrices.
• Image analysis: selection of key visual features among many descriptors.
• Finance: distilling relevant market indicators.
• IoT and sensor networks: filtering essential signals from multidimensional sensor feeds.

The fully parallelizable nature and deterministic feature selection mechanics of GRF are especially advantageous where computational scalability and transparency are prioritized.

7. Synthesis and Significance Relative to Other Methods

GRF augments the classic random forest paradigm by shifting from uniform treatment of features to guided, weighted selection, thereby supporting domain adaptation, interpretability, and computational efficiency. Unlike sequential approaches (e.g., GRRF) that may introduce significant tree correlation and limited scalability, GRF’s independent tree construction maintains randomness while applying directional penalization to less informative features.

Table: Distinction of GRF and Related Feature Selection Approaches

Method	Feature Guidance	Tree Dependency	Parallelizable	Feature Subset Size	Accuracy
RF	None	Independent	Yes	All	Baseline
GRF	External (weights)	Independent	Yes	Moderate	Improved (on most datasets)
GRRF	External (weights)	Sequential	No	Fewest	Sometimes lower

By blending statistical rigor (through penalized splits) with practical engineering (parallel training), GRF stands as a canonical methodological “Random-Forest Teacher”—it teaches the forest which features to use, enables scalable learning, and exposes the process for scientific scrutiny and practical deployment (Deng, 2013).