Random Forest (RF) Overview

• Random Forest (RF) is an ensemble learning method that builds multiple decision trees using bootstrap sampling and random feature subsets for robust classification and regression.
• It stabilizes predictions by averaging outputs from diverse trees, reducing variance and preventing overfitting while maintaining non-parametric flexibility.
• Extensions such as pruning, weighting, and online adaptations enhance RF's interpretability and efficiency in handling high-dimensional and complex datasets.

Random Forest (RF) is an ensemble learning method for classification and regression that was developed by Breiman and is built by combining many decision trees grown on randomized versions of the training data. In the standard formulation, each tree is trained on a bootstrap sample and each split is chosen from a random subset of features; classification is obtained by majority vote and regression by averaging. RF is described in the cited literature as a fully non-parametric, robust, nonlinear method that optimizes predictive accuracy by stabilizing tree estimates through ensembling, while also serving as a foundation for pruning, weighting, feature-selection, visualization, online, and dependence-aware extensions (Fawagreh et al., 2015, Ehrlinger, 2015).

1. Ensemble formulation and predictive mechanism

Given a training set with $N$ samples and $S$ features, RF grows a collection of trees by repeatedly drawing bootstrap samples from the training data and recursively splitting nodes using randomly sampled feature subsets. In the formulation described for classification, each tree $T_i$ is built by drawing a bootstrap sample of instances, sampling candidate features at each node, choosing the best split feature $B(\overrightarrow{F_S})$, and continuing until no more instances remain to split on. The resulting forest is a vector of trees $\overrightarrow{RF}$ or $RF=\langle t_1,t_2,\dots,t_n\rangle$ (Fawagreh et al., 2015, Fawagreh et al., 2015).

For classification, the ensemble prediction is the class receiving the most votes from the constituent trees,

$$\hat{y}(x)=\operatorname{mode}\{h_t(x)\mid t=1,\dots,T\},$$

while for regression the forest prediction is the average of the tree outputs,

$$\hat{f}_{rf}(x)=\frac{1}{B}\sum_{b=1}^{B}\hat{f}_b(x).$$

The cited regression literature also emphasizes an equivalent weighted-nearest-neighbor representation in which RF predictions can be written as a weighted sum of training responses,

$\hat{y}(x_0)=\sum_{i=1}^{n} w_i(x_0)y_i,$

with nonnegative weights summing to one (Ehrlinger, 2015, Koster et al., 2024).

Bootstrap aggregation induces the familiar in-bag and out-of-bag decomposition. In the regression tutorial based on randomForestSRC, about $63.2\%$ of the training cases appear in a bootstrap sample and the remaining $S$0 form the out-of-bag set for that tree. OOB prediction is then formed by averaging only over trees that did not use the corresponding observation, yielding an internal error estimate that plays the role of test error without explicit cross-validation (Ehrlinger, 2015).

This ensemble formulation is central to later reinterpretations of RF. It supports classical point prediction, class-probability estimation, weighted empirical forecast distributions, and a variety of post-hoc transformations that do not alter the trained trees themselves but instead alter how the forest is read, compressed, or aggregated.

2. Diversity, generalization, and statistical behavior

A recurring theme in the RF literature is that ensemble performance depends on both the strength of individual trees and their diversity. The pruning papers explicitly build on the empirical claim that ensembles tend to perform better when there is significant diversity among the constituent models. In that setting, diversity between two classifiers $S$1 and $S$2 is measured by the fraction of training instances on which they disagree,

$S$3

where $S$4 if $S$5 and $S$6 otherwise (Fawagreh et al., 2015).

This disagreement-based view underlies both classical RF intuition and later diversity-aware refinements. If many trees behave similarly, the forest accumulates redundancy; if they make different errors while remaining individually competent, majority vote or averaging can reduce variance and improve generalization. The pruning literature explicitly interprets large forests as containing substantial redundancy, and several methods attempt to identify small subsets of trees that preserve or improve performance by maximizing behavioral diversity (Fawagreh et al., 2015).

A related line of work argues that RF should not be understood primarily through a naïve notion of model capacity. In particular, "There is no Double-Descent in Random Forests" argues that when complexity is measured using theoretically grounded quantities based on the average Rademacher complexity of the constituent trees, RF exhibits a single descent rather than a double-descent curve. The same paper emphasizes that diversity and bias both matter: too little diversity collapses the ensemble toward a single tree, while too much diversity yields trees that no longer produce correct outputs with sufficient frequency; between these extremes lies a broad regime of roughly equal performance (Buschjäger et al., 2021).

For high-dimensional sparse regression, the diversity story interacts with feature selection. "Targeting predictors in random forest regression" shows that a pre-estimation targeting step can increase the probability of splitting along strong predictors, strengthen individual trees, and improve predictive accuracy, while simultaneously increasing correlation among trees. In the macroeconomic and financial applications reported there, the best trade-off typically occurs when selecting the best $S$7 of predictors, with improvements in predictive accuracy of targeted RF relative to ordinary RF of up to $S$8 (Borup et al., 2020).

Taken together, these results frame RF as a method whose empirical success depends less on unrestricted capacity than on the interaction among bootstrap resampling, feature subsampling, tree strength, and ensemble diversity.

3. Interpretability, diagnostics, and probabilistic representations

Although RF is often treated as a black box, the cited work describes a substantial interpretability toolkit. In regression, the ggRandomForests workflow built on randomForestSRC emphasizes OOB error curves, permutation variable importance (VIMP), minimal depth, variable dependence, partial dependence, and interaction diagnostics. In the Boston Housing example, randomForestSRC uses defaults such as ntree = 1000, mtry = 5, and nodesize = 5, and reports OOB mean squared error as the model’s error rate (Ehrlinger, 2015).

Permutation VIMP measures the degradation in OOB prediction error after permuting a predictor, while minimal depth is a structural importance measure based on how close to the root a variable first appears in a tree. Smaller minimal depth indicates stronger importance, and var.select computes both minimal depth and VIMP. Variable dependence plots visualize OOB predictions against individual predictors, whereas partial dependence plots estimate a risk-adjusted marginal effect by averaging predictions over the empirical distribution of the remaining covariates (Ehrlinger, 2015).

For survival analysis, Random Survival Forests extend the RF construction to right-censored time-to-event data by replacing the split rule with a survival criterion such as the log-rank statistic and by assigning each terminal node a Kaplan–Meier or cumulative hazard estimate. The ggRandomForests survival tutorial shows how the same visualization strategy transfers to RSF, including OOB error trajectories, VIMP, minimal depth, survival curves, variable dependence, partial dependence, coplots, and interaction surfaces on the Primary Biliary Cirrhosis data (Ehrlinger, 2016).

Another interpretability route uses the RF proximity matrix. "Visualizing Random Forest with Self-Organising Map" defines proximity $S$9 by counting how often two samples fall into the same terminal node across trees and normalizing by the number of trees, with dissimilarity $T_i$0. That paper argues that RF-SOM offers a more feature-level view of the forest than RF-MDS, because each neuron in the self-organizing map remains an interpretable prototype vector while the topology is driven by RF-induced dissimilarity (Płoński et al., 2014).

Probabilistic forecasting extends the weighted-average interpretation. In quantile-style RF representations, the conditional distribution at $T_i$1 is the weighted empirical distribution of training responses. "Simplifying Random Forests' Probabilistic Forecasts" exploits this by sparsifying the RF weights: keeping only the top-$T_i$2 largest weights and renormalizing them. On 18 regression datasets, Top20 typically matches full RF in median CRPS and Top50 slightly improves it; for mean forecasts, Top20 and Top50 often outperform the full RF in median squared error. The paper interprets this as evidence that many RF forecast distributions concentrate most of their mass on a relatively small subset of influential training points (Koster et al., 2024).

4. Pruning, compression, and tree weighting

A major branch of RF research treats the fitted forest as overcomplete and seeks either to prune the ensemble aggressively or to replace trees with more compact surrogates. "An Outlier Detection-based Tree Selection Approach to Extreme Pruning of Random Forests" introduces LOFB-DRF, in which each tree is represented by its vector of predictions on the training set, Local Outlier Factor scores are computed in that prediction space, and trees are ranked by

$T_i$3

The top-$T_i$4 trees form a pruned forest. On 10 real datasets, the paper reports pruning levels between $T_i$5 and $T_i$6 while matching or surpassing RF accuracy in most cases, and states that in most cases $T_i$7 times faster classification can be achieved with the $T_i$8 pruning level (Fawagreh et al., 2015).

A related method, CLUB-DRF, clusters tree prediction vectors with a categorical clustering scheme and retains the best OOB-performing representative from each cluster. On 15 UCI datasets, this clustering-based pruning method reports extreme pruning levels of $T_i$9 while often retaining or improving RF accuracy. The motivating interpretation is the same: large forests contain many behaviorally redundant trees, and one representative from each cluster can suffice for prediction (Fawagreh et al., 2015).

Compression can also be carried out at the level of individual trees rather than at the level of forest membership. "An Approximation Method for Fitted Random Forests" proposes replacing each fitted tree by a multinomial logistic regression, or a GAM extension, that predicts the probability that an input falls into each leaf. The resulting surrogate forest stores regression coefficients and leaf-wise responses rather than split structures and thresholds, with the goal of significantly reducing model size while preserving prediction quality in large-scale production settings and on small devices with limited memory (Popuri, 2022).

Unequal tree weighting provides a complementary route. "Optimal Weighted Random Forests" studies regression forests in which the usual equal-weight average is replaced by data-driven weights chosen by Mallows-type criteria. The paper proposes 1step-WRF$B(\overrightarrow{F_S})$0 and 2steps-WRF$B(\overrightarrow{F_S})$1, proves that under some regularity conditions their squared loss and risk are asymptotically identical to those of the infeasible best possible model averaging estimator, and reports that they outperform the equal-weight forest and two other weighted RFs in most cases on real-world datasets (Chen et al., 2023).

These lines of work share a common premise: fitted RF objects can be smaller, faster, or better calibrated than the default equal-weight, full-ensemble formulation without changing the basic tree vocabulary of the method.

5. Extensions for feature selection, extrapolation, dependence, and online learning

Several RF variants modify the basic algorithm to address specific structural weaknesses. For feature selection in high-dimensional settings, "Guided Random Forest in the RRF Package" defines a guided split gain

$B(\overrightarrow{F_S})$2

where $B(\overrightarrow{F_S})$3 is a feature-importance score from an initial ordinary RF. With $B(\overrightarrow{F_S})$4, the gain is scaled directly by normalized importance. On 10 high-dimensional gene datasets, RF applied to the features selected by GRF outperformed RF applied to all features on 9 datasets, with significant differences at the $B(\overrightarrow{F_S})$5 level on 7 of them (Deng, 2013).

In regression problems involving global trends or extrapolation, "Regression-Enhanced Random Forests" proposes fitting a penalized parametric regression on an expanded feature set, computing residuals, and then fitting an RF to those residuals. The final predictor is

$B(\overrightarrow{F_S})$6

Because the parametric component can extrapolate outside the range of the training responses while the forest models local residual structure, RERF improves over RF in the paper’s extrapolation-oriented simulation and real-data examples (Zhang et al., 2019).

For dependent errors, "Random Forests for dependent data" develops RF-GLS, which rewrites tree splitting as a global OLS optimization and then replaces the OLS loss with a GLS loss using a working precision matrix $B(\overrightarrow{F_S})$7. In that framework, standard RF becomes the special case with an identity working covariance matrix, and the paper proves consistency of RF-GLS under beta-mixing error processes. As a byproduct, it also establishes consistency of RF for beta-mixing processes (Saha et al., 2020).

For streaming data, "AMF: Aggregated Mondrian Forests for Online Learning" replaces batch-grown CART-style trees with online Mondrian trees and uses a Context Tree Weighting recursion to perform exact aggregation over all prunings of each tree. The resulting predictor is competitive with the optimal pruning of the Mondrian tree while remaining fully online, one-pass, and parameter-free in the sense emphasized by the paper (Mourtada et al., 2019).

These extensions show that RF is not a single immutable algorithm but a family of procedures whose tree ensemble core can be adapted to high-dimensional sparsity, extrapolation, dependence, and online learning constraints without abandoning the forest paradigm.

6. Diversity-conscious recent refinements, limitations, and recurring debates

Recent work continues to revisit the two classical RF ingredients—tree strength and diversity—by making feature usage across trees explicitly non-uniform. "Heterogeneous Random Forest" introduces feature-depth memory across trees: features used near the root in earlier trees receive lower sampling weights in later trees. The method defines feature depth $B(\overrightarrow{F_S})$8, cumulative depth $B(\overrightarrow{F_S})$9, and feature weights

$\overrightarrow{RF}$0

so that features used deep or not used at all become more likely to appear in subsequent feature subspaces. On 52 datasets, HRF reportedly outperformed other ensemble methods in terms of accuracy across the majority of datasets, while simulation studies indicated that its benefits were strongest on datasets with fewer noise features (Kim et al., 2024).

A related 2025 proposal, the "Diversity Conscious Refined Random Forest," combines iterative feature refinement with correlation-based clustering of trees. Features are ranked by an information-gain-ratio-based importance measure weighted by OOB tree quality; the forest is then refined and finally pruned by clustering highly correlated trees and retaining the highest-AUC representative from each cluster. On 8 benchmark datasets, the method reports improved accuracy compared to standard RF and tree-count reductions ranging from a few percent to nearly half, depending on the dataset (Bhattarai et al., 1 Jul 2025).

At the same time, the literature repeatedly documents limitations that remain intrinsic to RF or arise in specific deployments. Large forests can impose substantial memory and latency costs; interpretability remains difficult even with VIMP and partial dependence; full RF prediction is bounded by the observed response range in regression, which limits extrapolation; and post-hoc pruning or targeting can underperform on some datasets even when average results are favorable. The pruning papers explicitly note cases such as the audit and vote datasets where the original RF slightly outperforms all tested pruned variants, albeit by small margins (Fawagreh et al., 2015).

A persistent misconception is that RF succeeds simply because it is a large-capacity ensemble of overfitted trees. The cited work suggests a narrower conclusion. RF is effective because its randomness, averaging, and diversity mechanisms place it in a robust regime where deep trees can be combined without the classical overfitting behavior of a single tree, while later extensions exploit the same principle by reshaping tree selection, tree weighting, feature access, or post-hoc aggregation rather than abandoning the forest itself (Buschjäger et al., 2021).