Hyperparameters and Tuning Strategies for Random Forest

Abstract: The random forest algorithm (RF) has several hyperparameters that have to be set by the user, e.g., the number of observations drawn randomly for each tree and whether they are drawn with or without replacement, the number of variables drawn randomly for each split, the splitting rule, the minimum number of samples that a node must contain and the number of trees. In this paper, we first provide a literature review on the parameters' influence on the prediction performance and on variable importance measures. It is well known that in most cases RF works reasonably well with the default values of the hyperparameters specified in software packages. Nevertheless, tuning the hyperparameters can improve the performance of RF. In the second part of this paper, after a brief overview of tuning strategies we demonstrate the application of one of the most established tuning strategies, model-based optimization (MBO). To make it easier to use, we provide the tuneRanger R package that tunes RF with MBO automatically. In a benchmark study on several datasets, we compare the prediction performance and runtime of tuneRanger with other tuning implementations in R and RF with default hyperparameters.

• The paper demonstrates that tuning hyperparameters like mtry, sample size, node size, and number of trees can lead to measurable improvements in metrics such as AUC, Brier score, and log-loss.
• It compares various tuning strategies, including manual, grid, random search, and SMBO, highlighting SMBO’s efficiency in exploring the hyperparameter space.
• Empirical results underline that while default settings perform adequately, customized tuning can significantly enhance predictive accuracy in complex data scenarios.

Hyperparameters and Tuning Strategies for Random Forest

Overview of Key Hyperparameters

The paper "Hyperparameters and Tuning Strategies for Random Forest" (1804.03515) extensively examines the crucial hyperparameters of the Random Forest (RF) algorithm and their effects on model performance. RF, introduced by Breiman, has become an essential non-parametric technique for both classification and regression tasks due to its robustness and predictive power. Key hyperparameters include the number of variables at each split (mtry), sample size, node size, number of trees, and the splitting rule. Their proper configuration is essential for optimizing the predictive capability and computational efficiency of RF.

Effect of Hyperparameters on Model Performance

The study reviews existing literature on the impact of each hyperparameter. The mtry parameter, suggesting the number of candidate variables for splitting, influences tree diversity and prediction stability. Default values, commonly $\sqrt{p}$ for classification and $p/3$ for regression (where $p$ is the number of variables), can often be exceeded for improved outcomes. The sample size and replacement options affect tree orthogonality and accuracy but must be tuned to the dataset context. Node size determines tree depth; a smaller size can lead to overfitting, particularly for small sample sizes.

The number of trees, typically set high, should be sufficient to achieve stable performance benefits without substantial computational overhead. Splitting criterion options, such as Gini impurity and others, extend RF's flexibility, albeit introducing biases that require careful management, especially in high-dimension or imbalanced datasets.

Tuning Strategies: Approaches and Software

The paper transitions into discussing various tuning strategies, emphasizing the importance of optimizing hyperparameters to surpass default settings typically offered by software. It covers manual, grid, and random search methodologies and advances into sophisticated techniques like Sequential Model-Based Optimization (SMBO), which efficiently explores the hyperparameter space by iterating evaluations of promising configurations. The actual performance benchmarks confirm that SMBO-style optimization, as implemented in the tuneRanger R package, yields substantial improvements in predictive accuracy and computational efficiency over alternatives.

Benchmark Study: Comparative Analysis

To validate tuning practices, an extensive benchmark was conducted using datasets from OpenML. The results consistently highlight the marginal yet sometimes critical improvements in AUC, Brier score, and log-loss when mtry, sample size, and node size are finely tuned. The study reaffirms that while RF may relatively benefit less from tuning than other models like SVM, the performance enhancements in select scenarios justify the customization effort.

Implications and Future Directions

This research emphasizes the nuanced role of hyperparameter tuning in extracting superior performance from RF models. Practically, insights are instrumental for practitioners handling complex data structures or facing performance saturation with default settings. The findings also highlight gaps in the literature, particularly on variable importance measures, warranting focused empirical research to refine variable selection and better understand RF's broader applicability.

Conclusion

In conclusion, the paper provides a comprehensive framework and practical guidance for optimizing Random Forests through tailored hyperparameter tuning strategies. Through this detailed empirical analysis, it not only delineates prevalent gaps in RF methodology but also sets a foundation for ongoing exploration into enhancing automated machine learning pipelines and tailored model configurations.