Towards understanding the bias in decision trees (2501.04903v3)

Abstract: There is a widespread and longstanding belief that machine learning models are biased towards the majority (or negative) class when learning from imbalanced data, leading them to neglect or ignore the minority (or positive) class. In this study, we show that this belief is not necessarily correct for decision trees, and that their bias can actually be in the opposite direction. Motivated by a recent simulation study that suggested that decision trees can be biased towards the minority class, our paper aims to reconcile the conflict between that study and decades of other works. First, we critically evaluate past literature on this problem, finding that failing to consider the data generating process has led to incorrect conclusions about the bias in decision trees. We then prove that, under specific conditions related to the predictors, decision trees fit to purity and trained on a dataset with only one positive case are biased towards the minority class. Finally, we demonstrate that splits in a decision tree are also biased when there is more than one positive case. Our findings have implications on the use of popular tree-based models, such as random forests.

• The paper provides a detailed theoretical and empirical analysis showing that decision trees can bias toward the minority class under specific imbalanced conditions.
• It introduces key theorems demonstrating unbiased performance with a single predictor but reveals positive bias when multiple predictors guide splits.
• Simulation studies confirm that non-optimal splits in decision trees overpredict minority prevalence, highlighting the need for refined techniques in imbalanced settings.

Understanding Bias in Decision Trees for Imbalanced Data

The paper authored by Nathan Phelps, Daniel J. Lizotte, and Douglas G. Woolford provides a comprehensive examination of the bias inherent in decision trees, especially when applied to imbalanced data sets. There exists a prevailing belief within the machine learning community that decision trees, like other machine learning models, display bias towards the majority class. This research challenges this assumption by providing both theoretical and empirical evidence that decision trees can, under certain conditions, exhibit bias toward the minority class.

Key Findings

A principal contribution of this paper is an exploration into the theoretical underpinnings of bias in decision trees. The paper particularly focuses on decision trees trained to purity—a common configuration in random forests—demonstrating that their splits display an inclination towards the minority class when assessing imbalanced data with specific conditions on predictors.

Noteworthy is Theorem 1, which establishes an unbiased expectation for decision trees when only a single predictor is involved, provided the data generation process and distribution assumptions hold. However, Theorem 2 extends this inquiry to datasets with multiple predictors, revealing a positive bias in prevalence estimates of the minority class when the decision tree algorithm selects a single predictor for splits. The paper also posits that this bias intensifies with an increase in predictor count.

The simulation studies further reinforce these theoretical insights, showcasing scenarios under which decision trees overpredict their positive class prevalence compared to the actual distribution of classes. Particularly, conditions where the algorithm does not make optimal splits fail to rectify this bias. This occurs despite implementing various scenarios—altering distribution assumptions for predictors from uniform to normal and lognormal—indicating the robustness but variation in the magnitude of the observed phenomena.

Implications and Future Directions

This paper has significant implications for the application of decision trees and tree-based ensemble models like random forests in domains with imbalanced data, such as medical diagnosis and risk prediction in natural disaster scenarios. The theoretical and empirical results suggest that existing practices around data and model adjustments for dealing with class imbalance may require reevaluation specifically for tree-based models. These findings encourage the consideration of alternative techniques and a deeper understanding of the data generation process underlying the model training.

On a theoretical level, the research raises critical questions about structural biases in other machine learning models, urging further investigation into predictive behavior under varying conditions of data imbalance. There remains the need for more comprehensive studies to delineate the extent to which these findings generalize to larger datasets and the role of hyperparameter tuning in mediating bias.

The authors propose using constraints such as single predictor splits when dealing with data regions near purity to minimize bias. However, future research might explore more sophisticated modifications that fully eliminate bias while retaining predictive performance.

Conclusion

In summary, this paper provides a critical reevaluation of decision trees' performance with imbalanced datasets, coupling rigorous theoretical analysis with comprehensive simulations. It underscores the importance of exploring the nuanced behaviors of tree-based models, contributing a pivotal perspective to ongoing discussions in the machine learning community regarding class imbalance. This work paves the way for methodological innovations to advance the reliability and efficacy of predictive analytics in various high-stakes domains.