To SMOTE, or not to SMOTE? (2201.08528v3)

Abstract: Balancing the data before training a classifier is a popular technique to address the challenges of imbalanced binary classification in tabular data. Balancing is commonly achieved by duplication of minority samples or by generation of synthetic minority samples. While it is well known that balancing affects each classifier differently, most prior empirical studies did not include strong state-of-the-art (SOTA) classifiers as baselines. In this work, we are interested in understanding whether balancing is beneficial, particularly in the context of SOTA classifiers. Thus, we conduct extensive experiments considering three SOTA classifiers along the weaker learners used in previous investigations. Additionally, we carefully discern proper metrics, consistent and non-consistent algorithms and hyper-parameter selection methods and show that these have a significant impact on prediction quality and on the effectiveness of balancing. Our results support the known utility of balancing for weak classifiers. However, we find that balancing does not improve prediction performance for the strong ones. We further identify several other scenarios for which balancing is effective and observe that prior studies demonstrated the utility of balancing by focusing on these settings.

• The paper finds that SMOTE benefits weak classifiers but does not improve performance for strong models like CatBoost.
• The study employs extensive experiments on 73 datasets using metrics such as AUC and F1 to evaluate data balancing techniques.
• The paper shows that threshold optimization offers a simpler, computationally efficient alternative to SMOTE for enhancing label metrics.

An Analysis of "To SMOTE, or not to SMOTE?"

The paper "To SMOTE, or not to SMOTE?" presents an insightful investigation into the efficacy of data balancing techniques, particularly focusing on SMOTE (Synthetic Minority Over-sampling Technique), within the context of imbalanced binary classification problems using tabular data. It challenges the often-standard practice of data balancing when employing both traditional and state-of-the-art (SOTA) classifiers.

Overview

The primary objective of the paper is to reconsider the utility of data balancing in improving classification performance, especially with SOTA classifiers such as LightGBM, XGBoost, and CatBoost. Traditional approaches to handling imbalanced datasets involve either duplicating minority samples or generating synthetic samples like those created with SMOTE. The paper further dissects the effectiveness of these techniques by contrasting them against a range of classifiers that vary from relatively weak models to highly potent ones.

Methodology

The paper's methodology is robust, utilizing extensive experimentation across 73 datasets, derived from a combination of well-known empirical repositories. The datasets span a significant variety of class imbalance scenarios. The authors tested a suite of supervised learning classifiers, including decision trees, SVMs, MLPs, Adaboost, and modern boosting methods. Importantly, the paper includes multiple hyper-parameter configurations to ensure comprehensive analysis. Metrics such as AUC, logloss, F1, and balanced accuracy were employed to evaluate classifier performance, with attention given to both a-priori and validation-derived hyper-parameter settings.

Key Findings

1. Impact on Weak vs. Strong Classifiers: The paper confirms that while SMOTE and its variants can enhance the predictive performance of weak classifiers, they do not yield similar benefits for strong classifiers. For SOTA classifiers, particularly CatBoost, balancing did not improve performance beyond what was achieved by simply optimizing the model itself.
2. Metrics Consideration: The effectiveness of data balancing is closely tied to the choice of evaluation metric. The paper underscores the importance of distinguishing between proper metrics (e.g., AUC) and label metrics (e.g., F1), demonstrating variance in impact when employing balancing techniques.
3. Threshold Optimization: The results suggest that for label metrics, optimizing the decision threshold can rival SMOTE-like techniques in improving classification performance, offering a simpler and more computationally efficient approach.
4. Theoretical Backing: The authors provide theoretical context supporting the empirical findings, affirming that strong classifiers inherently manage the skewness of data better due to their design, which is grounded in probabilistic prediction.

Implications

The paper's implications are twofold. Practically, it advises against routine application of SMOTE-like oversampling methods for strong classifiers, especially when computational resources are of concern or when proper metrics are employed. Theoretically, it raises questions about the necessity of synthetic sample generation in modern machine learning workflows, especially given the substantial tuning overhead and potential risk of overfitting.

Future Directions

This research invites further inquiry into exploring other phenomena contributing to difficulties in imbalanced classification beyond mere class distribution, possibly encompassing data noise and drift. Additionally, it presents a compelling case for re-evaluating custom algorithms designed for imbalanced data and their standing relative to contemporary general-purpose classifiers.

Conclusion

Overall, "To SMOTE, or not to SMOTE?" presents a nuanced, empirically-backed argument that refines the conventional wisdom surrounding class imbalance strategies in machine learning. The work serves as a critical resource for researchers and practitioners, prompting a reassessment of standard practices in the face of increasingly capable classification models.