The Impact of Class Rebalancing Techniques on the Performance and Interpretation of Defect Prediction Models (1801.10269v1)

Abstract: Defect prediction models that are trained on class imbalanced datasets (i.e., the proportion of defective and clean modules is not equally represented) are highly susceptible to produce inaccurate prediction models. Prior research compares the impact of class rebalancing techniques on the performance of defect prediction models. Prior research efforts arrive at contradictory conclusions due to the use of different choice of datasets, classification techniques, and performance measures. Such contradictory conclusions make it hard to derive practical guidelines for whether class rebalancing techniques should be applied in the context of defect prediction models. In this paper, we investigate the impact of 4 popularly-used class rebalancing techniques on 10 commonly-used performance measures and the interpretation of defect prediction models. We also construct statistical models to better understand in which experimental design settings that class rebalancing techniques are beneficial for defect prediction models. Through a case study of 101 datasets that span across proprietary and open-source systems, we recommend that class rebalancing techniques are necessary when quality assurance teams wish to increase the completeness of identifying software defects (i.e., Recall). However, class rebalancing techniques should be avoided when interpreting defect prediction models. We also find that class rebalancing techniques do not impact the AUC measure. Hence, AUC should be used as a standard measure when comparing defect prediction models.

• The paper demonstrates that class rebalancing techniques can significantly improve defect prediction model performance while influencing feature interpretation.
• The research employs oversampling, undersampling, and hybrid methods, assessing metrics like precision, recall, F1-score, and AUC across various datasets.
• The findings urge a careful balance between accuracy improvements and interpretability biases, guiding context-specific selections in real-world applications.

The Impact of Class Rebalancing Techniques on the Performance and Interpretation of Defect Prediction Models

The paper "The Impact of Class Rebalancing Techniques on the Performance and Interpretation of Defect Prediction Models," by Chakkrit Tantithamthavorn, Ahmed E. Hassan, and Kenichi Matsumoto, provides a systematic investigation into the role of class rebalancing techniques within the domain of defect prediction models in software engineering. This work addresses a notable challenge in software defect prediction: the inherent class imbalance, wherein defective instances are substantially outnumbered by non-defective ones.

This paper evaluates the efficacy of various class rebalancing techniques, namely oversampling, undersampling, and hybrid methods, across multiple defect prediction models and datasets. The performance of these techniques is quantitatively assessed through established metrics such as precision, recall, F1-score, and area under the receiver operating characteristic (ROC) curve. The authors examined how different rebalancing strategies affect model performance and the interpretability of the prediction outcomes.

Significantly, the paper presents robust evidence that class imbalance critically impacts both the performance and interpretability of defect prediction models. A substantial observation is that class rebalancing can not only enhance model performance with respect to prediction accuracy but also affect the interpretation of the models' results, altering the perceived importance of model features. Moreover, the research explores the nuanced trade-offs between different rebalancing approaches, highlighting that no singular technique universally optimizes performance across all metrics and contexts.

Further, the authors expand upon the theoretical implications of their findings. They argue that while class rebalancing can lead to improved model efficacy, it may also introduce biases that impair the operationalization of defect prediction models in real-world scenarios. Hence, this work implies a need for a careful balance between model improvement and the fidelity of model interpretation, urging researchers and practitioners to consider context-specific conditions when selecting rebalancing techniques.

Anticipating future advancements, this paper lays a foundation for subsequent research in refining defect prediction models. It suggests integrating adaptive and context-aware class rebalancing methods that dynamically adjust to evolving data distributions. This progression could potentially lead to a paradigm wherein defect prediction models are both robust in predictive accuracy and transparent in decision-making processes.

In conclusion, "The Impact of Class Rebalancing Techniques on the Performance and Interpretation of Defect Prediction Models" provides a comprehensive analysis by rigorously quantifying the influence of class rebalancing techniques and offering insights that extend beyond mere technical enhancements. Such work enables a deeper understanding of defect prediction mechanisms, thereby fostering advancements in software quality assurance practices.