Negative Results for Software Effort Estimation (1609.05563v3)

Abstract: Context:More than half the literature on software effort estimation (SEE) focuses on comparisons of new estimation methods. Surprisingly, there are no studies comparing state of the art latest methods with decades-old approaches. Objective:To check if new SEE methods generated better estimates than older methods. Method: Firstly, collect effort estimation methods ranging from "classical" COCOMO (parametric estimation over a pre-determined set of attributes) to "modern" (reasoning via analogy using spectral-based clustering plus instance and feature selection, and a recent "baseline method" proposed in ACM Transactions on Software Engineering).Secondly, catalog the list of objections that lead to the development of post-COCOMO estimation methods.Thirdly, characterize each of those objections as a comparison between newer and older estimation methods.Fourthly, using four COCOMO-style data sets (from 1991, 2000, 2005, 2010) and run those comparisons experiments.Fifthly, compare the performance of the different estimators using a Scott-Knott procedure using (i) the A12 effect size to rule out "small" differences and (ii) a 99% confident bootstrap procedure to check for statistically different groupings of treatments). Results: The major negative results of this paper are that for the COCOMO data sets, nothing we studied did any better than Boehm's original procedure. Conclusions: When COCOMO-style attributes are available, we strongly recommend (i) using that data and (ii) use COCOMO to generate predictions. We say this since the experiments of this paper show that, at least for effort estimation,how data is collected is more important than what learner is applied to that data.