Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Is Hyper-Parameter Optimization Different for Software Analytics? (2401.09622v3)

Published 17 Jan 2024 in cs.SE and cs.LG

Abstract: Yes. SE data can have "smoother" boundaries between classes (compared to traditional AI data sets). To be more precise, the magnitude of the second derivative of the loss function found in SE data is typically much smaller. A new hyper-parameter optimizer, called SMOOTHIE, can exploit this idiosyncrasy of SE data. We compare SMOOTHIE and a state-of-the-art AI hyper-parameter optimizer on three tasks: (a) GitHub issue lifetime prediction (b) detecting static code warnings false alarm; (c) defect prediction. For completeness, we also show experiments on some standard AI datasets. SMOOTHIE runs faster and predicts better on the SE data--but ties on non-SE data with the AI tool. Hence we conclude that SE data can be different to other kinds of data; and those differences mean that we should use different kinds of algorithms for our data. To support open science and other researchers working in this area, all our scripts and datasets are available on-line at https://github.com/yrahul3910/smoothness-hpo/.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. What is wrong with topic modeling? and how to fix it using search-based software engineering. Information and Software Technology, 98:74–88, 2018. ISSN 0950-5849. doi: https://doi.org/10.1016/j.infsof.2018.02.005. URL https://www.sciencedirect.com/science/article/pii/S0950584917300861.
  2. How to “dodge” complex software analytics. IEEE Transactions on Software Engineering, 47(10):2182–2194, 2019.
  3. Better software analytics via “duo”: Data mining algorithms using/used-by optimizers. Empirical Software Engineering, 25:2099–2136, 2020.
  4. Instance-based learning algorithms. Machine Learning, 6:37–66, 01 1991. doi: 10.1023/A:1022689900470.
  5. Opentuner: An extensible framework for program autotuning. In International Conference on Parallel Architectures and Compilation Techniques (PACT), Edmonton, Canada, Aug 2014. URL http://groups.csail.mit.edu/commit/papers/2014/ansel-pact14-opentuner.pdf.
  6. Jon Louis Bentley. Multidimensional binary search trees used for associative searching. Commun. ACM, 18(9):509–517, sep 1975. ISSN 0001-0782. doi: 10.1145/361002.361007. URL https://doi.org/10.1145/361002.361007.
  7. Random search for hyper-parameter optimization. Journal of machine learning research, 13(2), 2012.
  8. Algorithms for hyper-parameter optimization. Advances in neural information processing systems, 24, 2011.
  9. Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 13(2):e1484, 2023.
  10. G Boetticher. The promise repository of empirical software engineering data. http://promisedata. org/repository, 2007.
  11. Convex optimization. Cambridge university press, 2004.
  12. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321–357, 2002.
  13. A metrics suite for object oriented design. IEEE Transactions on software engineering, 20(6):476–493, 1994.
  14. Hebo: pushing the limits of sample-efficient hyper-parameter optimisation. Journal of Artificial Intelligence Research, 74:1269–1349, 2022.
  15. Hpobench: A collection of reproducible multi-fidelity benchmark problems for hpo. arXiv preprint arXiv:2109.06716, 2021.
  16. Scalable global optimization via local bayesian optimization. Advances in neural information processing systems, 32, 2019.
  17. Bohb: Robust and efficient hyperparameter optimization at scale. In International Conference on Machine Learning, pages 1437–1446. PMLR, 2018.
  18. Alessio Farcomeni. A review of modern multiple hypothesis testing, with particular attention to the false discovery proportion. Statistical methods in medical research, 17(4):347–388, 2008.
  19. Hyperparameter optimization. In Automated machine learning, pages 3–33. Springer, Cham, 2019.
  20. Tuning for software analytics: Is it really necessary? Information and Software Technology, 76:135–146, 2016a. ISSN 0950-5849. doi: https://doi.org/10.1016/j.infsof.2016.04.017. URL https://www.sciencedirect.com/science/article/pii/S0950584916300738.
  21. Tuning for software analytics: Is it really necessary? Information and Software Technology, 76:135–146, 2016b.
  22. Peer instruction in online software testing and continuous integration: A replication study. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Software Engineering Education and Training, pages 199–204, 2022.
  23. Greg. Computing the smoothness of gda with respect to covariance matrix. Mathematics Stack Exchange. URL https://math.stackexchange.com/q/4647791. URL:https://math.stackexchange.com/q/4647791 (version: 2023-02-27).
  24. On establishing a benchmark for evaluating static analysis alert prioritization and classification techniques. In Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement, pages 41–50, 2008.
  25. A model building process for identifying actionable static analysis alerts. In 2009 International conference on software testing verification and validation, pages 161–170. IEEE, 2009.
  26. Predictive entropy search for efficient global optimization of black-box functions. Advances in neural information processing systems, 27, 2014.
  27. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13(4):455, 1998.
  28. Detecting false alarms from automatic static analysis tools: How far are we? arXiv preprint arXiv:2202.05982, 2022.
  29. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  30. The "bigse" project: Lessons learned from validating industrial text mining. In Proceedings of the 2nd International Workshop on BIG Data Software Engineering, BIGDSE ’16, page 65–71, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450341523. doi: 10.1145/2896825.2896836. URL https://doi.org/10.1145/2896825.2896836.
  31. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics, 6(1):4–22, 1985.
  32. Continual prediction of bug-fix time using deep learning-based activity stream embedding. IEEE Access, 8:10503–10515, 2020.
  33. Hyperband: A novel bandit-based approach to hyperparameter optimization. The Journal of Machine Learning Research, 18(1):6765–6816, 2017.
  34. Smac3: A versatile bayesian optimization package for hyperparameter optimization. The Journal of Machine Learning Research, 23(1):2475–2483, 2022.
  35. Deeptriage: Exploring the effectiveness of deep learning for bug triaging. In Proceedings of the ACM India joint international conference on data science and management of data, pages 171–179, 2019.
  36. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics, pages 50–60, 1947.
  37. Thomas J McCabe. A complexity measure. IEEE Transactions on software Engineering, (4):308–320, 1976.
  38. Jonas Močkus. On bayesian methods for seeking the extremum. In Optimization Techniques IFIP Technical Conference: Novosibirsk, July 1–7, 1974, pages 400–404. Springer, 1975.
  39. Using bad learners to find good configurations. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, page 257–267, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450351058. doi: 10.1145/3106237.3106238. URL https://doi.org/10.1145/3106237.3106238.
  40. An empirical study of text-based machine learning models for vulnerability detection. Empirical Software Engineering, 28(2):38, 2023.
  41. Yahpo gym-an efficient multi-objective multi-fidelity benchmark for hyperparameter optimization. In International Conference on Automated Machine Learning, pages 3–1. PMLR, 2022.
  42. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1):148–175, 2015.
  43. Automated parameter optimization of classification techniques for defect prediction models. In 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE), pages 321–332, 2016. doi: 10.1145/2884781.2884857.
  44. Perceptions, expectations, and challenges in defect prediction. IEEE Transactions on Software Engineering, 46(11):1241–1266, 2018.
  45. Is there a" golden" feature set for static warning identification? an experimental evaluation. In Proceedings of the 12th ACM/IEEE international symposium on empirical software engineering and measurement, pages 1–10, 2018.
  46. Automatically learning semantic features for defect prediction. In Proceedings of the 38th International Conference on Software Engineering, pages 297–308, 2016.
  47. Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA, 2006.
  48. Claes Wohlin. Guidelines for snowballing in systematic literature studies and a replication in software engineering. In Proceedings of the 18th international conference on evaluation and assessment in software engineering, pages 1–10, 2014.
  49. Hey, you have given me too many knobs!: Understanding and dealing with over-designed configuration in system software. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, page 307–319, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450336758. doi: 10.1145/2786805.2786852. URL https://doi.org/10.1145/2786805.2786852.
  50. Learning to recognize actionable static code warnings (is intrinsically easy). Empirical Software Engineering, 26:1–24, 2021.
  51. On the value of oversampling for deep learning in software defect prediction. IEEE Transactions on Software Engineering, 48(8):3103–3116, 2021.
  52. Old but gold: Reconsidering the value of feedforward learners for software analytics. arXiv preprint arXiv:2101.06319, 2021.
  53. How to find actionable static analysis warnings: A case study with findbugs. IEEE Transactions on Software Engineering, 2023.
  54. Hyper-parameter optimization: A review of algorithms and applications. arXiv preprint arXiv:2003.05689, 2020.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now