2000 character limit reached
Dynamic Test Case Prioritization in Industrial Test Result Datasets (2402.02925v1)
Published 5 Feb 2024 in cs.SE
Abstract: Regression testing in software development checks if new software features affect existing ones. Regression testing is a key task in continuous development and integration, where software is built in small increments and new features are integrated as soon as possible. It is therefore important that developers are notified about possible faults quickly. In this article, we propose a test case prioritization schema that combines the use of a static and a dynamic prioritization algorithm. The dynamic prioritization algorithm rearranges the order of execution of tests on the fly, while the tests are being executed. We propose to use a conditional probability dynamic algorithm for this. We evaluate our solution on three industrial datasets and utilize Average Percentage of Fault Detection for that. The main findings are that our dynamic prioritization algorithm can: a) be applied with any static algorithm that assigns a priority score to each test case b) can improve the performance of the static algorithm if there are failure correlations between test cases c) can also reduce the performance of the static algorithm, but only when the static scheduling is performed at a near optimal level.
- History-based test case prioritization for failure information. In Alex Potanin, Gail C. Murphy, Steve Reeves, and Jens Dietrich, editors, 23rd Asia-Pacific Software Engineering Conference, APSEC 2016, Hamilton, New Zealand, December 6-9, 2016, pages 385–388. IEEE Computer Society, 2016.
- Failure history data-based test case prioritization for effective regression test. In Proceedings of the Symposium on Applied Computing, SAC ’17, page 1409–1415, New York, NY, USA, 2017. Association for Computing Machinery.
- Search algorithms for regression test case prioritization. IEEE Trans. Software Eng., 33(4):225–237, 2007.
- Jackson A. Prado Lima and Silvia R. Vergilio. Multi-armed bandit test case prioritization in continuous integration environments: A trade-off analysis. In Everton Cavalcante, Francisco Dantas, and Thaís Batista, editors, SAST 20: 5th Brazilian Symposium on Systematic and Automated Software Testing, Natal, Brazil, October 19-23, 2020, pages 21–30. ACM, 2020.
- Dynamic test prioritization of product lines: An application on configurable simulation models. Softw. Qual. J., 29(4):943–988, 2021.
- A prioritization approach for software test cases based on bayesian networks. In Matthew B. Dwyer and Antónia Lopes, editors, Fundamental Approaches to Software Engineering, 10th International Conference, FASE 2007, Held as Part of the Joint European Conferences, on Theory and Practice of Software, ETAPS 2007, Braga, Portugal, March 24 - April 1, 2007, Proceedings, volume 4422 of Lecture Notes in Computer Science, pages 276–290. Springer, 2007.
- An empirical study on bayesian network-based approach for test case prioritization. In First International Conference on Software Testing, Verification, and Validation, ICST 2008, Lillehammer, Norway, April 9-11, 2008, pages 278–287. IEEE Computer Society, 2008.
- Analytic hierarchy process-based regression test case prioritization technique enhancing the fault detection rate. Soft Comput., 26(15):6953–6968, 2022.
- Test case selection and prioritization using machine learning: a systematic literature review. Empir. Softw. Eng., 27(2):29, 2022.
- REMAP: using rule mining and multi-objective search for dynamic test case prioritization. In 11th IEEE International Conference on Software Testing, Verification and Validation, ICST 2018, Västerås, Sweden, April 9-13, 2018, pages 46–57. IEEE Computer Society, 2018.
- Employing rule mining and multi-objective search for dynamic test case prioritization. J. Syst. Softw., 153:86–104, 2019.
- Test case prioritization for black box testing. In 31st Annual International Computer Software and Applications Conference (COMPSAC 2007), volume 1, pages 465–474, 2007.
- Reinforcement learning based test case prioritization for enhancing the security of software. In Geoffrey I. Webb, Zhongfei Zhang, Vincent S. Tseng, Graham Williams, Michalis Vlachos, and Longbing Cao, editors, 7th IEEE International Conference on Data Science and Advanced Analytics, DSAA 2020, Sydney, Australia, October 6-9, 2020, pages 663–672. IEEE, 2020.
- Reinforcement learning for automatic test case prioritization and selection in continuous integration. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2017, page 12–22, New York, NY, USA, 2017. Association for Computing Machinery.
- Per Erik Strandberg. The Westermo test results data set. Technical report, GitHub, https://github.com/westermo/test-results-dataset. https://doi.org/10.48550/arXiv.2211.13622, 2022.
- Simple techniques work surprisingly well for neural network test prioritization and active learning (replicability study). In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2022, page 139–150, New York, NY, USA, 2022. Association for Computing Machinery.
- S. Yoo and M. Harman. Regression testing minimization, selection and prioritization: a survey. Software Testing, Verification and Reliability, 22(2):67–120, 2012.
- TERMINATOR: better automated UI test case prioritization. In Marlon Dumas, Dietmar Pfahl, Sven Apel, and Alessandra Russo, editors, Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2019, Tallinn, Estonia, August 26-30, 2019, pages 883–894. ACM, 2019.
- Test case prioritization based on simulation annealing algorithm. In ICCAI ’22: 8th International Conference on Computing and Artificial Intelligence, Tianjin, China, March 18 - 21, 2022, pages 235–240. ACM, 2022.
- Test re-prioritization in continuous testing environments. In 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 69–79, 2018.