Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Assessing test artifact quality -- A tertiary study (2402.09541v1)

Published 14 Feb 2024 in cs.SE

Abstract: Context: Modern software development increasingly relies on software testing for an ever more frequent delivery of high quality software. This puts high demands on the quality of the central artifacts in software testing, test suites and test cases. Objective: We aim to develop a comprehensive model for capturing the dimensions of test case/suite quality, which are relevant for a variety of perspectives. Method: We have carried out a systematic literature review to identify and analyze existing secondary studies on quality aspects of software testing artifacts. Results: We identified 49 relevant secondary studies. Of these 49 studies, less than half did some form of quality appraisal of the included primary studies and only 3 took into account the quality of the primary study when synthesizing the results. We present an aggregation of the context dimensions and factors that can be used to characterize the environment in which the test case/suite quality is investigated. We also provide a comprehensive model of test case/suite quality with definitions for the quality attributes and measurements based on findings in the literature and ISO/IEC 25010:2011. Conclusion: The test artifact quality model presented in the paper can be used to support test artifact quality assessment and improvement initiatives in practice. Furtherm Information and Software Technology 139 (2021): 106620ore, the model can also be used as a framework for documenting context characteristics to make research results more accessible for research and practice.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (118)
  1. A systematic review of search-based testing for non-functional system properties. Information and Software Technology 51, 957–976. doi:10.1016/j.infsof.2008.12.005.
  2. A systematic literature review of the test case prioritization technique for sequence of events. International Journal of Applied Engineering Research 12, 1389–1395.
  3. Factor determination in prioritizing test cases for event sequences: A systematic literature review. Journal of Telecommunication, Electronic and Computer Engineering 10, 119–124.
  4. Model-based testing using UML activity diagrams: A systematic mapping study. Computer Science Review 33, 98–112. doi:10.1016/j.cosrev.2019.07.001.
  5. On the search for industry-relevant regression testing research. Empirical Software Engineering 24, 2020–2055. doi:10.1007/s10664-018-9670-1.
  6. A critical appraisal tool for systematic literature reviews in software engineering. Inf. Softw. Technol. 112, 48–50.
  7. A systematic review of the application and empirical investigation of search-based test case generation. IEEE Transactions on Software Engineering 36, 742–762. doi:10.1109/TSE.2009.52.
  8. A systematic review of concolic testing with aplication of test criteria, in: Proceedings of the 20th International Conference on Enterprise Information Systems, ICEIS 2018, Funchal, Madeira, Portugal, March 21-24, 2018, Volume 2, pp. 121–132.
  9. Semi-automatic generation of a software testing lightweight ontology from a glossary based on the ONTO6 methodology. Frontiers in Artificial Intelligence and Applications 249, 263–276. doi:10.3233/978-1-61499-161-8-263.
  10. Towards classification of concurrency bugs based on observable properties, in: Proceedings of IEEE/ACM 1st International Workshop on Complex Faults and Failures in Large Software Systems (COUFLESS), pp. 41–47.
  11. Test code quality and its relation to issue handling performance. IEEE Transactions on Software Engineering 40, 1100–1125.
  12. A Systematic Literature Review of Test Case Prioritization Using Genetic Algorithms. IEEE Access 7, 126355–126375. doi:10.1109/ACCESS.2019.2938260.
  13. A systematic mapping addressing Hyper-Heuristics within Search-based Software Testing. Information and Software Technology 114, 176–189. doi:10.1016/j.infsof.2019.06.012.
  14. Ontology-based development of testing related tools, in: Proceedings of the Twentieth International Conference on Software Engineering & Knowledge Engineering (SEKE), pp. 697–702.
  15. Quality Factors of Test Cases: A Systematic Literature Review. Univ Utara Malaysia Press, Sintok.
  16. Binary mutation testing through dynamic translation, in: Proceedings of IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2012), pp. 1–12.
  17. When, how, and why developers (do not) test in their ides, in: Proceedings of the 10th Joint Meeting on Foundations of Software Engineering, pp. 179–190.
  18. Guide to the Software Engineering Body of Knowledge (SWEBOK(R)): Version 3.0. 3rd ed., IEEE Computer Society Press, Washington, DC, USA.
  19. How Good Are My Tests?, in: Proceedings of IEEE/ACM 8th Workshop on Emerging Trends in Software Metrics (WETSoM), pp. 9–14. doi:10.1109/WETSoM.2017.2.
  20. Test case reuse based on ontology, in: Proceedings of the 15th IEEE Pacific Rim International Symposium on Dependable Computing, PRDC, pp. 103–108. doi:10.1109/PRDC.2009.25.
  21. Test case prioritization: A systematic mapping study. Software Quality Journal 21, 445–478. doi:10.1007/s11219-012-9181-z.
  22. A review of software testing. Information and Software Technology 30, 189–198.
  23. A systematic mapping study of software product lines testing. Information and Software Technology 53, 407–423. doi:10.1016/j.infsof.2010.12.003.
  24. Regression tests provenance data in the continuous software engineering context, in: Proceedings of the 2nd Brazilian Symposium on Systematic and Automated Software Testing, SAST, pp. 10:1–10:6. doi:10.1145/3128473.3128483.
  25. Test case prioritization: A systematic review and mapping of the literature, in: Proceedings of the 31st Brazilian Symposium on Software Engineering, SBES, pp. 34–43. doi:10.1145/3131151.3131170.
  26. A taxonomy and catalog of runtime software-fault monitoring tools. IEEE Transactions on software Engineering 30, 859–872.
  27. Systematic reviews: CRD’s guidance for undertaking reviews in healthcare. York: University of York NHS Centre for Reviews & Dissemination .
  28. Web application testing: A systematic literature review. Journal of Systems and Software 91, 174–201. doi:10.1016/j.jss.2014.01.010.
  29. Machine learning applied to software testing: A systematic mapping study. IEEE Transactions on Reliability 68, 1189–1212. doi:10.1109/TR.2019.2892517.
  30. Understanding flaky tests: the developer’s perspective, in: Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering - ESEC/FSE 2019, ACM Press, Tallinn, Estonia. pp. 830–840. URL: http://dl.acm.org/citation.cfm?doid=3338906.3338945, doi:10.1145/3338906.3338945.
  31. SERP-test: a taxonomy for supporting industry–academia communication. Software Quality Journal 25, 1269–1305. doi:10.1007/s11219-016-9322-x.
  32. Empirical evaluations of regression test selection techniques: A systematic review, in: Proceedings of the 2nd International Symposium on Empirical Software Engineering and Measurement, ESEM, pp. 22–31. doi:10.1145/1414004.1414011.
  33. Model-based security testing: a taxonomy and systematic classification. Software Testing, Verification and Reliability 26, 119–148.
  34. Continuous software engineering: A roadmap and agenda. Journal of Systems and Software 123, 176–189.
  35. Vertical Test Reuse for Embedded Systems: A Systematic Mapping Study, in: Proceedings of the 41st Euromicro Conference on Software Engineering and Advanced Applications, EUROMICRO-SEAA, pp. 317–324. doi:10.1109/SEAA.2015.46.
  36. Refactoring: improving the design of existing code. Addison-Wesley Professional.
  37. An ontology for guiding performance testing, in: Proceedings of IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT), Volume II, pp. 400–407. doi:10.1109/WI-IAT.2014.62.
  38. Smells in software test code: A survey of knowledge in industry and academia. Journal of systems and software 138, 52–81.
  39. Smells in software test code: A survey of knowledge in industry and academia. Journal of Systems and Software 138, 52–81. doi:10.1016/j.jss.2017.12.013.
  40. A systematic literature review of literature reviews in software testing. Information and Software Technology 80, 195–216. doi:10.1016/j.infsof.2016.09.002.
  41. When and what to automate in software testing? A multi-vocal literature review. Information and Software Technology 76, 92–117. doi:10.1016/j.infsof.2016.04.015.
  42. Software test-code engineering: A systematic mapping. Information and Software Technology 58, 123–147. doi:10.1016/j.infsof.2014.06.009.
  43. Toward a theory of test data selection. IEEE Transactions on software Engineering , 156–173.
  44. Pizza versus pinsa: On the perception and measurability of unit test code quality, in: Proceedings of the 36th IEEE International Conference on Software Maintenance and Evolution, IEEE. pp. 336–347.
  45. Standard Glossary of Terms Used in Software Testing. Technical Report. Technical report, International Software Testing Qualifications Board ….
  46. Model-based testing for software safety: a systematic mapping study. Software Quality Journal 26, 1327–1372. doi:10.1007/s11219-017-9386-2.
  47. Deriving a unified fault taxonomy for event-based systems, in: Proceedings of the 6th ACM International Conference on Distributed Event-based Systems, pp. 167–178.
  48. A systematic literature review of test breakage prevention and repair techniques. Information and Software Technology 113, 1–19. doi:10.1016/j.infsof.2019.05.001.
  49. ISO/IEC JTC 1/SC 7 Software and systems engineering, 2011. ISO/IEC 25010:2011 Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – System and software quality models. URL: https://www.iso.org/standard/35733.html.
  50. Maintaining software with a security perspective, in: Proceedings of the International Conference on Software Maintenance, pp. 194–203.
  51. A family of test adequacy criteria for database-driven applications. ACM SIGSOFT Software Engineering Notes 28, 98–107.
  52. Analyzing an automotive testing process with evidence-based software engineering. Information and Software Technology 55, 1237–1259. doi:10.1016/j.infsof.2013.01.005.
  53. Effective regression test case selection: A systematic literature review. ACM Computing Surveys 50. doi:10.1145/3057269.
  54. Test-Case Quality – Understanding Practitioners’ Perspectives, in: Proceedings of the 20th International Conference on Product-Focused Software Process Improvement (PROFES), Springer, Barcelona, Spain. pp. 37–52.
  55. Test case prioritization approaches in regression testing: A systematic literature review. Information and Software Technology 93, 74–93. doi:10.1016/j.infsof.2017.08.014.
  56. A Comprehensive Investigation of Modern Test Suite Optimization Trends, Tools and Techniques. IEEE Access 7, 89093–89117. doi:10.1109/ACCESS.2019.2926384.
  57. What’s up with software metrics?–a preliminary mapping study. Journal of systems and software 83, 37–51.
  58. A systematic review of systematic review process research in software engineering. Information and Software Technology 55, 2049–2075.
  59. Systematic literature reviews in software engineering-A tertiary study. Information and Software Technology 52, 792–805. doi:10.1016/j.infsof.2010.03.006.
  60. Evidence-based software engineering and systematic reviews. volume 4. CRC Press.
  61. Evidence-based software engineering, in: Proceedings of the 26th International Conference on Software Engineering, pp. 273–281.
  62. The measurement of observer agreement for categorical data. Biometrics 33, 159–174. URL: http://www.jstor.org/stable/2529310.
  63. Problems, causes and solutions when adopting continuous delivery—A systematic literature review. Information and Software Technology 82, 55–79. doi:10.1016/j.infsof.2016.10.001.
  64. Control and data flow structural testing criteria for aspect-oriented programs. Journal of Systems and Software 80, 862–882.
  65. Deepgauge: Multi-granularity testing criteria for deep learning systems, in: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, pp. 120–131.
  66. A fault taxonomy for component-based software. Electron. Notes Theor. Comput. Sci. 82, 55–65.
  67. Continuous testing and solutions for testing problems in continuous delivery: A systematic literature review. Computacion y Sistemas 22, 1009–1038. doi:10.13053/CyS-22-3-2794.
  68. xUnit Test Patterns: Refactoring Test Code. 1 ed., Addison-Wesley Professional.
  69. On rapid releases and software testing: a case study and a semi-systematic literature review. Empirical Software Engineering 20, 1384–1425. doi:10.1007/s10664-014-9338-4.
  70. Specifications for Web Services Testing: A Systematic Review, in: Proceedings of IEEE World Congress on Services, SERVICES, pp. 152–159. doi:10.1109/SERVICES.2015.31.
  71. Test case selection: A systematic literature review. International Journal of Software Engineering and Knowledge Engineering 24, 653–676. doi:10.1142/S0218194014500259.
  72. An approach to quality engineering of TTCN-3 test specifications. International Journal on Software Tools for Technology Transfer 10, 309. URL: https://link-springer-com.miman.bib.bth.se/article/10.1007/s10009-008-0075-0, doi:10.1007/s10009-008-0075-0.
  73. A Systematic Mapping Study on Test Generation from Input/Output Transition Systems, in: Proceedings of the 41st Euromicro Conference on Software Engineering and Advanced Applications, EUROMICRO-SEAA, pp. 333–340. doi:10.1109/SEAA.2015.66.
  74. A systematic literature review on modified condition and decision coverage, in: Proceedings of the Symposium on Applied Computing, SAC, pp. 1301–1308. doi:10.1145/2554850.2555004.
  75. Deepxplore: Automated whitebox testing of deep learning systems, in: Proceedings of the 26th ACM Symposium on Operating Systems Principles - SOSP, pp. 1–18.
  76. What is the Vocabulary of Flaky Tests?, in: Proceedings of the 17th International Conference on Mining Software Repositories, ACM, Seoul Republic of Korea. pp. 492–502. URL: https://dl.acm.org/doi/10.1145/3379597.3387482, doi:10.1145/3379597.3387482.
  77. A systematic literature review of techniques and metrics to reduce the cost of mutation testing. Journal of Systems and Software 157. doi:10.1016/j.jss.2019.07.100.
  78. A systematic mapping study on higher order mutation testing. Journal of Systems and Software 154, 92–109. doi:10.1016/j.jss.2019.04.031.
  79. Regression testing of web service: A systematic mapping study. ACM Computing Surveys 47. doi:10.1145/2631685.
  80. A Systematic Review on Test Suite Reduction: Approaches, Experiment’s Quality Evaluation, and Guidelines. IEEE Access 6, 11816–11841. doi:10.1109/ACCESS.2018.2809600.
  81. Monitoring user interactions for supporting failure reproduction, in: Proceedings of the 21st International Conference on Program Comprehension (ICPC), pp. 73–82.
  82. Framework for evaluating regression test selection techniques, in: Proceedings of the 16th International Conference on Software Engineering, pp. 201–210.
  83. The experimental applications of search-based techniques for model-based testing: Taxonomy and systematic literature review. Applied Soft Computing Journal 49, 1094–1117. doi:10.1016/j.asoc.2016.08.030.
  84. Combinatorial interaction testing of software product lines: A mapping study. Journal of Computer Science 12, 379–398. doi:10.3844/jcssp.2016.379.398.
  85. Test case design for context-aware applications: Are we there yet? Information and Software Technology 88, 1–16. doi:10.1016/j.infsof.2017.03.008.
  86. Testing techniques selection: A systematic mapping study, in: Proceedings of the XXXIII Brazilian Symposium on Software Engineering, SBES, pp. 347–356. doi:10.1145/3350768.3352571.
  87. An ontology based approach for test scenario management. Communications in Computer and Information Science 141 CCIS, 91–100. doi:10.1007/978-3-642-19423-8_10.
  88. The approaches to quantify web application security scanners quality: A review. International Journal of Advanced Computer Research 8, 285–312. doi:10.19101/IJACR.2018.838012.
  89. A systematic review of state-based test tools. International Journal on Software Tools for Technology Transfer 17, 59–76. doi:10.1007/s10009-013-0291-0.
  90. Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices. IEEE Access 5, 3909–3943. doi:10.1109/ACCESS.2017.2685629.
  91. A systematic review on search based mutation testing. Information and Software Technology 81, 19–35. doi:10.1016/j.infsof.2016.01.017.
  92. Systematic literature review on regression test prioritization techniques. Informatica (Slovenia) 36, 379–408.
  93. Software testing with an operational profile: OP definition. ACM Computing Surveys 46. doi:10.1145/2518106.
  94. ROoST: Reference ontology on software testing. Applied Ontology 12, 59–90. doi:10.3233/AO-170177.
  95. On the relation of test smells to software code quality, in: Proceedings of IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 1–12.
  96. Towards a taxonomy for simulink model mutations, in: Proceedings of the 7th IEEE International Conference on Software Testing, Verification and Validation Workshops, pp. 206–215.
  97. A Systematic Review on Software Testing Ontologies. Communications in Computer and Information Science 1010, 144–160. doi:10.1007/978-3-030-29238-6_11.
  98. Research on software security testing. World Academy of science, engineering and Technology 70, 647–651.
  99. Evaluation and measurement of software process improvement—a systematic literature review. IEEE Transactions on Software Engineering 38, 398–424.
  100. Taxonomies in software engineering: A Systematic mapping study and a revised taxonomy development method. Information and Software Technology 85, 43–59. doi:10.1016/j.infsof.2017.01.006.
  101. A taxonomy of model-based testing approaches. Software testing, verification and reliability 22, 297–312.
  102. Model-driven architecture based testing: A systematic literature review. Information and Software Technology 102, 30–48. doi:10.1016/j.infsof.2018.05.004.
  103. An empirical study of bugs in test code, in: Proceedings of IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 101–110.
  104. Refactoring test code, in: Proceedings of the 2nd International Conference on Extreme Programming and Flexible Processes in Software Engineering (XP), pp. 92–95.
  105. On the detection of test smells: A metrics-based approach for general fixture and eager test. IEEE Transactions on Software Engineering 33, 800–817. doi:10.1109/TSE.2007.70745.
  106. An ontology-based knowledge framework for software testing. Communications in Computer and Information Science 780, 212–226. doi:10.1007/978-981-10-6989-5_18.
  107. An ontology-based knowledge sharing portal for software testing, in: Proceedings of IEEE International Conference on Software Quality, Reliability and Security Companion, QRS-C, pp. 472–479. doi:10.1109/QRS-C.2017.82.
  108. Maturing software engineering knowledge through classifications: A case study on unit testing techniques. IEEE Transactions on Software Engineering 35, 551–565. doi:10.1109/TSE.2009.13.
  109. A unified classification system for research in the computing disciplines. Information and Software Technology 47, 245–255.
  110. A tertiary study on model-based testing areas, tools and challenges: Preliminary results, in: Proceedings of the 21st Ibero-American Conference on Software Engineering, pp. 15–28.
  111. Literature Review of Empirical Research Studies within the Domain of Acceptance Testing, in: Proceedings of the 42nd Euromicro Conference on Software Engineering and Advanced Applications, SEAA, pp. 181–188. doi:10.1109/SEAA.2016.33.
  112. Mutation-based inter-class testing, in: Proceedings of the 5th Asia-Pacific Software Engineering Conference (APSEC), pp. 174–181. doi:10.1109/APSEC.1998.733717.
  113. Rethinking the taxonomy of fault detection techniques, in: Proceedings of the 11th International Conference on Software Engineering, pp. 53–62.
  114. Studying the co-evolution of production and test code in open source and industrial developer test processes through repository mining. Empirical Software Engineering 16, 325–364.
  115. Unit testing approaches for BPEL: A systematic review, in: Sulaiman, S., Noor, N.M.M. (Eds.), Proceedings of the 16th Asia-Pacific Software Engineering Conference (APSEC), pp. 316–322. doi:10.1109/APSEC.2009.72.
  116. Identifying relevant studies in software engineering. Information and Software Technology 53, 625–637. doi:10.1016/j.infsof.2010.12.010.
  117. Software unit test coverage and adequacy. ACM computing surveys 29, 366–427.
  118. Developing a Software Testing Ontology in UML for a Software Growth Environment of Web-Based Applications, in: Yang, H. (Ed.), Software Evolution with UML and XML. IGI Global, pp. 263–295. doi:10.4018/978-1-59140-462-0.ch009.
Citations (18)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets