Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Enriching Automatic Test Case Generation by Extracting Relevant Test Inputs from Bug Reports (2312.14898v1)

Published 22 Dec 2023 in cs.SE

Abstract: The quality of a software is highly dependent on the quality of the tests it is submitted to. Writing tests for bug detection is thus essential. However, it is time-consuming when done manually. Automating test cases generation has therefore been an exciting research area in the software engineering community. Most approaches have been focused on generating unit tests. Unfortunately, current efforts often do not lead to the generation of relevant inputs, which limits the efficiency of automatically generated tests. Towards improving the relevance of test inputs, we present \name, a technique for exploring bug reports to identify input values that can be fed to automatic test generation tools. In this work, we investigate the performance of using inputs extracted from bug reports with \name to generate test cases with Evosuite. The evaluation is performed on the Defects4J benchmark. For Defects4J projects, our study has shown that \name successfully extracted 68.68\% of relevant inputs when using regular expression in its approach versus 50.21\% relevant inputs without regular expression. Further, our study has shown the potential to improve the Line and Instruction Coverage across all projects. Overall, we successfully collected relevant inputs that led to the detection of 45 bugs that were previously undetected by the baseline.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. An industrial evaluation of unit test generation: Finding real faults in a financial application. In 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP), pages 263–272. IEEE, 2017.
  2. Parameter tuning or default values? an empirical investigation in search-based software engineering. Empirical Software Engineering, 18:594–623, 2013.
  3. A framework for automated testing of javascript web applications. In Proceedings of the 33rd International Conference on Software Engineering, pages 571–580, 2011.
  4. A survey of symbolic execution techniques. ACM Computing Surveys (CSUR), 51(3):1–39, 2018.
  5. Extracting structural information from bug reports. In Proceedings of the 2008 international working conference on Mining software repositories, pages 27–30, 2008.
  6. Automatically generating realistic test input from web services. In Proceedings of 2011 IEEE 6th International Symposium on Service Oriented System (SOSE), pages 13–24. IEEE, 2011.
  7. Exe: Automatically generating inputs of death. ACM Transactions on Information and System Security (TISSEC), 12(2):1–38, 2008.
  8. Symbolic execution for software testing: three decades later. Communications of the ACM, 56(2):82–90, 2013.
  9. Heike Wehrheim Cedric Richter. Tssb-3m: Mining single statement bugs at massive scale. In MSR, 2022.
  10. Improving web application testing with user session data. In 25th International Conference on Software Engineering, 2003. Proceedings., pages 49–59. IEEE, 2003.
  11. Evosuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering, pages 416–419, 2011.
  12. Evosuite at the sbst 2013 tool competition. pages 406–409, 03 2013.
  13. Does automated unit test generation really help software testers? a controlled empirical study. ACM Transactions on Software Engineering and Methodology (TOSEM), 24(4):1–49, 2015.
  14. Improving search-based test suite generation with dynamic symbolic execution. In 2013 ieee 24th international symposium on software reliability engineering (issre), pages 360–369. IEEE, 2013.
  15. Extending a search-based test generator with adaptive dynamic symbolic execution. In Proceedings of the 2014 international symposium on software testing and analysis, pages 421–424, 2014.
  16. Dart: Directed automated random testing. In Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation, pages 213–223, 2005.
  17. Perflearner: Learning from bug reports to understand and generate performance test frames. In Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, pages 17–28, 2018.
  18. A theoretical and empirical study of search-based testing: Local, global, and hybrid search. IEEE Transactions on Software Engineering, 36(2):226–247, 2010.
  19. Defects4j: A database of existing faults to enable controlled testing studies for java programs. In Proceedings of the 2014 international symposium on software testing and analysis, pages 437–440, 2014.
  20. James C King. Symbolic execution and program testing. Communications of the ACM, 19(7):385–394, 1976.
  21. Adoption of software testing in open source projects–a preliminary study on 50,000 projects. In 2013 17th european conference on software maintenance and reengineering, pages 353–356. IEEE, 2013.
  22. An empirical study of adoption of software testing in open source projects. In 2013 13th International Conference on Quality Software, pages 103–112. IEEE, 2013.
  23. Understanding the test automation culture of app developers. In 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pages 1–10. IEEE, 2015.
  24. JaCoCo Java Code Coverage Library. https://www.jacoco.org/jacoco/.
  25. Automatic text input generation for mobile testing. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), pages 643–653. IEEE, 2017.
  26. Directed test generation using symbolic grammars. In Proceedings of the 22nd IEEE/ACM International Conference on Automated Software Engineering, pages 134–143, 2007.
  27. Link: exploiting the web of data to generate test inputs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis, pages 373–384, 2014.
  28. Search-based test input generation for string data types using the results of web queries. In 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation, pages 141–150. IEEE, 2012.
  29. Leveraging existing tests in automated test generation for web applications. In Proceedings of the 29th ACM/IEEE international conference on Automated software engineering, pages 67–78, 2014.
  30. Feedback-directed random test generation. In 29th International Conference on Software Engineering (ICSE’07), pages 75–84. IEEE, 2007.
  31. Reformulating branch coverage as a many-objective optimization problem. In 2015 IEEE 8th international conference on software testing, verification and validation (ICST), pages 1–10. IEEE, 2015.
  32. Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets. IEEE Transactions on Software Engineering, 44(2):122–158, 2017.
  33. Defect prediction guided search-based software testing. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, pages 448–460, 2020.
  34. Fully automatic and precise detection of thread safety violations. In Proceedings of the 33rd ACM SIGPLAN conference on Programming Language Design and Implementation, pages 521–530, 2012.
  35. Configuring test generators using bug reports: a case study of gcc compiler and csmith. In Proceedings of the 36th Annual ACM Symposium on Applied Computing, pages 1750–1758, 2021.
  36. Cute and jcute: Concolic unit testing and explicit path model-checking tools: (tool paper). In Computer Aided Verification: 18th International Conference, CAV 2006, Seattle, WA, USA, August 17-20, 2006. Proceedings 18, pages 419–423. Springer, 2006.
  37. Cute: A concolic unit testing engine for c. ACM SIGSOFT Software Engineering Notes, 30(5):263–272, 2005.
  38. Automated discovery of valid test strings from the web using dynamic regular expressions collation and natural language processing. In 2012 12th International Conference on Quality Software, pages 79–88. IEEE, 2012.
  39. Do automatically generated unit tests find real faults? an empirical study of effectiveness and challenges (t). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 201–211. IEEE, 2015.
  40. Generation of string test input from web using regular expression. International Journal of Computer Applications, 975:8887, 2014.
  41. Saying ’hi!’ is not enough: mining inputs for effective test generation. In Proceedings of the 32nd International Conference on Automated Software Engineering, pages 44–49. IEEE Computer Society, 2017.
  42. Mutation-inspired symbolic execution for software testing. IET Software, 16(5):478–492, 2022.
  43. Symstra: A framework for generating object-oriented unit tests using symbolic execution. In Tools and Algorithms for the Construction and Analysis of Systems: 11th International Conference, TACAS 2005, Held as Part of the Joint European Conferences on Theory and Practice of Software, ETAPS 2005, Edinburgh, UK, April 4-8, 2005. Proceedings 11, pages 365–381. Springer, 2005.
  44. Hao Zhong. Enriching compiler testing with real program from bug report. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pages 1–12, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Wendkûuni C. Ouédraogo (6 papers)
  2. Laura Plein (5 papers)
  3. Kader Kaboré (3 papers)
  4. Andrew Habib (9 papers)
  5. Jacques Klein (89 papers)
  6. David Lo (229 papers)
  7. Tegawendé F. Bissyandé (82 papers)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets