Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

MR-Scout: Automated Synthesis of Metamorphic Relations from Existing Test Cases (2304.07548v4)

Published 15 Apr 2023 in cs.SE

Abstract: Metamorphic Testing (MT) alleviates the oracle problem by defining oracles based on metamorphic relations (MRs), that govern multiple related inputs and their outputs. However, designing MRs is challenging, as it requires domain-specific knowledge. This hinders the widespread adoption of MT. We observe that developer-written test cases can embed domain knowledge that encodes MRs. Such encoded MRs could be synthesized for testing not only their original programs but also other programs that share similar functionalities. In this paper, we propose MR-Scout to automatically synthesize MRs from test cases in open-source software (OSS) projects. MR-Scout first discovers MR-encoded test cases (MTCs), and then synthesizes the encoded MRs into parameterized methods (called codified MRs), and filters out MRs that demonstrate poor quality for new test case generation. MR-Scout discovered over 11,000 MTCs from 701 OSS projects. Experimental results show that over 97% of codified MRs are of high quality for automated test case generation, demonstrating the practical applicability of MR-Scout. Furthermore, codified-MRs-based tests effectively enhance the test adequacy of programs with developer-written tests, leading to 13.52% and 9.42% increases in line coverage and mutation score, respectively. Our qualitative study shows that 55.76% to 76.92% of codified MRs are easily comprehensible for developers.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Testing Web Enabled Simulation at Scale Using Metamorphic Testing. In 43rd IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, ICSE (SEIP) 2021, Madrid, Spain, May 25-28, 2021. IEEE, 140–149. https://doi.org/10.1109/ICSE-SEIP52600.2021.00023
  2. Assessing Robustness of ML-Based Program Analysis Tools using Metamorphic Program Transformations. In 36th IEEE/ACM International Conference on Automated Software Engineering, ASE 2021, Melbourne, Australia, November 15-19, 2021. IEEE, 1377–1381. https://doi.org/10.1109/ASE51524.2021.9678706
  3. Andrea Arcuri and Lionel C. Briand. 2014. A Hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Softw. Test. Verification Reliab. 24, 3 (2014), 219–250. https://doi.org/10.1002/STVR.1486
  4. Generating metamorphic relations for cyber-physical systems with genetic programming: an industrial case study. In ESEC/FSE ’21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece, August 23-28, 2021, Diomidis Spinellis, Georgios Gousios, Marsha Chechik, and Massimiliano Di Penta (Eds.). ACM, 1264–1274. https://doi.org/10.1145/3468264.3473920
  5. Evolutionary generation of metamorphic relations for cyber-physical systems. In GECCO ’22: Genetic and Evolutionary Computation Conference, Companion Volume, Boston, Massachusetts, USA, July 9 - 13, 2022, Jonathan E. Fieldsend and Markus Wagner (Eds.). ACM, 15–16. https://doi.org/10.1145/3520304.3534077
  6. MeMo: Automatically identifying metamorphic relations in Javadoc comments for test automation. J. Syst. Softw. 181 (2021), 111041. https://doi.org/10.1016/j.jss.2021.111041
  7. Hudson Borges and Marco Túlio Valente. 2018. What’s in a GitHub Star? Understanding Repository Starring Practices in a Social Coding Platform. J. Syst. Softw. 146 (2018), 112–129. https://doi.org/10.1016/j.jss.2018.09.016
  8. Cristian Cadar and Koushik Sen. 2013. Symbolic execution for software testing: three decades later. Commun. ACM 56, 2 (2013), 82–90. https://doi.org/10.1145/2408776.2408795
  9. SemMT: A Semantic-Based Testing Approach for Machine Translation Systems. ACM Trans. Softw. Eng. Methodol. 31, 2 (2022), 34e:1–34e:36. https://doi.org/10.1145/3490488
  10. Metamorphic Testing: A New Approach for Generating Next Test Cases. Technical Report. Technical Report HKUST-CS98-01, Department of Computer Science, The Hong Kong University of Science and Technology.
  11. Metamorphic Testing: A Review of Challenges and Opportunities. ACM Comput. Surv. 51, 1 (2018), 4:1–4:27. https://doi.org/10.1145/3143561
  12. METRIC: METamorphic Relation Identification based on the Category-choice framework. J. Syst. Softw. 116 (2016), 177–190. https://doi.org/10.1016/j.jss.2015.07.037
  13. InterEvo-TR: Interactive Evolutionary Test Generation With Readability Assessment. IEEE Trans. Software Eng. 49, 4 (2023), 2580–2596. https://doi.org/10.1109/TSE.2022.3227418
  14. Alastair F. Donaldson. 2019. Metamorphic testing of Android graphics drivers. In Proceedings of the 4th International Workshop on Metamorphic Testing, MET@ICSE 2019, Montreal, QC, Canada, May 26, 2019, Xiaoyuan Xie, Pak-Lok Poon, and Laura L. Pullum (Eds.). IEEE / ACM, 1. https://doi.org/10.1109/MET.2019.00008
  15. Alastair F. Donaldson and Andrei Lascu. 2016. Metamorphic testing for (graphics) compilers. In Proceedings of the 1st International Workshop on Metamorphic Testing, MET@ICSE 2016, Austin, Texas, USA, May 16, 2016. ACM, 44–47. https://doi.org/10.1145/2896971.2896978
  16. EvoSuite. 2023. EvoSuite. Retrieved August 20, 2023 from https://www.evosuite.org/
  17. Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: automatic test suite generation for object-oriented software. In SIGSOFT/FSE’11 19th ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE-19) and ESEC’11: 13th European Software Engineering Conference (ESEC-13), Szeged, Hungary, September 5-9, 2011, Tibor Gyimóthy and Andreas Zeller (Eds.). ACM, 416–419. https://doi.org/10.1145/2025113.2025179
  18. Gordon Fraser and Andrea Arcuri. 2013a. EvoSuite: On the Challenges of Test Case Generation in the Real World. In Sixth IEEE International Conference on Software Testing, Verification and Validation, ICST 2013, Luxembourg, Luxembourg, March 18-22, 2013. IEEE Computer Society, 362–369. https://doi.org/10.1109/ICST.2013.51
  19. Gordon Fraser and Andrea Arcuri. 2013b. Whole Test Suite Generation. IEEE Trans. Software Eng. 39, 2, 276–291. https://doi.org/10.1109/TSE.2012.14
  20. Gordon Fraser and Andreas Zeller. 2011. Generating parameterized unit tests. In Proceedings of the 20th International Symposium on Software Testing and Analysis, ISSTA 2011, Toronto, ON, Canada, July 17-21, 2011, Matthew B. Dwyer and Frank Tip (Eds.). ACM, 364–374. https://doi.org/10.1145/2001420.2001464
  21. SBST Tool Competition 2022. In 15th IEEE/ACM International Workshop on Search-Based Software Testing, SBST@ICSE 2022, Pittsburgh, PA, USA, May 9, 2022. IEEE, 25–32. https://doi.org/10.1145/3526072.3527538
  22. GitHub. 2023. GitHub. Retrieved August 20, 2023 from https://github.com/
  23. Grammarly. 2023. Grammarly. Retrieved August 20, 2023 from http://grammarly.com
  24. Achievements, Open Problems and Challenges for Search Based Software Testing. In 8th IEEE International Conference on Software Testing, Verification and Validation, ICST 2015, Graz, Austria, April 13-17, 2015. IEEE Computer Society, 1–12. https://doi.org/10.1109/ICST.2015.7102580
  25. Handbook 151: Nist/sematech e-handbook of statistical methods. (2002).
  26. Characterizing usages, updates and risks of third-party libraries in Java projects. Empir. Softw. Eng. 27, 4 (2022), 90. https://doi.org/10.1007/s10664-022-10131-8
  27. Test oracle assessment and improvement. In Proceedings of the 25th International Symposium on Software Testing and Analysis, ISSTA 2016, Saarbrücken, Germany, July 18-20, 2016, Andreas Zeller and Abhik Roychoudhury (Eds.). ACM, 247–258. https://doi.org/10.1145/2931037.2931062
  28. Junit. 2023a. Junit4. Retrieved August 20, 2023 from https://junit.org/junit4/javadoc/4.13/org/junit/Assert.html
  29. Junit. 2023b. Junit5. Retrieved August 20, 2023 from https://junit.org/junit5/
  30. Junit. 2023c. Junit5 Assertions. Retrieved August 20, 2023 from https://junit.org/junit5/docs/5.0.3/api/org/junit/jupiter/api/Assertions.html
  31. Alexander Kampmann and Andreas Zeller. 2019. Carving parameterized unit tests. In Proceedings of the 41st International Conference on Software Engineering: Companion Proceedings, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019, Joanne M. Atlee, Tevfik Bultan, and Jon Whittle (Eds.). IEEE / ACM, 248–249. https://doi.org/10.1109/ICSE-COMPANION.2019.00098
  32. Upulee Kanewala and James M. Bieman. 2013. Using machine learning techniques to detect metamorphic relations for programs without test oracles. In IEEE 24th International Symposium on Software Reliability Engineering, ISSRE 2013, Pasadena, CA, USA, November 4-7, 2013. IEEE Computer Society, 1–10. https://doi.org/10.1109/ISSRE.2013.6698899
  33. Predicting metamorphic relations for testing scientific software: a machine learning approach using graph kernels. Softw. Test. Verification Reliab. 26, 3 (2016), 245–269. https://doi.org/10.1002/stvr.1594
  34. Graph-based seed object synthesis for search-based unit testing. In ESEC/FSE ’21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece, August 23-28, 2021, Diomidis Spinellis, Georgios Gousios, Marsha Chechik, and Massimiliano Di Penta (Eds.). ACM, 1068–1080. https://doi.org/10.1145/3468264.3468619
  35. Metamorphic Model-Based Testing of Autonomous Systems. In 2nd IEEE/ACM International Workshop on Metamorphic Testing, MET@ICSE 2017, Buenos Aires, Argentina, May 22, 2017. IEEE Computer Society, 35–41. https://doi.org/10.1109/MET.2017.6
  36. Fuzzing Deep Learning Compilers with HirGen. , 248–260 pages. https://doi.org/10.1145/3597926.3598053
  37. Metamorphic Testing and Certified Mitigation of Fairness Violations in NLP Models. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, Christian Bessiere (Ed.). ijcai.org, 458–465. https://doi.org/10.24963/ijcai.2020/64
  38. OpenAI. 2023. ChatGPT. Retrieved August 20, 2023 from https://openai.com/blog/chatgpt
  39. Oracle. 2023. Java Language Specification. Retrieved August 20, 2023 from https://docs.oracle.com/javase/specs/
  40. Carlos Pacheco and Michael D. Ernst. 2007. Randoop: feedback-directed random testing for Java. In Companion to the 22nd Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2007, October 21-25, 2007, Montreal, Quebec, Canada, Richard P. Gabriel, David F. Bacon, Cristina Videira Lopes, and Guy L. Steele Jr. (Eds.). ACM, 815–816. https://doi.org/10.1145/1297846.1297902
  41. Matteo Paltenghi and Michael Pradel. 2023. MorphQ: Metamorphic Testing of the Qiskit Quantum Computing Platform. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 2413–2424. https://doi.org/10.1109/ICSE48619.2023.00202
  42. PITest. 2023. PITest. Retrieved August 20, 2023 from https://pitest.org/
  43. Theoretical and Empirical Analyses of the Effectiveness of Metamorphic Relation Composition. IEEE Trans. Software Eng. 48, 3 (2022), 1001–1017. https://doi.org/10.1109/TSE.2020.3009698
  44. John A Rice. 2006. Mathematical statistics and data analysis. Cengage Learning.
  45. A Template-Based Approach to Describing Metamorphic Relations. In 2nd IEEE/ACM International Workshop on Metamorphic Testing, MET@ICSE 2017, Buenos Aires, Argentina, May 22, 2017. IEEE Computer Society, 3–9. https://doi.org/10.1109/MET.2017.3
  46. A Survey on Metamorphic Testing. IEEE Trans. Software Eng. 42, 9 (2016), 805–824. https://doi.org/10.1109/TSE.2016.2532875
  47. Metamorphic testing of RESTful web APIs. In Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018, Michel Chaudron, Ivica Crnkovic, Marsha Chechik, and Mark Harman (Eds.). ACM, 882. https://doi.org/10.1145/3180155.3182528
  48. METRIC$^{+}$+: A Metamorphic Relation Identification Technique Based on Input Plus Output Domains. IEEE Trans. Software Eng. 47, 9 (2021), 1764–1785. https://doi.org/10.1109/TSE.2019.2934848
  49. μ𝜇\muitalic_μMT: a data mutation directed metamorphic relation acquisition methodology. In Proceedings of the 1st International Workshop on Metamorphic Testing, MET@ICSE 2016, Austin, Texas, USA, May 16, 2016. ACM, 12–18. https://doi.org/10.1145/2896971.2896974
  50. Evolutionary improvement of assertion oracles. In ESEC/FSE ’20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020, Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann (Eds.). ACM, 1178–1189. https://doi.org/10.1145/3368089.3409758
  51. TestNG. 2023. TestNG. Retrieved August 20, 2023 from https://testng.org/doc/
  52. Retrofitting Unit Tests for Parameterized Unit Testing. In Fundamental Approaches to Software Engineering - 14th International Conference, FASE 2011, Held as Part of the Joint European Conferences on Theory and Practice of Software, ETAPS 2011, Saarbrücken, Germany, March 26-April 3, 2011. Proceedings (Lecture Notes in Computer Science, Vol. 6603), Dimitra Giannakopoulou and Fernando Orejas (Eds.). Springer, 294–309. https://doi.org/10.1007/978-3-642-19811-3_21
  53. To what extent do DNN-based image classification models make unreliable inferences? Empir. Softw. Eng. 26, 4 (2021), 84. https://doi.org/10.1007/s10664-021-09985-1
  54. MR-Scout. 2023. MR-Scout. Retrieved August 20, 2023 from https://mr-scout.github.io
  55. Shuai Wang and Zhendong Su. 2020. Metamorphic Object Insertion for Testing Object Detection Systems. (2020), 1053–1065. https://doi.org/10.1145/3324884.3416584
  56. An Empirical Study of Usages, Updates and Risks of Third-Party Libraries in Java Projects. In IEEE International Conference on Software Maintenance and Evolution, ICSME 2020, Adelaide, Australia, September 28 - October 2, 2020. IEEE, 35–45. https://doi.org/10.1109/ICSME46990.2020.00014
  57. Metamorphic Testing of Deep Learning Compilers. Proc. ACM Meas. Anal. Comput. Syst. 6, 1 (2022), 15:1–15:28. https://doi.org/10.1145/3508035
  58. Automatic Discovery and Cleansing of Numerical Metamorphic Relations. In 2019 IEEE International Conference on Software Maintenance and Evolution, ICSME 2019, Cleveland, OH, USA, September 29 - October 4, 2019. IEEE, 235–245. https://doi.org/10.1109/ICSME.2019.00035
  59. Search-based inference of polynomial metamorphic relations. In ACM/IEEE International Conference on Automated Software Engineering, ASE ’14, Vasteras, Sweden - September 15 - 19, 2014, Ivica Crnkovic, Marsha Chechik, and Paul Grünbacher (Eds.). ACM, 701–712. https://doi.org/10.1145/2642937.2642994
  60. Metamorphic Relations for Enhancing System Understanding and Use. IEEE Trans. Software Eng. 46, 10 (2020), 1120–1154. https://doi.org/10.1109/TSE.2018.2876433
  61. MockSniffer: Characterizing and Recommending Mocking Decisions for Unit Tests. In 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020, Melbourne, Australia, September 21-25, 2020. IEEE, 436–447. https://doi.org/10.1145/3324884.3416539
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Congying Xu (5 papers)
  2. Valerio Terragni (12 papers)
  3. Hengcheng Zhu (4 papers)
  4. Jiarong Wu (13 papers)
  5. Shing-Chi Cheung (54 papers)
Citations (5)
View on Semantic Scholar