Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

GenMorph: Automatically Generating Metamorphic Relations via Genetic Programming (2312.15302v3)

Published 23 Dec 2023 in cs.SE

Abstract: Metamorphic testing is a popular approach that aims to alleviate the oracle problem in software testing. At the core of this approach are Metamorphic Relations (MRs), specifying properties that hold among multiple test inputs and corresponding outputs. Deriving MRs is mostly a manual activity, since their automated generation is a challenging and largely unexplored problem. This paper presents GenMorph, a technique to automatically generate MRs for Java methods that involve inputs and outputs that are boolean, numerical, or ordered sequences. GenMorph uses an evolutionary algorithm to search for effective test oracles, i.e., oracles that trigger no false alarms and expose software faults in the method under test. The proposed search algorithm is guided by two fitness functions that measure the number of false alarms and the number of missed faults for the generated MRs. Our results show that GenMorph generates effective MRs for 18 out of 23 methods (mutation score >20%). Furthermore, it can increase Randoop's fault detection capability in 7 out of 23 methods, and Evosuite's in 14 out of 23 methods. When compared with AutoMR, a state-of-the-art MR generator, GenMorph also outperformed its fault detection capability in 9 out of 10 methods.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,” IEEE transactions on software engineering, vol. 41, no. 5, pp. 507–525, 2014.
  2. G. Fraser and A. Arcuri, “Evosuite: automatic test suite generation for object-oriented software,” in Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering, 2011, pp. 416–419.
  3. C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball, “Feedback-directed random test generation,” in 29th International Conference on Software Engineering (ICSE’07).   IEEE, 2007, pp. 75–84.
  4. S. Lukasczyk, F. Kroiß, and G. Fraser, “Automated unit test generation for python,” in International Symposium on Search Based Software Engineering.   Springer, 2020, pp. 9–24.
  5. P. McMinn, “Search-based software test data generation: a survey,” Software testing, Verification and reliability, vol. 14, no. 2, pp. 105–156, 2004.
  6. A. Panichella, F. M. Kifetew, and P. Tonella, “Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets,” IEEE Transactions on Software Engineering, vol. 44, no. 2, pp. 122–158, 2017.
  7. M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “A comprehensive survey of trends in oracles for software testing,” University of Sheffield, Department of Computer Science, Tech. Rep. CS-13-01, 2013.
  8. T. Y. Chen, S. C. Cheung, and S. M. Yiu, “Metamorphic testing: A new approach for generating next test cases,” Technical Report HKUST-CS98-01, Department of Computer Science, The Hong Kong University of Science and Technology, Tech. Rep., 1998.
  9. T. Y. Chen, F.-C. Kuo, H. Liu, P.-L. Poon, D. Towey, T. H. Tse, and Z. Q. Zhou, “Metamorphic testing: A review of challenges and opportunities,” ACM Computing Surveys, vol. 51, no. 1, pp. 4:1–4:27, Jan. 2018.
  10. S. Segura, G. Fraser, A. Sanchez, and A. Ruiz-Cortés, “A survey on metamorphic testing,” IEEE Transactions on Software Engineering, vol. 42, no. 9, pp. 805–824, Sept 2016.
  11. S. Shamshiri, R. Just, J. M. Rojas, G. Fraser, P. McMinn, and A. Arcuri, “Do automatically generated unit tests find real faults? an empirical study of effectiveness and challenges,” in Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’15.   IEEE Press, 2015, p. 201–211. [Online]. Available: https://doi.org/10.1109/ASE.2015.86
  12. T. Y. Chen and T. Tse, “New visions on metamorphic testing after a quarter of a century of inception,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2021, pp. 1487–1490.
  13. J. Ahlgren, M. E. Berezin, K. Bojarczuk, E. Dulskyte, I. Dvortsova, J. George, N. Gucevska, M. Harman, M. Lomeli, E. Meijer et al., “Testing web enabled simulation at scale using metamorphic testing,” in Proceedings of the 43rd International Conference on Software Engineering, 2021.
  14. Google, “graphicsfuzz,” https://github.com/google/graphicsfuzz, 2022.
  15. D. C. Jarman, Z. Q. Zhou, and T. Y. Chen, “Metamorphic testing for adobe data analytics software,” in 2017 IEEE/ACM 2nd International Workshop on Metamorphic Testing (MET), May 2017, pp. 21–27.
  16. M. Lindvall, D. Ganesan, R. Ardal, and R. Wiegand, “Metamorphic model-based testing applied on nasa dat – an experience report,” in Software Engineering (ICSE), 2015 IEEE/ACM 37th IEEE International Conference on, vol. 2, May 2015, pp. 129–138.
  17. T. Y. Chen, “Metamorphic testing: A simple approach to alleviate the oracle problem,” in Fifth IEEE International Symposium on Service Oriented System Engineering (SOSE), 2010, June 2010, pp. 1–2.
  18. B. Zhang, H. Zhang, J. Chen, D. Hao, and P. Moscato, “Automatic discovery and cleansing of numerical metamorphic relations,” in 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2019, pp. 235–245.
  19. J. Zhang, J. Chen, D. Hao, Y. Xiong, B. Xie, L. Zhang, and H. Mei, “Search-based inference of polynomial metamorphic relations,” in Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering, ser. ASE ’14.   New York, NY, USA: ACM, 2014, pp. 701–712. [Online]. Available: http://doi.acm.org/10.1145/2642937.2642994
  20. G. Jahangirova, D. Clark, M. Harman, and P. Tonella, “Test oracle assessment and improvement,” in Proceedings of the 25th International Symposium on Software Testing and Analysis, ISSTA 2016, Saarbrücken, Germany, July 18-20, 2016.   ACM, 2016, pp. 247–258.
  21. G. Jahangirova, D. Clark, M. Harman, and P. Tonella, “An empirical validation of oracle improvement,” IEEE Transactions on Software Engineering, 2019.
  22. G. Jahangirova, D. Clark, M. Harman, and P. Tonella, “Oasis: oracle assessment and improvement tool,” in Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2018, Amsterdam, The Netherlands, July 16-21, 2018, F. Tip and E. Bodden, Eds.   ACM, 2018, pp. 368–371.
  23. J. Ayerdi, V. Terragni, G. Jahangirova, A. Arrieta, and P. Tonella, “Replication package,” https://doi.org/10.5281/zenodo.10067096, 2023, last access: Nov 2023.
  24. Z. Q. Zhou, L. Sun, T. Y. Chen, and D. Towey, “Metamorphic relations for enhancing system understanding and use,” IEEE Transactions on Software Engineering, 2018. [Online]. Available: https://doi.org/10.1109/TSE.2018.2876433
  25. V. Terragni, G. Jahangirova, P. Tonella, and M. Pezzè, “Evolutionary improvement of assertion oracles,” in ESEC/FSE ’20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020.   ACM, 2020, pp. 1178–1189.
  26. F. Molina, P. Ponzio, N. Aguirre, and M. Frias, “Evospex: An evolutionary algorithm for learning postconditions,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), 2021, pp. 1223–1235.
  27. G. Fraser and A. Arcuri, “Whole test suite generation,” IEEE Transactions on Software Engineering, vol. 39, no. 2, pp. 276–291, 2012.
  28. R. Just, “The major mutation framework: Efficient and scalable mutation analysis for java,” in Proceedings of the 2014 international symposium on software testing and analysis, 2014, pp. 433–436.
  29. A. Duque-Torres, D. Pfahl, R. Ramler, and C. Klammer, “A replication study on predicting metamorphic relations at unit testing level,” in 2022 IEEE 29th International Conference on Software Analysis, Evolution and Reengineering (SANER).   IEEE, 2022, pp. 698–708.
  30. Apache, “Apache Commons Math - pow method,” https://commons.apache.org/proper/commons-math/javadocs/api-3.6.1/org/apache/commons/math3/util/ArithmeticUtils.html\#pow(int,\%20int), 2023.
  31. U. Kanewala and J. M. Bieman, “Using machine learning techniques to detect metamorphic relations for programs without test oracles,” in 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE).   IEEE, 2013, pp. 1–10.
  32. U. Kanewala, “Techniques for automatic detection of metamorphic relations,” in 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.   IEEE, 2014, pp. 237–238.
  33. K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan, “A fast and elitist multiobjective genetic algorithm: Nsga-ii,” IEEE Transactions on Evolutionary Computation, vol. 6, no. 2, pp. 182–197, 2002.
  34. B. L. Miller, B. L. Miller, D. E. Goldberg, and D. E. Goldberg, “Genetic algorithms, tournament selection, and the effects of noise,” Complex Systems, vol. 9, no. 3, pp. 193–212, 1995.
  35. D. Whitley, “A Genetic Algorithm Tutorial,” Statistics and Computing, vol. 4, no. 2, pp. 65–85, 1994.
  36. M. F. Brameier and W. Banzhaf, “A comparison with tree-based genetic programming,” Linear Genetic Programming, pp. 173–192, 2007.
  37. Apache, “Apache Commons Math,” https://commons.apache.org/proper/commons-math/, 2022.
  38. ——, “Apache Commons Lang,” https://commons.apache.org/proper/commons-lang/, 2022.
  39. Google, “Google Guava,” https://guava.dev/, 2022.
  40. H. Coles, T. Laurent, C. Henard, M. Papadakis, and A. Ventresque, “Pit: a practical mutation testing tool for java,” in Proceedings of the 25th international symposium on software testing and analysis, 2016, pp. 449–452.
  41. B. Hardin and U. Kanewala, “Using semi-supervised learning for predicting metamorphic relations,” in 2018 IEEE/ACM 3rd International Workshop on Metamorphic Testing (MET).   IEEE, 2018, pp. 14–17.
  42. K. Rahman and U. Kanewala, “Predicting metamorphic relations for matrix calculation programs,” in 2018 IEEE/ACM 3rd International Workshop on Metamorphic Testing (MET).   IEEE, 2018, pp. 10–13.
  43. P. Zhang, X. Zhou, P. Pelliccione, and H. Leung, “Rbf-mlmr: A multi-label metamorphic relation prediction approach using rbf neural network,” IEEE access, vol. 5, pp. 21 791–21 805, 2017.
  44. K. Rahman, I. Kahanda, and U. Kanewala, “Mrpredt: Using text mining for metamorphic relation prediction,” in Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, 2020, pp. 420–424.
  45. A. Dwarakanath, M. Ahuja, S. Sikand, R. M. Rao, R. J. C. Bose, N. Dubash, and S. Podder, “Identifying implementation bugs in machine learning based image classifiers using metamorphic testing,” in Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2018, pp. 118–128.
  46. A. Nair, K. Meinke, and S. Eldh, “Leveraging mutants for automatic prediction of metamorphic relations using machine learning,” in Proceedings of the 3rd ACM SIGSOFT International Workshop on Machine Learning Techniques for Software Quality Evaluation, 2019, pp. 1–6.
  47. Z. Hui and S. Huang, “Achievements and challenges of metamorphic testing,” in Fourth World Congress on Software Engineering (WCSE), 2013, Dec 2013, pp. 73–77.
  48. C. Murphy, G. E. Kaiser, and L. Hu, “Properties of machine learning applications for use in metamorphic testing,” 2008.
  49. U. Kanewala and J. M. Bieman, “Using machine learning techniques to detect metamorphic relations for programs without test oracles,” in IEEE 24th International Symposium on Software Reliability Engineering (ISSRE), 2013, Nov 2013, pp. 1–10.
  50. U. Kanewala, J. M. Bieman, and A. Ben-Hur, “Predicting metamorphic relations for testing scientific software: a machine learning approach using graph kernels,” Software Testing, Verification and Reliability, 2015. [Online]. Available: http://dx.doi.org/10.1002/stvr.1594
  51. A. Blasi, A. Gorla, M. D. Ernst, M. Pezzè, and A. Carzaniga, “Memo: Automatically identifying metamorphic relations in javadoc comments for test automation,” J. Syst. Softw., vol. 181, p. 111041, 2021.
  52. R. Poli, J. Kennedy, and T. Blackwell, “Particle swarm optimization,” Swarm intelligence, vol. 1, no. 1, pp. 33–57, 2007.
  53. J. Ayerdi, V. Terragni, A. Arrieta, P. Tonella, G. Sagardui, and M. Arratibel, “Generating metamorphic relations for cyber-physical systems with genetic programming: An industrial case study,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2021.   New York, NY, USA: Association for Computing Machinery, 2021, p. 1264–1274. [Online]. Available: https://doi.org/10.1145/3468264.3473920
  54. V. Terragni, G. Jahangirova, P. Tonella, and M. Pezzè, “Gassert: A fully automated tool to improve assertion oracles,” in Proceedings of the 43nd IEEE/ACM International Conference on Software Engineering, Demonstration Track ICSE-DEMO 2021, Virtual Event, Spain, 24 May – 1 June, 2021 (to appear), 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Jon Ayerdi (2 papers)
  2. Valerio Terragni (12 papers)
  3. Gunel Jahangirova (13 papers)
  4. Aitor Arrieta (16 papers)
  5. Paolo Tonella (42 papers)
Citations (3)
View on Semantic Scholar