Open-source Defect Injection Benchmark Testbed for the Evaluation of Testing

Abstract: A natural method to evaluate the effectiveness of a testing technique is to measure the defect detection rate when applying the created test cases. Here, real or artificial software defects can be injected into the source code of software. For a more extensive evaluation, the injection of artificial defects is usually needed and can be performed via mutation testing using code mutation operators. However, to simulate complex defects arising from a misunderstanding of design specifications, mutation testing might reach its limit in some cases. In this paper, we present an open-source benchmark testbed application that employs a complement method of artificial defect injection. The application is compiled after artificial defects are injected into its source code from predefined building blocks. The majority of the functions and user interface elements are covered by creating front-end-based automated test cases that can be used in experiments.