Metamorphic Testing: A New Approach for Generating Next Test Cases (2002.12543v1)

Abstract: In software testing, a set of test cases is constructed according to some predefined selection criteria. The software is then examined against these test cases. Three interesting observations have been made on the current artifacts of software testing. Firstly, an error-revealing test case is considered useful while a successful test case which does not reveal software errors is usually not further investigated. Whether these successful test cases still contain useful information for revealing software errors has not been properly studied. Secondly, no matter how extensive the testing has been conducted in the development phase, errors may still exist in the software [5]. These errors, if left undetected, may eventually cause damage to the production system. The study of techniques for uncovering software errors in the production phase is seldom addressed in the literature. Thirdly, as indicated by Weyuker in [6], the availability of test oracles is pragmatically unattainable in most situations. However, the availability of test oracles is generally assumed in conventional software testing techniques. In this paper, we propose a novel test case selection technique that derives new test cases from the successful ones. The selection aims at revealing software errors that are possibly left undetected in successful test cases which may be generated using some existing strategies. As such, the proposed technique augments the effectiveness of existing test selection strategies. The technique also helps uncover software errors in the production phase and can be used in the absence of test oracles.

Metamorphic Testing: A Novel Approach for Generating Test Cases

This paper, published in 1998 by the Hong Kong University of Science and Technology, introduces a method known as Metamorphic Testing (MT), a novel software testing technique designed to enhance the detection of software errors. The approach focuses on generating new test cases derived from existing successful ones to reveal software defects that traditional methods often miss.

Overview and Methodology

Metamorphic Testing addresses a critical gap in software testing: the challenge of identifying software errors in situations where test oracles are absent or infeasible. It is particularly applicable to software systems in development and production phases. In many cases, successful test cases might appear to provide correct results, yet they may be missing subtle errors that surface only under specific conditions. The MT technique seeks to uncover these hidden errors by transforming successful test cases into new ones, thus increasing the probability that the underlying software defects are detected.

Key Contributions

1. Derivation of New Test Cases: By leveraging input-output pairs from previous successful test cases, Metamorphic Testing can generate new test cases aimed at uncovering software errors not initially apparent.
2. Applicability Without Test Oracles: The technique notably operates without the need for a predefined test oracle, making it suitable for complex applications where verifying output correctness is inherently challenging (e.g., graphical applications or systems with non-trivial calculations).
3. Integration with Existing Strategies: MT can be combined with conventional test case selection strategies, enhancing overall testing efficacy.

Examples and Application

The paper provides detailed examples to illustrate the efficacy of MT. These examples span diverse scenarios, such as binary search, sorting algorithms, and graph traversal, demonstrating MT’s ability to identify errors like incorrect path selection or unexpected overwriting of data.

For instance, in the context of a binary search algorithm, MT generates input variations that challenge the assumptions of order and data integrity, thus exposing potential indexing errors or unexpected behavior due to edge cases.

Implications

The implications of this research are significant for both theoretical and practical domains. Theoretically, it introduces a robust mechanism for error detection that does not rely on external validation via oracles. Practically, it enables developers to build more reliable software by systematically exploring and testing beyond conventional boundaries.

Future Developments

Looking forward, further research could involve the formalization of MT into a structured methodology applicable across various domains. Developing domain-specific guidelines would enhance its implementation, catering to particular nuances and requirements of different software systems.

Also, integrating MT with automated testing frameworks could automate the generation of metamorphic relations and subsequent test cases, leading to more efficient continuous integration and deployment pipelines.

Conclusion

Metamorphic Testing presents a valuable contribution to the field of software engineering by proposing a method to enhance error detection, especially in scenarios lacking traditional test oracles. Its applicability across various phases of software development makes it a versatile tool, enabling more comprehensive testing regimes and contributing to the overall robustness and reliability of software systems.