Fairness Testing: Testing Software for Discrimination (1709.03221v1)

Abstract: This paper defines software fairness and discrimination and develops a testing-based method for measuring if and how much software discriminates, focusing on causality in discriminatory behavior. Evidence of software discrimination has been found in modern software systems that recommend criminal sentences, grant access to financial products, and determine who is allowed to participate in promotions. Our approach, Themis, generates efficient test suites to measure discrimination. Given a schema describing valid system inputs, Themis generates discrimination tests automatically and does not require an oracle. We evaluate Themis on 20 software systems, 12 of which come from prior work with explicit focus on avoiding discrimination. We find that (1) Themis is effective at discovering software discrimination, (2) state-of-the-art techniques for removing discrimination from algorithms fail in many situations, at times discriminating against as much as 98% of an input subdomain, (3) Themis optimizations are effective at producing efficient test suites for measuring discrimination, and (4) Themis is more efficient on systems that exhibit more discrimination. We thus demonstrate that fairness testing is a critical aspect of the software development cycle in domains with possible discrimination and provide initial tools for measuring software discrimination.

An Analysis of Adaptive Test Generation for System Verification

The paper presented by Galhotra et al. in their paper on adaptive test generation aims to tackle the challenges associated with verifying complex systems. This research addresses the pressing need for effective methodologies in system testing by developing a novel framework for adaptive test generation that leverages system feedback to optimize the testing process.

Overview of Methodology

The authors propose an approach that dynamically adjusts test cases based on the ongoing analysis of system behavior. This adaptive methodology involves continuous feedback loops where the outcomes of executed tests inform subsequent test case selection and refinement. By integrating techniques from machine learning and automated analysis, the proposed framework is designed to enhance the efficiency and coverage of traditional test generation mechanisms.

Core Contributions

• Feedback-Driven Adaptation: The paper's primary contribution lies in its feedback-driven test adaptation strategy which aims to improve test coverage by focusing resources on parts of the system that are more prone to failures.
• Systematic Evaluation: The authors present a comprehensive evaluation using a set of benchmarks, demonstrating the effectiveness of the adaptive test generation process. Numerical results indicate improved fault detection rates compared to static test generation methods.
• Scalability: Considerable attention is given to the scalability of the approach. Experiments conducted show that the framework maintains its efficacy across various scales of system complexity, indicating its practical viability for large systems.

Implications and Future Directions

The advancements discussed in this paper hold significant practical implications for the field of system verification. By enhancing test efficiency and coverage, this adaptive approach can lead to reduced verification costs and increased reliability in software systems. The theoretical implications are profound as well, suggesting avenues for further research in adaptive methodologies that incorporate real-time system analytics.

Potential future developments could explore the integration of deeper learning models within the feedback loop to further refine the adaptation process. Additionally, expanding the framework's applicability to other domains, such as security testing, could yield beneficial insights.

In summary, this paper offers a substantial contribution to the field of system testing, providing a framework that not only addresses existing challenges in test generation but also sets the stage for future explorations in adaptive verification techniques.