Mutatis Mutandis: Revisiting the Comparator in Discrimination Testing

Abstract: Testing for discrimination consists of deriving a profile, known as the comparator, similar to the profile making the discrimination claim, known as the complainant, and comparing the outcomes of these two profiles. An important aspect for establishing discrimination is evidence, often obtained via discrimination testing tools that implement the complainant-comparator pair. In this work, we revisit the role of the comparator in discrimination testing. We argue for the causal modeling nature of deriving the comparator, and introduce a two-kinds classification for the comparator: the ceteris paribus (CP), and mutatis mutandis (MM) comparators. The CP comparator is the standard one among discrimination testing, representing an idealized comparison as it aims for having a complainant-comparator pair that only differs on membership to the protected attribute. As an alternative to it, we define the MM comparator, which requires that the comparator represents what would have been of the complainant without the effects of the protected attribute on the non-protected attributes. The complainant-comparator pair, in that case, may also be dissimilar in terms of all attributes. We illustrate these two comparators and their impact on discrimination testing using a real illustrative example. Importantly, we position generative models and, overall, machine learning methods as useful tools for constructing the MM comparator and, in turn, achieving more complex and realistic comparisons when testing for discrimination.