Analysis and sample-size determination for $2^K$ audit experiments with binary response and application to identification of effect of racial discrimination on access to justice

Abstract: Social scientists have increasingly turned to audit experiments to investigate discrimination in the market for jobs, loans, housing and other opportunities. In a typical audit experiment, researchers assign ``signals'' (the treatment) to subjects at random and compare success rates across treatment conditions. In the recent past there has been increased interest in using randomized multifactor designs for audit experiments, popularly called factorial experiments, in which combinations of multiple signals are assigned to subjects. Although social scientists have manipulated multiple factors like race, gender and income, the analyses have been mostly exploratory in nature. In this paper we lay out a comprehensive methodology for design and analysis of $2^K$ factorial designs with binary response using model-free, randomization-based Neymanian inference and demonstrate its application by analyzing the audit experiment reported in Libgober (2020). Specifically, we integrate and extend several sections of the randomization-based, finite-population literature for binary outcomes, including sample size and power calculations, and non-linear factorial estimators, extending results.