Identify which test (benchmark or outcome) is correct under MLRP

Determine, within the two-group decision-making framework with binary outcomes and group-specific thresholds t0 and t1, which of the benchmark test (comparison of decision rates across groups) or the standard outcome test (comparison of success rates among those receiving a positive decision across groups) correctly reflects the ordering of the thresholds under the monotone likelihood ratio property for the group-specific risk distributions, given that at least one must be correct but the correct one is a priori unknown.

Background

The paper proves that under the monotone likelihood ratio property (MLRP) on group-specific risk distributions, at least one of two classical analyses—the benchmark test and the outcome test—must correctly detect whether decision-makers apply unequal thresholds t0 and t1 across groups. Consequently, when both tests agree, their conclusion is guaranteed to be correct.

However, the authors note that although one of these tests must be correct under MLRP, it is not known which test will point in the right direction in a given setting without additional information (such as knowing which group has the higher base rate). This unresolved question limits the interpretability of single-test conclusions and motivates the combined test strategy.

References

In proving Proposition~\ref{prop:simple}, the key insight is that, under the MLRP, whether group G = 0 is the lower or higher base rate group, either the benchmark or the standard outcome test will correctly detect the relative ordering of t_0 and t_1---though we do not know which one.

— A Simple, Statistically Robust Test of Discrimination  (2407.06539 - Gaebler et al., 2024) in Section 2 (Statistical Guarantees), end of Proof of Proposition 1