Improvement consistency of MILP-based selection rules

Establish whether the selection rules constructed using mixed integer linear programs for common utility specifications—specifically, the linear-classifier selection rules for the classification rate utility and the MILP selection rules for the calibration utility with capacity constraints—are improvement consistent, i.e., that whenever there exists an algorithm in the specified class achieving a δ-fairness or δ-accuracy improvement over the status quo algorithm a0 in the population, the selection rule asymptotically selects a candidate algorithm whose group-specific accuracy utilities and fairness utilities converge in probability to those of an improving algorithm.

Background

The paper’s main consistency result shows that their proposed hypothesis test for fairness- or accuracy-improvability is consistent provided the algorithm-selection rule used in the training phase is “improvement consistent.” This property requires that, when a genuine improvement exists within the algorithm class, the selection rule asymptotically finds an algorithm whose fairness and accuracy utilities converge to those of an improving algorithm.

To make the method practical, the authors propose concrete selection rules in Appendix Section 6 based on mixed integer linear programs (MILPs) for two widely used utility specifications: (i) classification rate with linear classifiers and (ii) calibration utility under capacity constraints. Although these rules are designed to target fairness and accuracy improvements, their improvement consistency is not proved. Establishing this property would strengthen the theoretical guarantees and applicability of the testing framework in common settings.