Matched Pair Calibration for Ranking Fairness

• Matched pair calibration is a fairness method for ranking systems that constructs near-tied item pairs to assess and mitigate bias across sensitive groups.
• It applies statistical tests and constrained optimization using metrics like marginal pairwise calibration (MPC) to detect discrepancies in exposure and utility.
• Empirical studies on datasets such as MovieLens and TREC demonstrate that the technique improves fairness while maintaining utility in ranking performance.

Matched pair calibration is a family of methods for measuring and enforcing fairness in score-based ranking systems, characterized by the explicit construction and statistical comparison of pairs of items—aligned or “matched” across sensitive groups—at points of marginal ranking decision. These approaches generalize calibration intuitions from binary classification to ranking by focusing on outcome parity for items treated equivalently by the model. Across several methodological lines, matched pair calibration provides both fairness testing (as in (Korevaar et al., 2023)) and practical algorithmic interventions (as in (Sonoda, 2021, Beutel et al., 2019)) to detect, quantify, and mitigate ranking bias.

1. Pairwise Matching Frameworks for Fairness Evaluation

Matched pair calibration centers on constructing item pairs for which the model presents near-indifference, typically by identifying pairs $(i_g, i_{\neg g})$ from different groups with model scores $s(i_g)$ and $s(i_{\neg g})$ satisfying $0 \leq s(i_{\neg g}) - s(i_g) \leq \epsilon$ for small $\epsilon$. Formally, for item set $I$, observed data $D \subset I$, and group $g$,

$$\mathrm{MP}_\epsilon(g; D) := \left\{ (i_g, i_{\neg g}) : g(i_g) = g, g(i_{\neg g}) \neq g,\ i_g, i_{\neg g} \in D,\ 0 \leq s(i_{\neg g}) - s(i_g) \leq \epsilon \right\}.$$

Comparing outcomes $Y(i_g)$ and $Y(i_{\neg g})$ in these pairs quantifies whether the system assigns equivalent exposure or utility across groups in matched ranking decisions (Korevaar et al., 2023).

This matched-pair logic also underpins algorithmic interventions: in training data, all ordered pairs $(i, j)$ within a candidate set $R_q$ for each query $q$ are matched and labeled for pairwise preference (e.g., $l_{ij} = 1\{y_i > y_j\}$) (Sonoda, 2021). Protected-group membership indicators $g_k(x)$ define cross-group matches and enable encoding of fairness constraints directly on pairwise orderings.

2. Fairness Notions and Calibration Metrics in the Pairwise Setting

Matched pair approaches offer a range of fairness metrics and constraints:

Pairwise Calibration Test

The central matched-pair calibration metric measures average marginal outcome gaps: $$\mathrm{MPC}_\epsilon(g; D) = \frac{1}{|\mathrm{MP}_\epsilon(g; D)|} \sum_{(i_g, i_{\neg g}) \in \mathrm{MP}_\epsilon(g; D)} [ Y(i_{\neg g}) - Y(i_g) ].$$ Under group-neutral treatment at the margin, this expectation is zero. Systematically positive (or negative) values reveal exposure or utility bias against group $g$ among near-tied items (Korevaar et al., 2023).

Algorithmic Constraints

Pairwise fairness can be encoded as linear constraints of the form

$\hat l_q(x_i, x_j) := \sigma(h_q(x_i) - h_q(x_j)),$

where $\sigma(z) = 1/(1 + e^{-z})$ and $h$ is the ranking model’s scoring function. Standard constraints include:

• Statistical Pairwise Parity: Equal expected rank exposure across groups.

$$c^\mathrm{pair}_{kl}(q, x_{ij}, 1) = \frac{g_k(x_i) g_l(x_j)}{Z_{kl}}$$

• Inter-group and intra-group pairwise accuracy: Fine-grained accuracy constraints along group boundaries (Sonoda, 2021).

3. Optimization Strategies: Constrained Losses and Dual Weighting

Fair ranking with matched-pair calibration arises as a constrained minimization of the pairwise KL-divergence loss,

$$\min_\theta\ \mathcal{L}(\theta) \quad \text{subject to} \quad A(\theta) \leq 0,$$

where $A(\theta)$ stacks groupwise constraint violations, e.g., $$A_{kl}(\theta) = E[\langle \hat l_\theta(x_{ij}), c^\mathrm{pair}_{kl}(x_{ij}) \rangle] - \epsilon_{kl}$$ (Sonoda, 2021).

A core result shows the existence of closed-form label reweighting: $$w(q,x_{ij},l) = \frac{\tilde w(q,x_{ij},l)}{\sum_{l'} \tilde w(q,x_{ij},l')},\quad \tilde w(q,x_{ij},l) = \exp\Big(\sum_{k,l} \lambda_{kl} c^\mathrm{pair}_{kl}(q,x_{ij},l)\Big)$$ that, when applied to the empirical loss, is theoretically equivalent to training against the true label distribution modulo a distributional correction. The associated dual variables $\lambda_{kl}$ are updated iteratively via mirror-descent, alternating with retraining the model under current sample weights (Sonoda, 2021).

4. Alternative Regularization: Pairwise Correlation Penalization

Pairwise calibration also informs regularization approaches. Beutel et al. (Beutel et al., 2019) define a pairwise-fairness metric: $$\mathrm{PairwiseAccuracy} := P\big( c_q(j,j') = 1\,|\, y_{q,j} > y_{q,j'},\ j, j' \in R_q \big)$$ and propose explicitly regularizing the correlation of ranking success with the group label by minimizing $| \operatorname{Corr}_P(A,B) |$, where

$$A_i = \left[ g(f_\theta(q_i, v_{j_i})) - g(f_\theta(q_i,v_{j_i'}) \right] (y_{j_i} - y_{j_i'}), \quad B_i = (s_{j_i} - s_{j_i'})(y_{j_i} - y_{j_i'})$$

with $P$ a set of matched, randomized item pairs from online or offline experiments.

By including the term $\lambda \cdot R_P(\theta)$ in the loss, where $R_P(\theta) = |\operatorname{Corr}_P(A,B)|$, training explicitly seeks to decorrelate the model’s pairwise ordering errors from group membership (Beutel et al., 2019).

5. Algorithmic Construction and Computational Considerations

The standard procedure for constructing matched pairs for fairness testing or constraint enforcement is as follows:

1. Partition and sort items by group and score.
2. Iteratively scan through two sorted lists (group $g$ vs. not-$g$), appending a pair whenever $0 \leq s(i_{\neg g}) - s(i_g) \leq \epsilon$. Advancing the pointer on the lower scoring item enables efficient enumeration.
3. Complexity: $O(n \log n)$ per query, with $n$ the number of items (Korevaar et al., 2023).

For training interventions, randomized experiments (where pair positions are randomly swapped and outcomes recorded) ensure that groupwise matched pairs are not confounded by position bias or engagement effects (Beutel et al., 2019). In sample weighting or dual-weighted training (Sonoda, 2021), each training iteration involves (1) computing fairness violations, (2) dual gradient or mirror-descent updates for constraint multipliers, and (3) model retraining under updated instance weights.

6. Theoretical Guarantees and Analytical Connections

Matched pair calibration admits several theoretical properties:

• Generalized maximum-entropy: The closed-form for unbiased sample weights in (Sonoda, 2021) derives from maximum-entropy reasoning, ensuring that any solution to the dual-weighted problem can be interpreted as minimizing the KL-divergence to the true label distribution under fairness constraints.
• Unbiasedness: Weighted empirical risk minimization (ERM) is proved equivalent, up to a change in the sample distribution, to direct minimization of the fairness-constrained objective.
• Convergence: Dual updates converge under standard convexity and bounded-gradient assumptions.
• Marginal outcome connection: The matched pair calibration statistic operationalizes Becker's marginal outcome test for ranking: under the null of fair treatment, the average outcome in marginal pairs across groups should be equal. If not, “boosting” the affected group strictly improves the ranking objective (Korevaar et al., 2023).
• Failure of scalar calibration: Even if scores are isotonic-calibrated within group, persistent marginal MPC gaps indicate the inadequacy of scalar post-hoc calibration for ranking fairness.

7. Empirical Evidence and Practical Impact

Matched pair calibration has been empirically validated across domains:

• Datasets: TREC Experts (“W3C”), Engineering Students, MSLR Web Ranking (Sonoda, 2021); MovieLens 20M (Korevaar et al., 2023); large-scale production recommender logs (Beutel et al., 2019).
• Performance: On standard benchmarks, matched pair calibration achieves robust improvement in the fairness–utility trade-off:
  • On TREC, matched pair calibration achieves fairness up to $0.90$ (statistical parity) at AUC $\approx 0.65$, outperforming pointwise reweighting, post-processing, and in-processing Lagrangian baselines on the entire (fairness, AUC) Pareto frontier (Sonoda, 2021).
  • In the MovieLens case paper, boosting group scores directly reduces MPC gaps (bias), with minimal variation in NDCG, indicating no significant loss in utility (Korevaar et al., 2023).
  • In production systems, pairwise regularization reduced inter-group pairwise accuracy gap from $35.6\%$ to $2.6\%$ without harming overall engagement (Beutel et al., 2019).
• Implementation Guidelines: Randomizing exposure, discretizing by engagement level, careful balancing of group representation in matched pairs, and regular cross-group evaluation are recommended in practice. Scalability is achieved via pre-aggregation and stratified sampling (Beutel et al., 2019).

8. Connections to Related Methods

Matched pair calibration bridges several streams in the fairness literature:

Approach	Matching Principle	Notable References
Marginal outcome test	Compare outcomes at classifier or score threshold	Becker (1957); (Korevaar et al., 2023)
Pairwise error parity	Maximize parity in inter-group mis-ranking rates	(Beutel et al., 2019), Beutel et al. (2019)
Global calibration	Fit $E[Y|s] = s$ within/between groups	(Korevaar et al., 2023)

While error-based metrics are more sensitive to mis-rankings across the entire score range, matched pair calibration is focused on outcome parity at the margin of system indifference—thereby targeting the specific locus of mechanical unfairness in ranking settings.

9. Limitations and Ongoing Challenges

Matched pair calibration relies on sufficient overlap in the score distribution between groups; sparse or highly imbalanced contexts can limit its diagnostic fidelity. The requirement of precise score access and outcome observations constrains applicability in privacy-preserving or bandit settings.

A plausible implication is that, while matched pair calibration captures bias at the margin, persistent exposure or error-rate disparities outside this zone may still warrant additional fairness analysis. Empirical evidence suggests that even aggressive scalar calibration procedures fail to ensure marginal fairness, indicating that targeted pairwise interventions are necessary but not singly sufficient for exhaustive fairness control.

---

References:

• (Sonoda, 2021) A Pre-processing Method for Fairness in Ranking
• (Korevaar et al., 2023) Matched Pair Calibration for Ranking Fairness
• (Beutel et al., 2019) Fairness in Recommendation Ranking through Pairwise Comparisons