---
title: 'FairStacks: Fair Model Stacking'
url: https://www.emergentmind.com/topics/fairstacks
type: topic
---

# FairStacks: Fair Model Stacking

FairStacks is a convex “fair model stacking” meta-learner designed to expand and optimize the empirical Pareto frontier of the fairness–accuracy tradeoff in machine learning, enabling practitioners to maximize accuracy for a prescribed level of group fairness. By leveraging linear combinations of pre-trained base classifiers and enforcing a strict score-bias constraint, FairStacks consistently improves both the best-achievable fairness and accuracy on benchmark tasks, providing a model-agnostic and post-processing approach to algorithmic fairness [2206.00074].

## 1. Formalization of the Fairness–Accuracy Pareto Frontier

Let $H$ denote any finite collection of pre-trained predictors $h \in H$, where each $h$ maps features $x$ to a probability score $\hat h(x) \in [0,1]$ or a hard label $\{0,1\}$. Each model is evaluated by:

- an accuracy measure $\mathrm{Accuracy}(h) \in [0,1]$, increasing in “better” fits (e.g., classification accuracy, $1-\mathrm{MSE}$, etc.)
- a group-fairness measure $\mathrm{Fairness}(h)\in[0,1]$, increasing in “more fair” behavior (e.g., $\mathrm{Fairness}=1-|\text{Demographic Parity gap}|$)

**Pareto Optimal Model:** A model $h^* \in H$ is Pareto-optimal if there is no $h \in H$ with $\mathrm{Accuracy}(h) \geq \mathrm{Accuracy}(h^*)$ and $\mathrm{Fairness}(h) \geq \mathrm{Fairness}(h^*)$, with at least one inequality strict.

**TAF Curve:** The Tradeoff-between-Fairness-and-Accuracy curve for $H$ is $\mathrm{taf}_H : [0,1] \rightarrow [0,1]$, where
$$
\mathrm{taf}_H(f) \equiv \max \{ \mathrm{Accuracy}(h) : h\in H,\, \mathrm{Fairness}(h)\geq f \}
$$
$\mathrm{taf}_H(f)$ reports the best achievable accuracy by any model in $H$ meeting at least fairness level $f$. Practically, this is computed by sorting $H$ in decreasing fairness and tracking the running maximum of accuracy.

**FAUC:** For a nonnegative weight function $w(f)$, the weighted Fairness-Area-Under-the-Curve is
$$
\mathrm{fauc}^w(H) = \frac{\int_0^1 \mathrm{taf}_H(f) w(f) \mathrm{d}f}{\int_0^1 w(f) \mathrm{d}f}
$$
Unweighted ($w\equiv1$), $\mathrm{fauc}(H)$ is the ordinary area under the TAF curve. This enables direct, model-agnostic comparison of model families on fairness and accuracy, even when their tradeoff curves cross.

## 2. Score Bias and Group Fairness Constraint

FairStacks ensembles are constructed as linear combinations of base models $\{h_i\}_{i=1}^k$. For two contrast groups $G_1, G_2$ (e.g., $A=1$ vs $A=0$ for a binary attribute):

- **Score Bias:** For weight vector $w \in \mathbb{R}^k$ and score matrix $H(x)\in\mathbb{R}^k$,
  $$
  \widetilde{\mathrm{Bias}}(w^T H(\cdot)) = \mathbb{E}_{i\in G_1}[w^T H(x_i)] - \mathbb{E}_{i\in G_2}[w^T H(x_i)]
  $$
- For each base model $h_i$, $\widetilde{\mathrm{Bias}}(h_i)\equiv \mathbb{E}_{G_1}[h_i(x)] - \mathbb{E}_{G_2}[h_i(x)]$. By linearity,
  $$
  \widetilde{\mathrm{Bias}}(w^T H) = \sum_{i} w_i\,\widetilde{\mathrm{Bias}}(h_i)
  $$

The ensemble bias constraint directly encodes any group fairness constraint definable as a difference in expectation between groups.

## 3. FairStacks Convex Optimization Framework

FairStacks solves the constrained optimization:

$$
\begin{aligned}
  &\min_{w\in \mathbb{R}^k} &&\sum_{j=1}^n \mathcal{L}(w^T H(x_j),\, y_j) + \frac{\alpha}{2} \|w\|_2^2 \\
  &\text{subject to} && \left| \sum_{i=1}^k w_i\,\widetilde{\mathrm{Bias}}(h_i)\right| \leq \tau
\end{aligned} \tag{FS}
$$

where:
- $x_j, y_j$ are training features and labels,
- $H(x_j) = (h_1(x_j), \ldots, h_k(x_j)) \in \mathbb{R}^k$ is the vector of base-model scores,
- $\mathcal{L}(\eta, y)$ is a convex surrogate loss (e.g., squared error, binomial deviance),
- $\alpha \geq 0$ is a ridge penalty for regularization,
- $\tau \geq 0$ is a bias-budget parameter (smaller $\tau$ enforces stricter fairness).

Alternatively, a penalized form may be solved:
$$
\min_w \sum_j \mathcal{L}(w^T H(x_j), y_j) + \lambda^2\left(\sum_i w_i\,\widetilde{\mathrm{Bias}}(h_i)\right)^2 + \frac{\alpha}{2}\|w\|_2^2
$$
where $\lambda$ is varied across a grid to trace the fairness–accuracy frontier.

## 4. Algorithmic Solution and Implementation

The FairStacks algorithm proceeds as follows:

1. **Inputs:** Pre-trained base models $\{h_i\}_{i=1}^k$; training data $\{(x_j, y_j)\}_{j=1}^n$; protected groups $G_1, G_2$.
2. **Precompute:**
   - $H_{ji}=h_i(x_j)$ for all $j, i$.
   - $b_i = \widetilde{\mathrm{Bias}}(h_i) = \text{mean}_{j\in G_1} H_{ji} - \text{mean}_{j\in G_2} H_{ji}$.
3. **Grid search:** Select grid $\{\lambda_1,\lambda_2,...\}$ for bias penalty; grid for $\alpha$ (via cross-validation).
4. **For each $\lambda$:** Solve convex program for weights $w$ (via Newton or QP solvers):
   $$
   \min_w \sum_j \mathcal{L}(w^T H_j, y_j) + \lambda^2 \left( \sum_i w_i b_i \right)^2 + (\alpha/2)\|w\|^2
   $$
   - Compute fairness $f(\lambda) = 1 - |\sum_i w_i b_i|$; accuracy $a(\lambda) = 1 - \text{EmpiricalRisk}(w)$ or direct $\mathrm{Accuracy}(w^T H)$.
   - Accumulate $(f(\lambda), a(\lambda))$ pairs to form taf curve and compute area under it for fauc.

5. **Output:** $\mathrm{taf}_{\mathrm{FairStacks}}(f)$, $\mathrm{fauc}^w(\mathrm{FairStacks})$, ensemble weights $\{w(\lambda)\}$.

## 5. Theoretical Guarantees

FairStacks provides several formal properties:
- **Frontier Expansion (Proposition 1):** For original models $H$ and FairStacks solutions $FS(H) = \{w(\tau) : \tau \geq 0\}$,
  $$
  \mathrm{taf}_H(f) \leq \mathrm{taf}_{FS(H)}(f), \qquad \mathrm{fauc}^w(H) \leq \mathrm{fauc}^w(FS(H))
  $$
  Since all $h_i \in H$ can be realized as stacking solutions with $w=e_i$, and the stacking set is a superset of $H$, stacking always expands the Pareto frontier.

- **Monotonicity in Fairness (Theorem 2):** Under mild assumptions on the decision function $\delta(\eta)$ (e.g., thresholding), the decision-level fairness of weights $w(\tau)$ improves (bias decreases) as $\tau \to 0$, so the constraint yields (approximately) monotonic increases in group fairness.

## 6. Empirical Evaluation

Performance of FairStacks was evaluated on five benchmark datasets:

| Dataset                   | Size / Features                 | Protected Attributes        |
|---------------------------|---------------------------------|----------------------------|
| Adult Income              | $n \approx 48$k, $p \approx 100$| gender, race               |
| Bank Marketing            | $n \approx 45$k, $p \approx 58$ | age $\geq 40$?             |
| COMPAS Recidivism         | $n \approx 7$k, $p \approx 13$  | race, gender               |
| Default of Credit Card    | $n \approx 30$k, $p \approx 23$ | gender                     |
| Communities & Crime       | $n \approx 2$k, $p \approx 100$ | race                       |

Base learners included standard random-forest trees, “minipatch” trees (1,000 sub-sample trees), Scikit-learn/XGBoost classifiers, and three in-processing fairness methods: Adversarial Debiasing [Zhang et al. 2018], Reduction-Based Fair Classification [Agarwal et al. 2018], and Fair Adversarial GBT [Grari et al. 2019]. Data were split $50\%$ train / $25\%$ stacking-CV / $25\%$ test; 5-fold CV for $(\lambda, \alpha)$ was performed within the stacking split to optimize the $80\%$-step-weighted fauc. Brier score was used for $\mathcal{L}$ to emphasize calibrated probabilities.

Selected results (FAUC, Demographic Parity, mean $\pm$ SE, 10 random splits):

| Method                  | Adult (gender) | COMPAS (race) | Default (gender) | C&C (race)   |
|-------------------------|----------------|---------------|------------------|-------------|
| Random Forest           | $0.772 \pm 0.001$  | $0.524 \pm 0.003$ | $0.848 \pm 0.001$     | $0.623 \pm 0.014$   |
| Zhang et al. (2018)     | $0.836 \pm 0.020$  | $0.629 \pm 0.018$ | $0.778 \pm 0.002$     | —       |
| Grari et al. (2019)     | $0.830 \pm 0.004$  | $0.641 \pm 0.009$ | $0.850 \pm 0.003$     | —       |
| Agarwal et al. (2018)   | $0.797 \pm 0.003$  | $0.622 \pm 0.002$ | $0.778 \pm 0.014$     | —       |
| FS on minipatch trees   | $0.851 \pm 0.001$  | $0.699 \pm 0.005$ | $0.856 \pm 0.002$     | $0.714 \pm 0.004$   |
| FS on RF trees          | $0.843 \pm 0.001$  | $0.773 \pm 0.004$ | $0.921 \pm 0.002$     | $0.695 \pm 0.003$   |
| FS on all classifiers   | $0.823 \pm 0.002$  | $0.735 \pm 0.005$ | $0.863 \pm 0.002$     | $0.689 \pm 0.002$   |
| FS Kitchen-Sink (all)   | $0.866 \pm 0.001$  | $0.796 \pm 0.002$ | $0.933 \pm 0.001$     | $0.734 \pm 0.002$   |

In all settings, FairStacks ensembles dominate competing methods—their taf curves are strictly above all baselines for every fairness level, yielding the highest area (FAUC). The “kitchen-sink” ensemble, pooling all base learners, consistently achieves the best fairness–accuracy balance.

## 7. Relevance and Context

FairStacks is a simple, entirely model-agnostic, post-processing meta-learner. It can accept any collection of pre-fitted predictors and, by solving a single convex program per fairness budget $\tau$, efficiently traces and expands the empirical Pareto frontier between fairness and accuracy. Theoretical guarantees and empirical results demonstrate pointwise expansion of the frontier and consistently higher FAUC on diverse real-world datasets, providing practitioners and researchers with an effective plug-and-play approach to optimize the fairness–accuracy tradeoff [2206.00074].

Source: https://www.emergentmind.com/topics/fairstacks