FairCLIP+: Multi-Attribute Fairness

• The paper introduces a novel multi-attribute fairness regularization approach by extending FairCLIP with a Sinkhorn distance-based loss to align image-text similarity distributions.
• It employs techniques like Optuna hyperparameter tuning and evaluations on datasets such as Harvard-FairVLMed and FairFace to assess fairness metrics including DPD, DEOdds, and AUC.
• Despite reduced distributional disparities, empirical results indicate that minimized Sinkhorn distances do not consistently improve downstream fairness or predictive performance, highlighting critical limitations.

FairCLIP+ is a generalization and extension of the FairCLIP regularization approach for improving group fairness in vision-LLMs such as CLIP. The focus of FairCLIP+ is on reducing disparities in model outputs across multiple sensitive attributes by aligning the distributions of image-text similarity scores. This framework arises from reproducibility studies (Bakker et al., 8 Sep 2025) that analyzed FairCLIP’s mathematical objectives, official codebase, and performance impact. While FairCLIP and FairCLIP+ introduce principled distance-based fairness regularization, empirical results show that minimization of the distance metric (Sinkhorn distance between group-wise and overall distributions) does not consistently improve downstream fairness or performance.

1. Formulation and Objective Function

FairCLIP+ extends FairCLIP’s fairness objective to encompass multiple sensitive attributes simultaneously, instead of only one (such as race or gender). The key regularization term added to the CLIP training loss is:

$$\mathcal{L}_{Fair+} = \sum_{\mathcal{A} \in \mathbb{A}} w_{\mathcal{A}} \sum_{\alpha \in \mathcal{A}} d(\mathcal{D}_{I,T,a}, \mathcal{D}_{I,T,a \mid a=\alpha})$$

where:

• $\mathbb{A}$ is the set of sensitive attributes considered (e.g., race, gender, ethnicity, language),
• each $\mathcal{A}$ is a sensitive attribute (with group levels $\alpha$),
• $w_{\mathcal{A}}$ is the nonnegative weight assigned to attribute $\mathcal{A}$ ($\sum_{\mathcal{A}} w_{\mathcal{A}} = 1$),
• $d(\cdot, \cdot)$ is the distance function, instantiated as Sinkhorn distance,
• $\mathcal{D}_{I,T,a}$ denotes the batch’s self-similarity score distribution (diagonal of image-text similarity matrix $M$),
• $\mathcal{D}_{I,T,a \mid a=\alpha}$ denotes the score distribution within sensitive group $\alpha$.

This loss is combined with the standard CLIP training objective and re-tuned using techniques such as Optuna-based hyperparameter search to examine effects on fairness and predictive performance.

2. Sinkhorn Distance Regularization

FairCLIP+ utilizes Sinkhorn distance to compare distributions from different sensitive groups versus the overall batch. The general Sinkhorn distance formula used is:

$$d(\mathcal{D}_B,\mathcal{D}_{B_\alpha}) = \inf_{z} \Big[ \mathbb{E}_{(x,y)\sim z}[c(x,y)] + \varepsilon H(z ~|~ \mathcal{D}_B \otimes \mathcal{D}_{B_\alpha}) \Big]$$

where:

• $c(x,y)$ is a transport cost function between scores $x$, $y$,
• $\varepsilon$ is a regularization parameter,
• $H$ is the relative entropy,
• $z$ is a coupling/distribution with marginals matching $\mathcal{D}_B, \mathcal{D}_{B_\alpha}$.

Minimizing these distances serves, in theory, to bring the output distributions of each group closer to that of the overall population. The regularization rate $\lambda$ balances this loss against the original objective.

3. Implementation Variants: FairCLIP, A-FairCLIP, and FairCLIP+

The official FairCLIP codebase and its mathematical description diverge on key computational details, such as normalization of similarity scores and which matrix diagonals are selected. A-FairCLIP is introduced as an implementation aligned with the theoretical formulation, correcting for such discrepancies and retuning hyperparameters. Both FairCLIP and A-FairCLIP are tested alongside FairCLIP+ to evaluate effects on fairness and performance.

The extension to FairCLIP+ avoids collapsing multiple demographic attributes into a single group (which can introduce data imbalances), instead computing a weighted sum of losses across all attributes.

4. Experimental Setup and Metrics

FairCLIP+’s effects are assessed on two datasets:

• Harvard-FairVLMed: SLO fundus images, clinical notes, and demographic attributes (Asian, Black, White; Female, Male; Hispanic, Non-Hispanic; English, Spanish, Others).
• FairFace: Balanced face data for group fairness evaluation.

Evaluations include:

• Zero-shot classification (e.g., glaucoma diagnosis): Similarity scores between image and prompts such as “photo of glaucoma” vs. “photo of non-glaucoma” are computed, then used for classification.
• Fairness metrics: Demographic Parity Distance (DPD), Difference in Equalized Odds (DEOdds), Area Under Curve (AUC), Equity-Scaled AUC (ES-AUC).
• Distributional metrics: Sinkhorn distance and Maximum Mean Discrepancy (MMD) between group and batch distributions.

5. Empirical Findings and Interpretations

The experiments show that FairCLIP+ and A-FairCLIP reliably reduce the Sinkhorn distances between subgroups and the overall population:

• Regularized similarity distributions become numerically more similar across groups.

However, these reductions in distributional distance do not consistently translate into improved fairness or predictive performance:

• DPD and DEOdds metrics often remain unchanged or are only marginally better.
• AUC and ES-AUC scores do not show notable improvement over standard CLIP fine-tuning.
• In some configurations, plain CLIP fine-tuning matches or exceeds FairCLIP+ in both fairness and performance.

This suggests that minimizing disparities in similarity distributions (via Sinkhorn distance) is not sufficient on its own to guarantee downstream improvements on standard fairness metrics or classification accuracy. A plausible implication is that alternative regularization strategies, cross-modal supervision, or rebalancing methods may be required to achieve substantive fairness improvements.

6. Limitations and Practical Considerations

FairCLIP+ showcases the technical feasibility of multi-attribute fairness regularization for vision-LLMs, but empirical outcomes are nuanced:

• The theoretical alignment between distribution regularization and fairness outcomes is weak; reducing group distribution distances may not move fairness metrics or predictive power proportionally.
• Subtle implementation details (e.g., how batches and diagonals are selected, normalization practices) have tangible impact on reproducibility.
• The approach does not address potential issues with sample imbalance, semantic drift in attributes, or multi-modal representations beyond strictly similarity scores.

Researchers are advised to treat Sinkhorn-based fairness regularizers as one tool among many, rather than as an all-encompassing solution for model debiasing.

7. Mathematical Summary

The core formulas from the FairCLIP+ framework are:

• FairCLIP+ regularization loss:

$$\mathcal{L}_{Fair+} = \sum_{\mathcal{A} \in \mathbb{A}} w_{\mathcal{A}} \sum_{\alpha \in \mathcal{A}} d(\mathcal{D}_{I,T,a}, \mathcal{D}_{I,T,a \mid a=\alpha})$$

• Sinkhorn distance:

$$d(\mathcal{D}_B, \mathcal{D}_{B_\alpha}) = \inf_{z} \left[ \mathbb{E}_{(x,y) \sim z} [c(x,y)] + \varepsilon H(z | \mathcal{D}_B \otimes \mathcal{D}_{B_\alpha}) \right]$$

These encode the central principle: promote fairness by regularizing across group distance metrics, though efficacy is context-dependent.

---

In sum, FairCLIP+ provides a rigorous path for regularizing vision-LLMs across multiple sensitive attributes by minimizing group-wise disparities in similarity score distributions. However, current empirical evidence (Bakker et al., 8 Sep 2025) shows that this strategy is not inherently sufficient for achieving group fairness in zero-shot diagnostic tasks, motivating further research into alternative fairness-enhancing mechanisms and more robust evaluation pipelines.