---
title: 'FairCLIP+: Multi-Attribute Fairness'
url: https://www.emergentmind.com/topics/fairclip-2b5f1e30-8bb7-4e40-8839-106b5c11f3a5
type: topic
---

# FairCLIP+: Multi-Attribute Fairness

FairCLIP+ is a generalization and extension of the FairCLIP regularization approach for improving group fairness in vision-language models such as CLIP. The focus of FairCLIP+ is on reducing disparities in model outputs across multiple sensitive attributes by aligning the distributions of image-text similarity scores. This framework arises from reproducibility studies [2509.06535] that analyzed FairCLIP’s mathematical objectives, official codebase, and performance impact. While FairCLIP and FairCLIP+ introduce principled distance-based fairness regularization, empirical results show that minimization of the distance metric (Sinkhorn distance between group-wise and overall distributions) does not consistently improve downstream fairness or performance.

## 1. Formulation and Objective Function

FairCLIP+ extends FairCLIP’s fairness objective to encompass multiple sensitive attributes simultaneously, instead of only one (such as race or gender). The key regularization term added to the CLIP training loss is:

$$
\mathcal{L}_{Fair+} = \sum_{\mathcal{A} \in \mathbb{A}} w_{\mathcal{A}} \sum_{\alpha \in \mathcal{A}} d(\mathcal{D}_{I,T,a}, \mathcal{D}_{I,T,a \mid a=\alpha})
$$

where:
- $\mathbb{A}$ is the set of sensitive attributes considered (e.g., race, gender, ethnicity, language),
- each $\mathcal{A}$ is a sensitive attribute (with group levels $\alpha$),
- $w_{\mathcal{A}}$ is the nonnegative weight assigned to attribute $\mathcal{A}$ ($\sum_{\mathcal{A}} w_{\mathcal{A}} = 1$),
- $d(\cdot, \cdot)$ is the distance function, instantiated as Sinkhorn distance,
- $\mathcal{D}_{I,T,a}$ denotes the batch’s self-similarity score distribution (diagonal of image-text similarity matrix $M$),
- $\mathcal{D}_{I,T,a \mid a=\alpha}$ denotes the score distribution within sensitive group $\alpha$.

This loss is combined with the standard CLIP training objective and re-tuned using techniques such as Optuna-based hyperparameter search to examine effects on fairness and predictive performance.

## 2. Sinkhorn Distance Regularization

FairCLIP+ utilizes Sinkhorn distance to compare distributions from different sensitive groups versus the overall batch. The general Sinkhorn distance formula used is:

$$
d(\mathcal{D}_B,\mathcal{D}_{B_\alpha}) = \inf_{z} \Big[ \mathbb{E}_{(x,y)\sim z}[c(x,y)] + \varepsilon H(z ~|~ \mathcal{D}_B \otimes \mathcal{D}_{B_\alpha}) \Big]
$$

where:
- $c(x,y)$ is a transport cost function between scores $x$, $y$,
- $\varepsilon$ is a regularization parameter,
- $H$ is the relative entropy,
- $z$ is a coupling/distribution with marginals matching $\mathcal{D}_B, \mathcal{D}_{B_\alpha}$.

Minimizing these distances serves, in theory, to bring the output distributions of each group closer to that of the overall population. The regularization rate $\lambda$ balances this loss against the original objective.

## 3. Implementation Variants: FairCLIP, A-FairCLIP, and FairCLIP+

The official FairCLIP codebase and its mathematical description diverge on key computational details, such as normalization of similarity scores and which matrix diagonals are selected. A-FairCLIP is introduced as an implementation aligned with the theoretical formulation, correcting for such discrepancies and retuning hyperparameters. Both FairCLIP and A-FairCLIP are tested alongside FairCLIP+ to evaluate effects on fairness and performance.

The extension to FairCLIP+ avoids collapsing multiple demographic attributes into a single group (which can introduce data imbalances), instead computing a weighted sum of losses across all attributes.

## 4. Experimental Setup and Metrics

FairCLIP+’s effects are assessed on two datasets:
- **Harvard-FairVLMed**: SLO fundus images, clinical notes, and demographic attributes (Asian, Black, White; Female, Male; Hispanic, Non-Hispanic; English, Spanish, Others).
- **FairFace**: Balanced face data for group fairness evaluation.

Evaluations include:
- **Zero-shot classification** (e.g., glaucoma diagnosis): Similarity scores between image and prompts such as “photo of glaucoma” vs. “photo of non-glaucoma” are computed, then used for classification.
- **Fairness metrics**: Demographic Parity Distance (DPD), Difference in Equalized Odds (DEOdds), Area Under Curve (AUC), Equity-Scaled AUC (ES-AUC).
- **Distributional metrics**: Sinkhorn distance and Maximum Mean Discrepancy (MMD) between group and batch distributions.

## 5. Empirical Findings and Interpretations

The experiments show that FairCLIP+ and A-FairCLIP reliably reduce the Sinkhorn distances between subgroups and the overall population:
- Regularized similarity distributions become numerically more similar across groups.

However, these reductions in distributional distance do **not consistently translate into improved fairness or predictive performance**:
- DPD and DEOdds metrics often remain unchanged or are only marginally better.
- AUC and ES-AUC scores do not show notable improvement over standard CLIP fine-tuning.
- In some configurations, plain CLIP fine-tuning matches or exceeds FairCLIP+ in both fairness and performance.

This suggests that minimizing disparities in similarity distributions (via Sinkhorn distance) is **not sufficient on its own** to guarantee downstream improvements on standard fairness metrics or classification accuracy. *A plausible implication is that alternative regularization strategies, cross-modal supervision, or rebalancing methods may be required to achieve substantive fairness improvements.*

## 6. Limitations and Practical Considerations

FairCLIP+ showcases the technical feasibility of multi-attribute fairness regularization for vision-language models, but empirical outcomes are nuanced:
- The theoretical alignment between distribution regularization and fairness outcomes is weak; reducing group distribution distances may not move fairness metrics or predictive power proportionally.
- Subtle implementation details (e.g., how batches and diagonals are selected, normalization practices) have tangible impact on reproducibility.
- The approach does not address potential issues with sample imbalance, semantic drift in attributes, or multi-modal representations beyond strictly similarity scores.

Researchers are advised to treat Sinkhorn-based fairness regularizers as one tool among many, rather than as an all-encompassing solution for model debiasing.

## 7. Mathematical Summary

The core formulas from the FairCLIP+ framework are:

- FairCLIP+ regularization loss:
  $$
  \mathcal{L}_{Fair+} = \sum_{\mathcal{A} \in \mathbb{A}} w_{\mathcal{A}} \sum_{\alpha \in \mathcal{A}} d(\mathcal{D}_{I,T,a}, \mathcal{D}_{I,T,a \mid a=\alpha})
  $$
- Sinkhorn distance:
  $$
  d(\mathcal{D}_B, \mathcal{D}_{B_\alpha}) = \inf_{z} \left[ \mathbb{E}_{(x,y) \sim z} [c(x,y)] + \varepsilon H(z | \mathcal{D}_B \otimes \mathcal{D}_{B_\alpha}) \right]
  $$

These encode the central principle: promote fairness by regularizing across group distance metrics, though efficacy is context-dependent.

---

In sum, FairCLIP+ provides a rigorous path for regularizing vision-language models across multiple sensitive attributes by minimizing group-wise disparities in similarity score distributions. However, current empirical evidence [2509.06535] shows that this strategy is not inherently sufficient for achieving group fairness in zero-shot diagnostic tasks, motivating further research into alternative fairness-enhancing mechanisms and more robust evaluation pipelines.

Source: https://www.emergentmind.com/topics/fairclip-2b5f1e30-8bb7-4e40-8839-106b5c11f3a5