---
title: 'DualFairVL: Multimodal Fair Medical Diagnosis'
url: https://www.emergentmind.com/topics/dualfairvl
type: topic
---

# DualFairVL: Multimodal Fair Medical Diagnosis

DualFairVL is a multimodal prompt-learning framework for fair medical image classification with vision-language models under sensitive demographic attributes and distribution shift. It is designed to jointly debias and align both vision and text modalities by separating sensitive attributes and target attributes in a parallel dual-branch architecture, constructing approximately orthogonal text anchors through linear projections, guiding cross-attention with those anchors, generating instance-aware visual prompts through a hypernetwork, and regularizing visual features with prototypes. The method is formulated for data \(D=\{(x_i,y_i,a_i)\}_{i=1}^N\), where \(x_i\) is a medical image, \(y_i\) the diagnostic label, and \(a_i\) a sensitive attribute such as sex, race, or skin type, and it is evaluated for both in-distribution and out-of-distribution fairness and accuracy [2508.18886].

## 1. Problem setting and conceptual rationale

DualFairVL addresses fairness in medical diagnosis when a model can exploit correlations between disease labels and protected or nuisance attributes, and when those correlations are exacerbated by scanner, hospital, acquisition-protocol, and population shifts. The paper’s central claim is that fairness in medical vision-language models cannot be reduced to a single-modality debiasing problem. In its formulation, debiasing only text or only vision leaves residual bias in the other modality, and because the model depends on cross-modal alignment, that residual bias induces cross-modal misalignment.

The argument against unimodal debiasing is explicit. Text-only debiasing can remove sensitive components from prompts, but visual features may still entangle disease-relevant information with demographic or acquisition cues. Vision-only debiasing may purify image features, but biased text anchors remain in the shared embedding space and continue to pull alignment toward spurious directions. The paper therefore frames the main failure mode as broken alignment between a partially debiased visual stream and a partially debiased textual stream.

This leads to four stated goals: fairness across demographic groups, robustness under out-of-distribution shifts, cross-modal alignment between debiased image and text representations, and parameter efficiency. A plausible implication is that DualFairVL belongs to a class of methods that treat fairness and robustness as coupled properties of multimodal representation geometry rather than as post hoc thresholding problems.

## 2. Dual-branch architecture and CLIP-based adaptation

DualFairVL is built on a pretrained CLIP backbone with a visual encoder \(\mathcal{V}=\{\mathcal{V}_l\}_{l=1}^{L_v}\), a text encoder \(\mathcal{T}=\{\mathcal{T}_l\}_{l=1}^{L_t}\), and a shared vision-language embedding space. In experiments, the backbone is CLIP ViT-B/16, and the pretrained CLIP is frozen; only prompt-related modules and associated lightweight components are trained [2508.18886].

The predictive core remains CLIP-style. Image and text embeddings are projected into the shared space as \(z_v,z_t\in\mathbb{R}^{d_{vl}}\), and prediction uses temperature-scaled cosine similarity with cross-entropy:
$$
\mathcal{L}_{CE} = -\sum_{i=1}^N \sum_{c=1}^C y_{ic} \log \frac{\exp\left(\langle z_{v,i}, z_{t,c} \rangle / \tau\right)}{\sum_{c'=1}^C \exp\left(\langle z_{v,i}, z_{t,c'} \rangle / \tau\right)}.
$$
The fairness-aware predictor is reformulated as
$$
f_{\mathcal{V}, \mathcal{T}, P}: X \times A \to Y,
$$
where \(P\) denotes trainable prompts.

The architectural novelty lies in a parallel dual-branch decomposition. One branch models the target attribute, denoted \(\mathrm{TA}\), and is intended to capture disease semantics relevant for diagnosis. The second branch models the sensitive attribute, denoted \(\mathrm{SA}\), and explicitly isolates identity-related information such as sex, race, or skin type. The two branches share the frozen CLIP encoders but use branch-specific prompts and branch-specific interactions. This design gives the model an explicit place to store sensitive information rather than allowing it to leak diffusely into a single latent representation.

Prompting occurs in both modalities. Textual prompts are constructed as
$$
\{p_1^t, p_2^t, \ldots, p_K^t, [\text{CLASS}]\}_{c=1}^C,
$$
where \(p^t=\{p_k^t\}_{k=1}^K\) are learnable text vectors. In the visual encoder, each branch introduces layer-aware visual prompts
$$
p_l^{v,b}=\{p_{l,k}^{v,b}\}_{k=1}^K,
$$
for branch \(b\in\{\mathrm{SA},\mathrm{TA}\}\). In the reported implementation, prompt length is \(4\) for both modalities, and the full trainable parameter count is \(3.6\text{M}\).

## 3. Text-guided attribute disentanglement and cross-modal prompt generation

A defining component of DualFairVL is the construction of text anchors that encode sensitive and target concepts separately. The text branch computes branch-specific states
$$
\tilde{W}_l^{\,b}=\mathcal{T}_l(\tilde{W}_{l-1}^{\,b}), \quad 1\le l\le L_t,
$$
and extracts the final embedding from the [EOS] token:
$$
z_t^{\,b}=\mathrm{TextProj}\!\left(w_{L_t}^{\,b}\right), \quad z_t^{\,b}\in\mathbb{R}^{d_{vl}}.
$$
Here \(z_t^{\mathrm{SA}}\) captures sensitive attributes, while \(z_t^{\mathrm{TA}}\) captures disease semantics but may still contain sensitive leakage.

The target text anchor is debiased by ridge-regularized projection away from the sensitive direction. The SA-aligned component of \(z_t^{\mathrm{TA}}\) is estimated as
$$
\hat{S} = z_t^{\mathrm{SA}} \big((z_t^{\mathrm{SA}})^{\top}z_t^{\mathrm{SA}}+\alpha I\big)^{-1} (z_t^{\mathrm{SA}})^{\top} z_t^{\mathrm{TA}},
$$
with \(\alpha=60\). The debiased target embedding is then
$$
\hat{z}_t^{\,\mathrm{TA}} = z_t^{\,\mathrm{TA}} - \hat{S}.
$$
The final text anchors are \(\hat{z}_t^{\mathrm{TA}}\) for the target branch and \(\hat{z}_t^{\mathrm{SA}}=z_t^{\mathrm{SA}}\) for the sensitive branch. The paper emphasizes that these anchors are approximately orthogonal rather than exactly orthogonal: as \(\alpha\to 0\), the projection approaches exact orthogonal projection onto the complement of the sensitive subspace, but the positive \(\alpha\) is used to avoid overly aggressive information removal.

These anchors then guide multimodal interaction. For each branch, an image is split into \(M\) patches and projected to \(E_0\in\mathbb{R}^{M\times d_v}\). Up to layer \(L_v-1\), branch-specific prompts are inserted into the ViT:
$$
\tilde{z}_{l}^{b}=[cls_{l}^{b},\_,\tilde{E}_{l}^{b}] ={\mathcal{V}_{l}([cls_{l-1}^{b},p_{l}^{v,b},\tilde{E}_{l-1}^{b}])}, \quad 1\le l\le L_v-1.
$$
The text anchor conditions cross-attention. After attribute embedding projection, the query vector is \(\mathbf{q}_{c}^{b}\in\mathbb{R}^{d_v}\), and the key comes from the visual sequence \(\mathbf{k}^{b}\in\mathbb{R}^{n\times d_v}\). The branch-specific instance-aware visual prompt at the last layer is computed by
$$
\mathbf{o}_{L_v}^{b,c} = \text{Softmax}\left( \frac{\mathbf{q}_{c}^{b}\mathbf{W}_{q}^{b}(\mathbf{k}^{b}\mathbf{W}_{k}^{b})^{\top}}{\sqrt{d_k}} \right)\mathbf{k}^{b}\mathbf{W}_{v}^{b},
$$
followed by
$$
p_{L_v}^{v,b} = \{\mathrm{LN}(\mathbf{o}_{L_v}^{b,c}+\mathbf{q}_{c}^{b}) \mid 1\le c\le C^b\}.
$$
The resulting prompt is injected into the final visual layer:
$$
[cls_{L_v}^{b},\_,\tilde{E}_{L_v}^{b}] = \mathcal{V}_{L_v}([cls_{L_v-1}^{b},p_{L_v}^{v,b},\tilde{E}_{L_v-1}^{b}]),
$$
and the class token is projected as
$$
z_v^{b}=\mathrm{ImageProj}(cls_{L_v}^{b}), \quad z_v^{b}\in\mathbb{R}^{d_{vl}}.
$$

The paper does not formulate alignment as a separate standalone contrastive penalty beyond the CLIP-style classification objective. Instead, alignment is induced by text-anchor-guided cross-attention, common embedding geometry, and branch-consistent multimodal fusion.

## 4. Hypernetwork, prototype regularization, and optimization objective

DualFairVL’s second major mechanism is a hypernetwork \(H(\cdot)\) that transforms initial visual prompts into branch- and layer-specific refined prompts. Each branch replaces \(\{p_l^{v,b}\}_{l=1}^{L_v}\) with \(\{\hat{p}_l^{v,b}\}_{l=1}^{L_v}\) via
$$
\hat{p}_l^{v,b} = h_l^b(p_l^{v,b}) = U_l^b\left(\text{ReLU}(D_l^b(p_l^{v,b}))\right).
$$
The adapter parameters are generated from a layer-aware attribute embedding \(I_l^b\):
$$
(U_l^b, D_l^b) = H(I_l^b) = (\mathbf{H}^U, \mathbf{H}^D)I_l^b,
$$
with \(\mathbf{H}^D \in \mathbb{R}^{(d_v\times h)\times I}\) and \(\mathbf{H}^U \in \mathbb{R}^{(h\times d_v)\times I}\). The embedding is defined by
$$
I_l^b = \begin{cases}
h_I\left([\mathbf{t}^b; e_l^b(l)]\right), & l < L_v \\
h_I(\hat{\mathbf{t}}^b), & l = L_v
\end{cases},
$$
where \(\mathbf{t}^b \in \mathbb{R}^{I'}\) is the attribute branch embedding, \(e_l^b(\cdot)\) encodes layer-specific information, \(\hat{\mathbf{t}}^b \in \mathbb{R}^{2I'}\) includes instance-aware attributes from the final layer, and \(h_I(\cdot)\) is an MLP with two fully-connected layers and ReLU.

The visual disentanglement stage uses prototypes. For a mini-batch \(\{x_i\}_{i\in\mathcal I}\), branch features are projected by a shared network \(g(\cdot):\mathbb{R}^{d_{vl}}\to\mathbb{R}^{d_p}\), producing
$$
\tilde{r}_i^b = g(z_{v,i}^b), \qquad
r_i^b = \frac{\tilde{r}_i^b}{\|\tilde{r}_i^b\|_2}.
$$
For the subset
$$
\mathcal{R}^b(i)=\{r_j^b \mid y_j=y_i,\ a_j=a_i\},
$$
the embeddings are assumed to lie on a unit hypersphere and to be well modeled by a von Mises–Fisher distribution. The prototype \(\mu_{y_i,a_i}^b\in\mathbb{R}^{d_p}\) is the central direction for samples with label \(y_i\) and attribute \(a_i\) in branch \(b\).

The compactness term is
$$
\mathcal{L}_{com} = -\mathbb{E}_{\substack{b\in B\\ i\in\mathcal I}} \log \frac{\exp\left((r_i^{b})^{\top}\mu_{y_i,a_i}^b/\varphi\right)} {\sum\limits_{b'\in B}\exp\left((r_i^{b})^{\top}\mu_{y_i,a_i}^{b'}/\varphi\right)},
$$
and the separability term is
$$
\mathcal{L}_{sep} = \mathbb{E}_{y\in Y,a\in A} \left( (\mu_{y,a}^{SA})^{\top}\mu_{y,a}^{TA}/\varphi \right).
$$
Prototypes are updated by exponential moving average:
$$
\mu_{y,a}^b = \operatorname{Normalize}\Big(\beta\,\mu_{y,a}^b + (1-\beta)\,\mathds{1}\{y_i=y,a_i=a\}\, r_i^b\Big).
$$
The paper uses \(\beta=0.5\) and \(\lambda=0.1\), and defines
$$
\mathcal{L}_{dis} = \mathcal{L}_{com} + \lambda \mathcal{L}_{sep}.
$$

The complete training objective is
$$
\mathcal{L}_{total} = \mathcal{L}_{CE}^{TA} + \mathcal{L}_{CE}^{SA} + \delta \mathcal{L}_{dis}.
$$
This objective localizes sensitive information to the SA branch, preserves task-discriminative information in the TA branch, and separates the two through prototype regularization. The paper explicitly notes that it does not introduce a separate adversarial fairness loss or a standalone multimodal contrastive alignment loss beyond CLIP-style cross-entropy and guided cross-modal fusion.

## 5. Fairness formulation, benchmarks, and empirical results

DualFairVL evaluates predictive performance with AUC and fairness with DPD and DEOdds. Demographic parity difference is defined as
$$
\text{DPD} = \tau(A_{\max}) - \tau(A_{\min}),
$$
where
$$
\tau(A) = E[h(x)\mid a=A].
$$
Perfect demographic parity corresponds to \(\text{DPD}=0\). For equalized odds, the group-specific rates are
$$
\text{FPR}(A) = P[h(x)=1 \mid a=A, Y=0], \qquad
\text{TPR}(A) = P[h(x)=1 \mid a=A, Y=1],
$$
and DEOdds is defined as the maximum inter-group difference in FPR and TPR. Equalized odds is achieved when \(\text{DEOdds}=0\) [2508.18886].

The evaluation covers eight medical imaging datasets across four modalities: Fitzpatrick17k, HAM10000, BCN20000, MSK, PAPILA, Harvard-GF3300, CheXpert, and MIMIC-CXR. The paper states the four modalities as optical coherence tomography, chest X-ray, fundus photography, and dermoscopy. Five datasets are used for in-distribution evaluation: Fitzpatrick17k, PAPILA, Harvard-GF3300, HAM10000, and CheXpert. Out-of-distribution evaluation includes dermatology domain generalization across HAM10000, BCN20000, and MSK, and chest X-ray transfer from CheXpert to MIMIC-CXR. Sensitive attributes include sex, race, and Fitzpatrick skin type, binarized following prior MedFair evaluation practice.

Reported training details include PyTorch, 4 NVIDIA RTX 2080 Ti GPUs, image size \(224\times 224\), AdamW, weight decay \(1\times 10^{-4}\), batch size \(32\), dataset-specific learning rate between \(10^{-2}\) and \(10^{-3}\), prompt length \(4\) for both modalities, temperature \(\tau=0.1\), attribute embedding dimensions \(I'=128\) projected to \(I=32\), local adapter hidden dimension \(h=24\), and averaging over 3 independent runs.

The reported in-distribution results show consistent gains in the fairness-accuracy tradeoff. On HAM10000 with sex as the sensitive attribute, DualFairVL reports AUC \(94.55\), DEOdds \(16.72\), and DPD \(6.22\). On PAPILA with sex, it reports AUC \(89.22\), DEOdds \(2.43\), and DPD \(5.04\). On Fitzpatrick17k with skin type, it reports AUC \(94.96\), DEOdds \(18.12\), and DPD \(4.56\). On Harvard-GF3300, it reports AUC \(87.41\), DEOdds \(8.42\), and DPD \(5.54\) for sex, and AUC \(86.35\), DEOdds \(7.02\), and DPD \(4.18\) for race. The paper particularly highlights PAPILA, where AUC is improved by \(4.26\%\) over prior methods while DEOdds and DPD are also reduced.

The out-of-distribution results are a major part of the method’s identity. On CheXpert \(\to\) MIMIC-CXR for sex, the source-domain metrics are AUC \(83.00\), DEOdds \(18.28\), and DPD \(0.88\), while the target-domain metrics are AUC \(78.40\), DEOdds \(4.57\), and DPD \(2.03\). For race, the source-domain metrics are AUC \(82.07\), DEOdds \(14.14\), and DPD \(1.74\), and the target-domain metrics are AUC \(78.96\), DEOdds \(3.36\), and DPD \(1.79\). In dermatology single-source domain generalization, HAM10000 \(\to\) BCN20000 yields AUC \(77.32\), DEOdds \(12.74\), and DPD \(2.11\), while HAM10000 \(\to\) MSK yields AUC \(84.21\), DEOdds \(5.60\), and DPD \(1.26\). In multi-source leave-one-out evaluation, the target HAM10000 result is AUC \(85.84\), DEOdds \(16.52\), and DPD \(6.01\); the target BCN20000 result is AUC \(80.22\), DEOdds \(9.45\), and DPD \(1.24\); and the target MSK result is AUC \(86.00\), DEOdds \(4.03\), and DPD \(0.97\).

Across these settings, the paper states that DualFairVL achieves state-of-the-art fairness and accuracy under both in- and out-of-distribution settings, outperforming full fine-tuning and parameter-efficient baselines with only \(3.6\text{M}\) trainable parameters.

## 6. Ablations, relation to adjacent methods, and limitations

The ablation studies isolate four major components: textual debiasing \(Proj(\cdot)\), visual debiasing \(\mathcal{L}_{dis}\), text-guided cross-attention \(Attn(\cdot)\), and the hypernetwork \(H(\cdot)\). The reported conclusion is that only the full combination produces consistent gains across in-distribution and out-of-distribution settings [2508.18886]. Textual debiasing alone gives poor results, supporting the paper’s claim that text-only debiasing is insufficient because visual bias remains and disrupts alignment. Adding text-guided cross-attention improves AUC, but fairness gains remain inconsistent unless the hypernetwork is also present. Adding \(\mathcal{L}_{dis}\) improves both performance and fairness, indicating that visual disentanglement remains necessary even after prompt-based multimodal fusion.

Prompt analyses refine this picture. The paper evaluates prompt lengths \(2,4,8,16\) and reports that prompt length \(4\) gives the best fairness-accuracy tradeoff in both in-distribution and out-of-distribution settings. It also compares layer-aware, instance-aware, and combined prompt-sharing strategies, with the combined strategy performing best overall. The depth analysis further states that layer-aware prompts work best in early to middle layers, instance-aware prompts are most effective at the final layer, and injecting instance-aware prompts too early can hurt learning. DualFairVL therefore uses layer-aware prompts in the 1st–11th layers and instance-aware prompts at the final layer.

Within the broader literature, DualFairVL can be situated among medical multimodal fairness methods that intervene at different levels of the representation pipeline. FairCLIP regularizes distributions of positive image-text similarities via a Sinkhorn-distance objective on Harvard-FairVLMed, thereby treating fairness as a distribution-alignment problem over multimodal similarity structure [2403.19949]. FairLLaVA regularizes pooled multimodal hidden states to be demographic-invariant through mutual-information minimization during LoRA-based fine-tuning of large vision-language assistants [2603.26008]. FVL-FP addresses group fairness in federated visual-language models through cross-layer demographic fair prompting, demographic subspace orthogonal projection, and fairness-aware prompt fusion [2505.01851]. MultiFair addresses multimodal medical classification through dual-level gradient modulation at the modality and demographic-group levels [2510.07328]. Against this background, DualFairVL’s distinctive feature is its integration of text-side sensitive-direction identification, dual-branch sensitive/target decomposition, text-anchor-guided cross-attention, hypernetwork-generated instance-aware visual prompts, and prototype-based visual disentanglement in a single parameter-efficient CLIP adaptation framework.

The paper also clarifies several points that are easy to misconstrue. DualFairVL is not presented as merely text-side debiasing, merely visual prompt tuning, or merely fairness regularization added to CLIP. It is a joint dual-modal alignment framework in which fairness is achieved by disentanglement, anchor projection, and branch-wise alignment. Likewise, the “approximately orthogonal” text anchors should not be read as exact orthogonalization: the positive ridge parameter \(\alpha=60\) explicitly relaxes the projection to avoid erasing task-relevant disease semantics.

The stated limitations are substantial. The framework is currently built on CLIP fine-tuning and has not yet been extended to other VLM backbones. The authors identify future work on more general dual-modal debiasing across foundation models, more complex and composite sensitive attributes, limited or incomplete sensitive-attribute annotations, and further efficiency and scalability for real-time clinical deployment. The evaluation also relies on DPD and DEOdds, which do not capture calibration disparities, subgroup uncertainty, intersectional fairness beyond binarized attributes, or causal fairness concerns. Finally, the method assumes access to sensitive labels during training, and the sensitive attributes and task labels are binarized following MedFair evaluation practice.

Source: https://www.emergentmind.com/topics/dualfairvl