FairLLaVA: Fairness-Aware Fine-Tuning

• FairLLaVA is a fairness-aware, parameter-efficient fine-tuning method for multimodal language models that targets reducing group performance disparities in medical image-conditioned generation.
• It employs a two-stage training pipeline with visual-language alignment followed by fairness-aware adaptation using LoRA adapters and a mutual-information-based regularizer.
• Empirical results on chest radiology and dermoscopy datasets show improved equity-scaled metrics, balancing overall utility with reduced demographic performance gaps.

Searching arXiv for the cited FairLLaVA paper and closely related fairness-aware PEFT / VLM work. FairLLaVA is a fairness-aware parameter-efficient fine-tuning method for large vision-language assistants that targets group performance disparities in image-conditioned medical generation by making hidden representations less predictive of demographic attributes such as age, gender or sex, and race (Bhosale et al., 27 Mar 2026). It is formulated for multimodal LLMs (MLLMs) used in chest radiology report generation from X-rays and dermoscopy visual question answering or diagnosis-oriented answer generation from skin lesion images. The method couples visual instruction tuning with a mutual-information-based regularizer, a lightweight demographic attribute classifier, and LoRA adapters, with the stated goal of reducing inter-group disparity without compromising overall generation quality.

1. Problem setting and fairness target

FairLLaVA addresses group fairness in multimodal LLMs, not fairness in the narrower sense of lexical neutrality or the presence of explicit demographic words. The motivating claim is that modern vision-language assistants can generate uneven-quality outputs across demographic groups even when the outputs do not explicitly mention demographic markers such as sex, race, or age. In the medical setting, the relevant harm is therefore differential clinical performance rather than overtly biased wording.

The paper argues that fairness in medical image-conditioned generation differs from fairness in standard language generation for four stated reasons. First, demographic information may be latent in the image itself, and acquisition pipelines may also correlate with demographics. Second, medical reports rarely include explicit protected terms, so lexical fairness notions are inadequate. Third, open-ended generation complicates fairness evaluation because multiple phrasings may be valid and clinically important omissions matter. Fourth, disparities are safety-critical because lower-quality outputs for one group can mean more omissions, weaker diagnostic descriptions, or degraded downstream decision support.

The protected attributes are dataset-specific. On MIMIC-CXR, the study uses Age, Race, Gender; on PadChest, Age, Gender; and on HAM10000, Age, Gender. The target is group performance parity, defined as reducing the difference in generation quality and clinical correctness across demographic groups.

For a metric $M$ and demographic attribute $a$, with subgroup set $\mathcal{Z}_a$, the per-group score is defined as

$$M_a = \mathbb{E}_{(x,r)\sim D \mid a}[M(\hat r, r)], \quad a \in \mathcal{Z}_a.$$

The fairness gap is then

$\Delta M_a = \max M_a - \min M_a.$

This formalization makes FairLLaVA a performance-parity method: the object of fairness is the gap between the best and worst subgroup performance for a given evaluation metric, rather than demographic parity of wording.

2. Architecture and training pipeline

FairLLaVA is presented as a two-stage training pipeline. In Stage 1: Visual-language alignment, the multimodal projector is fine-tuned so image features align with the frozen LLM, using the standard causal language modeling loss. In Stage 2: Fairness-aware fine-tuning, the vision encoder and LM backbone are frozen, LoRA adapters are inserted into the LM transformer blocks, and three components are trained jointly in alternating fashion: the projector, the LoRA adapters, and a lightweight demographic attribute classifier.

The main FairLLaVA instantiation uses Vicuna-7b-v1.5 as the LLM, BioMedCLIP as the vision encoder, a multimodal projector mapping image embeddings into LM space, LoRA adapters in LM transformer decoder blocks, and a small variational MLP demographic attribute classifier. The paper characterizes the method as architecture-agnostic because it only assumes a vision encoder, a projector into LM token space, and an autoregressive LLM with hidden states. It characterizes it as lightweight because it does not require full fine-tuning of the LLM, uses parameter-efficient LoRA adapters, and adds only a small classifier; on MIMIC-CXR, the extra DAC has about 57K parameters.

The end-to-end procedure in Stage 2 is explicit. For each minibatch, the model performs a forward pass through the image encoder, projector, and LM with LoRA; extracts tapped LM hidden states; pools hidden states into $h(x)$; and, for each attribute, first trains the DAC with $L_{\text{DAC}}$ using stop-gradient on $h$, then freezes the DAC and computes $L_{\text{DIM}}$ while also applying $L_{\text{LM}}$. At inference time, only image and instruction are needed; demographic attributes are not required.

A common misconception addressed implicitly by this design is that fairness interventions in MLLMs must be either post-hoc output filters or full-model retraining. FairLLaVA instead incorporates fairness directly into visual instruction tuning through parameter-efficient adaptation.

3. Technical formulation

The data are triplets

$a$0

where $a$1 is the image, $a$2 is the demographic attribute label or labels, and $a$3 is the target text or report. The vision encoder $a$4 extracts visual features,

$a$5

and a projector $a$6 maps them into LM space,

$a$7

If $a$8 denotes instruction tokens and $a$9, the autoregressive factorization is

$\mathcal{Z}_a$0

with the standard language modeling loss

$\mathcal{Z}_a$1

The fairness mechanism is based on minimizing the mutual information between a demographic attribute $\mathcal{Z}_a$2 and a hidden state $\mathcal{Z}_a$3 from layer $\mathcal{Z}_a$4:

$\mathcal{Z}_a$5

The intuition stated in the paper is that if $\mathcal{Z}_a$6 contains demographic cues, then a classifier can infer $\mathcal{Z}_a$7 from it; minimizing this dependence pushes $\mathcal{Z}_a$8 toward demographic invariance.

Because direct mutual-information computation is intractable, FairLLaVA uses a variational upper-bound surrogate with a lightweight demographic attribute classifier $\mathcal{Z}_a$9:

$$M_a = \mathbb{E}_{(x,r)\sim D \mid a}[M(\hat r, r)], \quad a \in \mathcal{Z}_a.$$0

The DAC itself is trained with

$$M_a = \mathbb{E}_{(x,r)\sim D \mid a}[M(\hat r, r)], \quad a \in \mathcal{Z}_a.$$1

To make the regularizer efficient, hidden states from a selected set of layers $$M_a = \mathbb{E}_{(x,r)\sim D \mid a}[M(\hat r, r)], \quad a \in \mathcal{Z}_a.$$2 are pooled as

$$M_a = \mathbb{E}_{(x,r)\sim D \mid a}[M(\hat r, r)], \quad a \in \mathcal{Z}_a.$$3

The demographic information minimization loss is then

$$M_a = \mathbb{E}_{(x,r)\sim D \mid a}[M(\hat r, r)], \quad a \in \mathcal{Z}_a.$$4

The combined objective is

$$M_a = \mathbb{E}_{(x,r)\sim D \mid a}[M(\hat r, r)], \quad a \in \mathcal{Z}_a.$$5

An important implementation nuance is that optimization is alternating, not simultaneous in the naive sense. The DAC is updated with $$M_a = \mathbb{E}_{(x,r)\sim D \mid a}[M(\hat r, r)], \quad a \in \mathcal{Z}_a.$$6 while stop-gradient is applied to hidden states; then $$M_a = \mathbb{E}_{(x,r)\sim D \mid a}[M(\hat r, r)], \quad a \in \mathcal{Z}_a.$$7 is frozen, and the LoRA parameters and projector are updated using $$M_a = \mathbb{E}_{(x,r)\sim D \mid a}[M(\hat r, r)], \quad a \in \mathcal{Z}_a.$$8 and $$M_a = \mathbb{E}_{(x,r)\sim D \mid a}[M(\hat r, r)], \quad a \in \mathcal{Z}_a.$$9. For multi-attribute training, the per-attribute losses are summed:

$\Delta M_a = \max M_a - \min M_a.$0

The location of the regularizer is central to the method. FairLLaVA applies it to LM hidden states, not to raw image encoder outputs. The ablation compares first-layer, middle-layer, last-layer, and mean-pooled representations, and the main experiments use the middle layer states.

4. Datasets, metrics, and evaluation protocol

The empirical study spans three medical datasets (Bhosale et al., 27 Mar 2026). MIMIC-CXR is a large chest X-ray report generation benchmark using the version from LLaVA-Rad work, with about 400,000 training images and 2,500 test images; age is binned into 0–45, 45–65, 65+, gender is Male, Female, and race uses the top four categories White, Black, Asian, Hispanic, with all others grouped as Others. PadChest uses chest X-rays with ChatGPT-translated English reports and further preprocessing, with about 87,000 training images and 8,000 test images, and focuses on report generation, especially Findings section extraction or generation. HAM10000 contains 10,015 images across seven diagnoses, with an 8000 train split and the remainder test; it uses GPT-4-generated diagnostic descriptions or QA-style supervision following SelfSynthX.

The evaluation includes both standard NLG metrics and clinically grounded metrics. The NLG metrics are BLEU-1, BLEU-4, and ROUGE-L. The clinical metrics are RadGraph-F1, CheXpert-F1, and GREEN. For HAM10000, the study also computes Accuracy after extracting the predicted diagnosis from generated output.

Fairness is assessed through inter-group disparity and through an equity-scaled metric. The disparity metric is the previously defined $\Delta M_a = \max M_a - \min M_a.$1. The equity-scaled score is

$\Delta M_a = \max M_a - \min M_a.$2

where $\Delta M_a = \max M_a - \min M_a.$3 is the overall average metric across all groups for attribute $\Delta M_a = \max M_a - \min M_a.$4. The paper’s argument is that fairness gaps alone can be misleading because a uniformly poor model can have a small gap; the equity-scaled formulation rewards models that are both high-performing overall and low-disparity across groups. The supplement notes a variance caveat: ES metrics can have high variance because they depend on subgroup extremes, small subgroup sample sizes, and uncertainty in both overall performance and gap estimates, so median values and bootstrap confidence intervals are reported in supplementary analysis.

Important reproducibility details include 1 epoch of training on all datasets, 8 NVIDIA RTX A6000 GPUs, approximate training time of 17 hours on MIMIC-CXR and 4 hours on PadChest, a 2-layer MLP DAC, and class-frequency-weighted cross-entropy for the DAC loss. Baseline comparisons include LLaVA, LLaVA-Rad, MedGemma-4B, MedGemma-27B, Qwen2.5-7B, DeepSeek-VL2, CheXagent, and fairness baselines based on Reweighting, Resampling, an Adversarial MLP classifier, and Chen et al. [8].

5. Experimental findings and ablations

The main quantitative message is that FairLLaVA improves fairness and often improves or preserves utility at the same time, especially under the equity-scaled metrics (Bhosale et al., 27 Mar 2026). On MIMIC-CXR, FairLLaVA-All achieves 7 of 12 best ES scores, shows the strongest consistency across text and clinical metrics, and has universal gains on ES-RadGraph-F1. In the direct comparison to Chen et al. [8] on the same setup, the reported equity-scaled CheXpert-F1 is 24.06, 23.85, and 24.13 for Race, Age, and Gender respectively for Chen et al., versus 69.21, 68.70, and 69.38 for FairLLaVA.

The paper distinguishes single-attribute debiasing from joint debiasing. Versions such as FairLLaVA-Race, FairLLaVA-Age, and FairLLaVA-Gender often improve the targeted attribute’s equity-scaled score, but can worsen the others due to demographic correlation and fairness or utility tradeoffs. This is the stated motivation for the joint debiasing version. The supplementary cross-sectional analysis further states that targeted debiasing reduces the intended attribute gap most strongly when confounding from the other attributes is controlled.

On PadChest, FairLLaVA-All achieves the best overall performance on all four reported metrics and leads 5 of 8 equity-scaled cells. It is best on Age ES-BLEU-4 and ES-RadGraph-F1, and on Gender ES-BLEU-1, ES-RadGraph-F1, and ES-GREEN. On HAM10000, FairLLaVA-All leads 5 of 6 equity-scaled metrics, including all Gender ES metrics, best Age ES-Acc, best Age ES-BLEU4, and competitive Age ES-ROUGE-L. The authors use these results to support generalization beyond grayscale chest radiology to RGB dermoscopy.

The ablation studies clarify the design choices. For hidden-state layer choice, the middle layer gives the best fairness-utility balance, reducing fairness gaps in 5/6 cells; mean pooling gives best overall utility but worse fairness; last-layer debiasing helps some specific gap but harms overall text quality. Supplementary Table S8 compares Pretrained DAC then debias with Joint DAC + MI + LM training and reports that joint training is better. For age-focused debiasing on MIMIC-CXR, ES-BLEU-4 improves from 2.48 to 5.47, ES-RadGraph-F1 from 3.50 to 3.75, and ES-GREEN from 2.08 to 2.36. Overall quality also improves: BLEU-1 from 31.77 to 35.28, BLEU-4 from 10.48 to 13.96, RadGraph-F1 from 27.16 to 28.49, and GREEN from 33.06 to 34.12. The supplement reports that varying attribute-specific MI weights and the LM loss weight causes only minor changes in ES metrics and overall performance.

The subgroup-wise analyses are also noteworthy. The paper stresses that larger subgroup size does not automatically imply better performance; for example, in MIMIC-CXR, White is the largest race cohort but is not the best-performing subgroup for several baselines. It also reports both cross-sectional fairness analysis and counterfactual individual-level matching based on matched CheXpert labels and high feature similarity. In the individual-level matched analysis, FairLLaVA reduces counterfactual fairness gaps relative to LLaVA-Rad across race, age, and gender, although the method remains framed as a group-fairness method.

6. Relation to adjacent work, limitations, and scope

FairLLaVA is part of a broader line of work that treats fairness as a property of internal adaptation and representation learning rather than only of data balancing or output filtering. In relation to frequency-based methods such as Reweighting and Resampling, the paper argues that demographic leakage may be embedded in the image and need not track sample counts. In relation to adversarial-classifier methods, it argues that strong pretrained discriminators can induce catastrophic forgetting and instead uses a lightweight DAC trained jointly with alternating optimization. In relation to ranking-based mitigation such as Chen et al. [8], it does not require ranked preference pairs or repeated inference over the whole training set.

Two neighboring arXiv works help situate FairLLaVA technically. FairLoRA studies fairness-driven low-rank adaptation in vision models and adds a regularizer that penalizes variance in average loss across groups, showing that fairness-aware LoRA can often improve worst-group metrics and disparity gaps without full fine-tuning (Sukumaran et al., 2024). FVL-FP addresses group fairness in federated visual-LLMs through cross-layer demographic fair prompting, demographic subspace orthogonal projection, and fair-aware prompt fusion, emphasizing prompt-level adaptation, representation-level projection, and fairness-aware aggregation (Chen et al., 3 May 2025). This suggests a broader pattern: parameter-efficient fairness interventions can be attached at the level of adapters, prompts, or hidden-state geometry rather than by retraining the full multimodal stack.

The scope of FairLLaVA is nevertheless bounded. It requires demographic annotations during training and evaluation; it targets group-level disparities, not a formal individual fairness or causal fairness criterion; and its conclusions depend on report-generation metrics such as RadGraph, GREEN, BLEU, and related measures. The paper also identifies a conceptual risk of suppressing useful signal, since some demographic patterns may correlate with disease prevalence, even though the supplementary prevalence analysis is presented as evidence that strong clinically meaningful subgroup prevalence trends are largely preserved. Empirically, the study is limited to two chest X-ray generation datasets and one dermoscopy dataset, so broader generalization to other modalities, hospitals, demographic taxonomies, or deployment settings remains open.

For deployment, the paper is explicit that improved equity-scaled metrics do not imply clinical safety. A plausible implication is that FairLLaVA is best understood as a representation-level mitigation strategy for medical MLLMs that reduces measured inter-group disparities while preserving clinical generation quality, but not as a substitute for external validation, subgroup auditing, or clinician oversight.