---
title: Differentiable MaxAccGap Loss
url: https://www.emergentmind.com/topics/differentiable-maxaccgap-loss
type: topic
---

# Differentiable MaxAccGap Loss

The Differentiable MaxAccGap loss is a fairness-driven regularization technique designed to minimize the diagnostic performance disparity across sensitive demographic groups by explicitly penalizing the maximum difference in model accuracy between groups in a differentiable manner. Introduced in the context of fairness-aware fine-tuning of vision-language models (VLMs) for medical glaucoma diagnosis, this loss facilitates end-to-end optimization for accuracy parity, making it directly compatible with stochastic gradient descent and modern deep learning frameworks. Its implementation is central to three Low-Rank Adaptation (LoRA)–based methods (FR-LoRA, GR-LoRA, and Hybrid-LoRA) that address fairness and data imbalance in large-scale medical AI settings while maintaining parameter efficiency [2512.03477].

## 1. Mathematical Formulation of Differentiable MaxAccGap Loss

Let $D = \{(x_i, y_i, s_i)\}_{i=1}^N$ denote the dataset, where $y_i \in \{0, 1\}$ is the binary glaucoma label, and $s_i \in S = \{s^{(1)}, \dots, s^{(|S|)}\}$ is a sensitive attribute (e.g., ethnicity). For each group $s$, $D_s$ is the subset of samples with $s_i = s$ and size $N_s$.

**Group-wise hard accuracy**:
\[
\mathrm{Acc}_s(\theta) = \frac{1}{N_s} \sum_{i:s_i=s} \mathbf{1}\{\widehat{y}_i = y_i\}, \quad \widehat{y}_i = \arg\max_k z_k
\]

**MaxAccGap**:
\[
\mathrm{MaxAccGap}(\theta) = \max_{s \in S} \mathrm{Acc}_s(\theta) - \min_{s \in S} \mathrm{Acc}_s(\theta)
\tag{1}
\]

Since this metric is non-differentiable, a soft (differentiable) surrogate replaces the indicator with $p_\theta(y \mid x)$, the model's predicted probability for the true class.

**Soft-accuracy for group $s$**:
\[
\mathrm{Acc}_s^{\mathrm{soft}}(\theta) = \frac{1}{N_s} \sum_{i:s_i=s} p_\theta(y_i \mid x_i)
\]

**Differentiable MaxAccGap**:
\[
\mathrm{MaxAccGap}^{\mathrm{soft}}(\theta) = \max_{s \in S} \mathrm{Acc}_s^{\mathrm{soft}}(\theta) - \min_{s \in S} \mathrm{Acc}_s^{\mathrm{soft}}(\theta)
\tag{2}
\]

This structure is piecewise-differentiable, with subgradients targeting the group-wise soft-accuracy extremes:
\[
\nabla_\theta \mathrm{MaxAccGap}^{\mathrm{soft}}
= \nabla_\theta \mathrm{Acc}_{s_+}^{\mathrm{soft}} - \nabla_\theta \mathrm{Acc}_{s_-}^{\mathrm{soft}}
\]
where $s_+, s_-$ attain the maximal and minimal group soft-accuracy, respectively.

## 2. Integration with LoRA-based Fairness Methods

Three algorithmic variants leverage the Differentiable MaxAccGap for fairness-aware adaptation:

- **FR-LoRA:** Directly adds the soft MaxAccGap as a regularization term to the cross-entropy objective:
  \[
  \mathcal{L}_{\mathrm{FR}}(\theta) = \mathcal{L}_{\mathrm{vanilla}}(\theta) + \lambda \, \mathrm{MaxAccGap}^{\mathrm{soft}}(\theta)
  \tag{3}
  \]
  Here, $\lambda$ tunes the accuracy-fairness trade-off.

- **GR-LoRA:** Applies inverse frequency reweighting for group cross-entropy, mitigating sample-size imbalance:
  \[
  \mathcal{L}_{\mathrm{GR}}(\theta) = \sum_{s\in S} w_s \mathcal{L}^s_{\mathrm{CE}}(\theta)
  \]
  with $w_s = \min\left(\frac{N}{N_s}, w_\text{max}\right)$. MaxAccGap is not an explicit loss term but becomes minimized through gradient balancing.

- **Hybrid-LoRA:** Combines both objectives:
  \[
  \mathcal{L}_{\mathrm{Hybrid}}(\theta) = \sum_{s \in S} w_s \mathcal{L}_{\mathrm{CE}}^s(\theta) + \lambda\,\mathrm{MaxAccGap}^{\mathrm{soft}}(\theta)
  \tag{4}
  \]

For all methods, only 0.24% of model parameters (via LoRA adapters) are fine-tuned, allowing parameter-efficient deployment in restricted clinical settings.

## 3. Implementation and Optimization Procedures

Efficient group-wise accumulation is executed in each mini-batch (approximately 8 samples per batch), using indexed tensors for accumulating $p_\theta(y|x)$ per group to estimate soft-accuracies. To prevent instability due to minority group over-sampling, group loss weights $w_s$ are capped at $w_{\max} = 10$. Gradient accumulation across four steps is adopted to ensure multiple-group representation for each update.

Stabilization procedures include linear learning-rate warmup for 100 steps, LoRA adapter dropout (0.05), and last-token pooling for VLMs. All LoRA adapters use rank $r = 32$ and scaling $\alpha = 64$.

The primary hyperparameters and their effects are outlined here:

| λ       | MaxAccGap Reduction         | Overall Accuracy Change |
|---------|----------------------------|------------------------|
| 0.1     | 45% reduction (3.80→2.10)  | +0.25pp                |
| 0.5     | Over-correction (gap↑ to 6.04) | +0.05pp                |
| 1.0     | 47% reduction (3.80→2.01)  | −0.15pp                |

$w_{\max}=10$ is empirically optimal; larger values induce training spikes, lower fail to adequately represent minority groups.

## 4. Empirical Evaluation and Quantitative Impact

Experiments on 10,000 fundus images for binary glaucoma diagnosis demonstrate substantial disparity reductions with minimal accuracy trade-off.

| Method         | Overall Acc (%) | MaxAccGap (%) |
| -------------- | --------------- | ------------- |
| Zero-Shot      | 50.15           | 3.95          |
| Vanilla LoRA   | 53.50           | 3.80          |
| FR-LoRA (0.5)  | 53.55           | 6.04          |
| GR-LoRA        | 53.15           | 1.17          |
| Hybrid-LoRA    | 53.45           | 3.80          |

GR-LoRA achieves the lowest MaxAccGap (1.17%), evenly distributing accuracy across Non-Hispanic (53.14%), Hispanic (53.95%), and Unknown (52.78%) groups. Ablation for FR-LoRA shows that strong regularization ($\lambda = 1.0$) can robustly reduce the MaxAccGap. Hybrid-LoRA generalizes MaxAccGap reduction across different attributes; for example, the race attribute saw reduction from 4.36% (Vanilla) to 1.74% (Hybrid).

## 5. Trade-offs, Limitations, and Generalization

Differentiable MaxAccGap optimizes for accuracy parity, which is deemed clinically relevant for diagnostic tasks. However, it is restricted to single-output (binary) classification in current application. Intersectional fairness across multiple sensitive attributes is not addressed, nor are alternative notions such as equalized odds (e.g., TPR, FPR parity). Soft-accuracy assumes probability calibration, which, if violated, could misdirect the gap gradient.

Task generalization proceeds by redefining group-wise soft metrics: for multi-class classification, compute soft-accuracy per class and group; for regression, apply group-wise $R^2$ or negative MSE within the same max-min regularization. The group balancing machinery is similarly extensible to attributes including age and socio-economic status.

## 6. Significance in Fair Medical AI and Deployment Considerations

By transforming the clinically interpretable metric of accuracy parity into a gradient-friendly regularizer, Differentiable MaxAccGap loss bridges theoretical fairness with practical, parameter-efficient adaptation of billion-parameter VLMs on medical data. The resulting FR-LoRA, GR-LoRA, and Hybrid-LoRA methods achieve up to 70% disparity reduction and 53.15% overall accuracy, requiring only 0.24% trainable parameters, offering a promising solution for fair medical AI in resource-constrained clinical environments [2512.03477].

Source: https://www.emergentmind.com/topics/differentiable-maxaccgap-loss