Worst-Group Loss in Robust Learning

• Worst-group loss is defined as the maximum expected loss over predefined groups, serving as a robust measure to identify the subgroup with the poorest performance.
• Researchers employ Group DRO methods that optimize the worst-group objective through min–max formulations and adaptive regularization to improve minority group outcomes.
• Practical implementations reveal trade-offs between average and worst-group accuracy, driving ongoing research in fairness, adversarial training, and privacy-preserving robustness.

Worst-group loss, also called worst-group risk, is the maximum expected loss over a predefined collection of groups, and is the central quantity that Group DRO methods seek to control. It is used to target the subgroup on which a model performs worst, rather than optimizing only average risk, and is therefore closely tied to robustness against spurious correlations, subgroup disparities, and tail failures. In the 0–1 loss setting, the corresponding worst-group accuracy is the minimum group accuracy, or equivalently one minus the maximum group error. Across recent work, the same basic object appears in group distributionally robust optimization, minimax group fairness, adversarially robust worst-class training, preference optimization for labeler groups, and differentially private robust learning (Ghosal et al., 2023, Liu et al., 2021, Zhou et al., 2024).

1. Formal definition and notation

Let $G$ be a finite set of groups, $P_g$ the group-conditional distribution for group $g$, $f_\theta$ a model with parameters $\theta$, and $\ell$ a loss. The per-group risk is

$$R_g(\theta) = \mathbb{E}_{(x,y)\sim P_g}[\ell(f_\theta(x), y)],$$

and the worst-group loss is

$$L_{\mathrm{worst}}(\theta) = \max_{g \in G} R_g(\theta).$$

This formulation appears throughout the literature, including group robustness under spurious correlations, minimax group fairness, and private worst-group risk minimization (Sagawa et al., 2019, Diana et al., 2020, Zhou et al., 2024).

When $\ell$ is the $0$–$P_g$0 loss, worst-group loss becomes worst-group error, and worst-group accuracy is the minimum over group accuracies: $P_g$1 Several papers use this equivalence explicitly. “Just Train Twice” defines worst-group accuracy as $P_g$2 minus the maximum group error, while multitask worst-group robustness work defines it as the minimum per-group accuracy (Liu et al., 2021, Kulkarni et al., 2023).

The groups themselves vary by application. In spurious-correlation benchmarks, groups are often defined as $P_g$3, where $P_g$4 is the label and $P_g$5 is an environment or spurious attribute, so that $P_g$6 (Ghosal et al., 2023). In fairness-sensitive medical imaging, the group set is comprised of single attribute–value pairs such as age bins, sex categories, and race categories (Kurian et al., 19 May 2026). In imbalanced tabular learning, the groups may simply be the majority and minority classes (Mustapha et al., 2023). In adversarial training for worst-class performance, the groups are the classes themselves (Pethick et al., 2023).

A standard equivalent formulation replaces the max over groups by a max over the simplex of group weights: $P_g$7 Because the inner maximization is linear in $P_g$8, it recovers the worst group in the hard-max case. This min–max form is the basis for most algorithmic instantiations of worst-group optimization (Sagawa et al., 2019, Mustapha et al., 2023).

2. Group DRO as the baseline worst-group objective

The empirical Group DRO objective replaces population group risks by empirical group-average losses. Given a labeled dataset $P_g$9, one minimizes

$g$0

where $g$1 is the number of samples in group $g$2 (Ghosal et al., 2023). This objective directly targets worst-group performance, and in practice it is often implemented through group-weight updates on the simplex, or by selecting the current worst group and backpropagating through that group’s average loss (Sagawa et al., 2019, Mustapha et al., 2023).

A central finding of the modern Group DRO literature is that naive optimization of the empirical worst-group loss can fail in overparameterized neural networks. Sagawa et al. observed that when models achieve vanishing average training loss, the worst-group training loss also vanishes, so poor worst-case test performance arises from poor generalization on some groups rather than failure to optimize the training objective (Sagawa et al., 2019). This shifts attention from the min–max form itself to the interaction between worst-group objectives and regularization.

The same work showed that stronger-than-typical regularization, especially stronger L2 penalties or early stopping, can substantially improve worst-group test accuracy while maintaining high average accuracy. It also introduced a group adjustment heuristic,

$g$3

which biases training toward groups with larger expected generalization gaps, reflecting the fact that smaller groups are more prone to overfitting (Sagawa et al., 2019). Later methods, including PG-DRO, retained this adjustment term as a “generalization adjustment” hyper-parameter (Ghosal et al., 2023).

Optimization schemes differ across settings. Sagawa et al. proposed a stochastic minimax algorithm using exponentiated-gradient updates on group weights and SGD updates on model parameters, with convergence guarantees in the convex case (Sagawa et al., 2019). In binary tabular imbalance, a simpler hard-max batch loss,

$g$4

was used directly, with backpropagation through the group that attained the maximum (Mustapha et al., 2023). These variations preserve the core principle: training pressure should be concentrated on the currently worst-performing group rather than on the average example.

3. Ambiguous, partial, and missing group information

A major limitation of classical Group DRO is its assumption that every training example has a known, hard group assignment. PG-DRO argues that this is restrictive when spurious attributes are ambiguous, as in Waterbirds images with mixed land/water backgrounds. It replaces one-hot group labels by a probabilistic group label vector $g$5, constructed from environment probabilities $g$6. The empirical worst-group risk becomes

$g$7

with effective group size $g$8 (Ghosal et al., 2023). This strictly generalizes Group DRO, since it reduces to the empirical Group DRO objective when all $g$9 are one-hot.

Partial group supervision leads to a different relaxation. “Towards Group Robustness in the presence of Partial Group Labels” formulates worst-group loss as a Rawlsian criterion and introduces Worst-off DRO, which optimizes over a constrained set of admissible group assignments. The constraint set fixes labeled points and matches group marginals within tolerance $f_\theta$0. Under Missing Completely At Random assumptions, the paper derives high-probability guarantees that the true assignment lies in this set, so optimizing the worst-off objective upper-bounds the oracle Group DRO objective with high probability (Lokhande et al., 2022). When full group labels are available, the method reduces exactly to Group DRO.

When no training-time group labels are available, several methods attempt to approximate worst-group optimization indirectly. JTT first trains an ERM model for a limited number of epochs, identifies the error set

$f_\theta$1

and then trains a second model that upweights or upsamples those examples (Liu et al., 2021). The motivation is that early-stopped ERM tends to fit spurious correlations first, so misclassified examples are enriched for the poorly performing groups.

SSA takes a pseudo-labeling route. It trains a spurious-attribute predictor from a small group-labeled set, uses adaptive group-wise confidence thresholds to spread pseudo-attributes across the unlabeled data, and then applies Group DRO on the resulting pseudo-groups (Nam et al., 2022). BAM instead amplifies bias in a first stage by introducing a learnable auxiliary variable for each training sample, then upweights examples that the bias-amplified model misclassifies in a second stage (Li et al., 2023). Fully bias-unsupervised work goes further: uLA uses pretrained self-supervised features to construct a bias proxy $f_\theta$2, estimates $f_\theta$3, and trains with the adjusted conditional

$f_\theta$4

while selecting checkpoints using a bias-unsupervised validation criterion over inferred cells $f_\theta$5 (Tsirigotis et al., 2023).

4. Reformulations across fairness, robustness, alignment, and privacy

Worst-group loss is not confined to standard classification error. In minimax group fairness, the same max-over-groups structure is applied to overall error, false positive rates, or false negative rates, with the goal of minimizing the maximum group loss rather than equalizing all group losses (Diana et al., 2020). This distinction is important: minimax fairness seeks to make the worst-off group “as well-off as possible,” whereas equal-outcome formulations may require degrading low-error groups to achieve parity.

In medical image classification, worst-group optimization has been specialized to operating-point fairness. The worst-group equalized-odds regularizer identifies, at each update, the most under-diagnosed positive subgroup and the most over-diagnosed negative subgroup, then penalizes their logit-margin deviations using log-sum-exp relaxations: $f_\theta$6 This targets Equalized Odds and Equalized Opportunity at clinically chosen thresholds rather than optimizing only aggregate predictive risk (Kurian et al., 19 May 2026).

In adversarial training, worst-group loss becomes worst-class adversarial risk. “Revisiting adversarial training for the worst-performing class” studies

$f_\theta$7

a min–max–max problem over parameters, classes, and adversarial perturbations (Pethick et al., 2023). Its CFOL algorithm uses online exponential-weights updates over classes to focus training on the worst-performing class.

Preference optimization in RLHF uses an analogous structure at the level of labeler groups. GRPO defines the worst-group preference loss

$f_\theta$8

and solves

$f_\theta$9

with multiplicative updates on group weights and gradient descent on policy parameters (Ramesh et al., 2024). Here the groups are preference-source groups such as demographics, ethnicities, or teams, and the objective is robust alignment rather than robust classification.

Under differential privacy, worst-group objectives introduce an additional complication: naive reweighting or upsampling can create weaker privacy guarantees for minority groups. ASC addresses this by adaptively controlling both per-group sampling rates and per-group clipping thresholds, so harder-to-learn groups can be sampled more often while maintaining consistent privacy guarantees across groups (Cairney-Leeming et al., 11 Feb 2026). At a more theoretical level, differentially private worst-group risk minimization has been studied with excess population-risk guarantees of order

$\theta$0

for one population-setting algorithm, and nearly optimal excess empirical risk of order

$\theta$1

in the offline setting (Zhou et al., 2024).

5. Empirical behavior and representative results

Across spurious-correlation benchmarks, worst-group optimization usually improves the minimum subgroup performance, often with a modest trade-off in average accuracy. With stronger regularization, Group DRO improved worst-group test accuracy by $\theta$2–$\theta$3 percentage points on Waterbirds, CelebA, and MultiNLI, while maintaining high average accuracy (Sagawa et al., 2019). JTT, which does not use training-time group labels, closed $\theta$4 of the gap in worst-group accuracy between ERM and Group DRO across Waterbirds, CelebA, MultiNLI, and CivilComments-WILDS (Liu et al., 2021).

PG-DRO showed that soft group aggregation can be practically effective. On Waterbirds, it reported worst-group accuracy $\theta$5, compared with $\theta$6 for G-DRO; on CivilComments-WILDS it reported $\theta$7, compared with $\theta$8 for G-DRO, while also supporting reduced annotation and zero-shot CLIP pseudo-labeling (Ghosal et al., 2023). This suggests that probabilistic group membership can be useful not only when labels are noisy, but also when annotations are absent and must be estimated.

Methods that avoid explicit worst-group objectives can still improve worst-group outcomes when their inductive biases suppress spurious features. Regularized multitask learning, which adds MLM or MIM auxiliary loss plus L1 regularization on shared representations, consistently outperformed JTT on both average and worst-group outcomes in the authors’ experiments, reaching $\theta$9 worst-group accuracy on Waterbirds under validation with group labels (Kulkarni et al., 2023). SCER, which directly regularizes alignment with spurious and core embedding directions, reported top worst-group accuracies such as $\ell$0 on Waterbirds, $\ell$1 on CelebA, $\ell$2 on MultiNLI, and $\ell$3 on CivilComments (Park et al., 6 Nov 2025).

Group-discovery methods have also been evaluated on worst-group metrics. AGRO jointly discovers error-prone groups and optimizes them through a CVaR-style robust objective, reporting $\ell$4 higher model performance on average on known worst-groups than prior group-discovery approaches used with G-DRO (Paranjape et al., 2022). In fairness-at-threshold settings, the worst-group equalized-odds regularizer reduced joint EOdds from $\ell$5 to $\ell$6 on RNFL-OCT and from $\ell$7 to $\ell$8 on MIMIC-CXR, with minimal AUC impact (Kurian et al., 19 May 2026). These results indicate that the “worst group” can refer to very different operational targets—classification error, thresholded TPR/FPR behavior, or preference alignment loss—without changing the underlying min–max logic.

6. Limitations, misconceptions, and open directions

Worst-group loss depends critically on how groups are defined. In tabular imbalance experiments, defining groups as majority versus minority class improved G-mean and ROC-AUC, but the same paper notes that mis-specified groups can reduce gDRO’s benefit or even harm overall performance (Mustapha et al., 2023). More generally, many methods assume that all relevant group combinations are represented during training. The systematic generalization experiments on sMPI3D showed that current algorithms can fail to improve over ERM when combinations of observed attribute values are missing, and that bias labels are still crucial for effective model selection in many ostensibly bias-unsupervised pipelines (Tsirigotis et al., 2023).

A common misconception is that optimizing worst-group loss is the same as equalizing outcomes across groups. Minimax group fairness explicitly rejects that equivalence: its goal is minimizing the maximum loss across groups, not forcing all groups to have the same loss (Diana et al., 2020). Another misconception is that simple reweighting or upsampling is always sufficient. Finetuning studies found that mini-batch upsampling and loss upweighting can induce a decrease in worst-group accuracy with training epochs, leading to performance no better than without class-balancing on some datasets, while subsetting or a mixture method may be more effective depending on group structure (LaBonte et al., 2024).

The role of data geometry remains an active point of interpretation. “Why does Throwing Away Data Improve Worst-Group Error?” argues, using extreme value theory, that the tails of the data distribution determine worst-group accuracy for linear classifiers, and that balancing by discarding data can restore geometric symmetry under Gumbel-type tails but not under Weibull-type tails (Chaudhuri et al., 2022). This suggests that some empirical successes of balancing or subset selection are not merely heuristic, but depend on the extremal structure of the training distribution.

Open directions follow directly from these limitations. Current work studies probabilistic groups, partial labels, adversarial group discovery, operating-point fairness, private optimization, and group-level distributional uncertainty under Wasserstein balls (Ghosal et al., 2023, Lokhande et al., 2022, Konti et al., 10 Sep 2025). A plausible implication is that worst-group loss is evolving from a single robust classification objective into a broader design principle: optimize the most vulnerable subgroup under whatever uncertainty model—label ambiguity, missing groups, distribution shift, or privacy constraint—is most salient for deployment.