Worst Accuracy in Model Evaluation

• Worst accuracy is a performance metric that focuses on a model's weakest segments rather than its average, revealing hidden failure modes.
• It evaluates minimum performance across subgroups, classes, tasks, or episodes to ensure no critical segment is neglected.
• The metric guides improvements in robustness and fairness by balancing overall performance gains with targeted enhancements on the lower tail.

Searching arXiv for papers on worst-group/worst-class accuracy to ground the article. Worst accuracy is a family of evaluation criteria that quantify model performance on the least well-served part of an evaluation domain rather than on the average case. In recent machine-learning literature, the term appears as worst-group accuracy under subpopulation shift, worst-class or worst-$k$ accuracy in class-conditional analysis, worst-case robust accuracy under adversarial or transformation perturbations, worst-task accuracy in multitask learning, and worst-episode accuracy in few-shot recognition (Park et al., 6 Nov 2025, Shao et al., 2023, Li et al., 2023, Michel et al., 2021, Fu et al., 2022). Across these settings, the common objective is to expose concentrated failure modes that can remain invisible when performance is summarized only by mean accuracy.

1. Formal definitions and mathematical variants

The literature does not use a single universal definition of worst accuracy; instead, the exact quantity depends on the evaluation axis. For subpopulation shift, one paper defines worst-group error for groups $g\in G$ with distributions $\mathbb P_g$ as

$$E_{\mathrm{wge}}(f):=\max_{g\in G}\mathbb P_{(x,y)\sim\mathbb P_g}\bigl(f(x)\neq y\bigr),$$

with worst-group accuracy

$$\mathrm{Acc}_{\mathrm{wg}}(f)=1-E_{\mathrm{wge}}(f).$$

For datasets indexed by label–attribute groups $g=(y,a)$, another standard form is

$\mathrm{WGA}(f)=\min_{g\in G}\mathrm{Acc}_g(f).$

In zero-shot class analysis, worst-class accuracy is $\mathrm{Worst}@1=\min_i Acc_i$, and worst-$k$ accuracy is

$$\mathrm{Worst}@k=\frac1k\min_{K\subseteq\{1,\dots,N\},\,|K|=k}\sum_{i\in K}Acc_i.$$

In adversarial robustness, worst-class robust risk is

$g\in G$0

and in multitask learning the corresponding loss-based objective is

$g\in G$1

Few-shot recognition introduces an episode-level notion, with average episode accuracy $g\in G$2 and worst-case accuracy $g\in G$3. Long-tailed learning uses the minimum per-class recall

$g\in G$4

as its worst-category quantity (Park et al., 6 Nov 2025, Idrissi et al., 2021, Shao et al., 2023, Li et al., 2023, Michel et al., 2021, Fu et al., 2022, Du et al., 2023).

These definitions are closely related but not interchangeable. Some are complements of worst-case error, some are minima over group-conditional accuracies, and others are maxima over loss values. This suggests that “worst accuracy” is best understood as an umbrella term for lower-tail performance criteria rather than as a single metric.

2. Why average accuracy is insufficient

A central motivation for worst-accuracy metrics is that average accuracy can remain high while specific groups or classes fail catastrophically. Under subpopulation shifts, average accuracy can remain high even though certain minority groups suffer catastrophic failures; worst-group accuracy is introduced precisely to ensure that no subpopulation is neglected (Park et al., 6 Nov 2025). In CLIP zero-shot classification, the reported overall top-1 accuracy on ImageNet can be 64.1% while ten categories have class-wise accuracy 0%, which directly motivates reporting Worst@1 or Worst@$g\in G$5 rather than only the mean (Shao et al., 2023).

The same pathology appears in robust learning. Standard adversarial training often yields good average robust accuracy while leaving some classes nearly undefended, motivating worst-class robust risk as a separate objective (Li et al., 2023). A later study states that on CIFAR-10 the best and worst class accuracies can be 74% and 23%, respectively, underlining that average robust performance does not guarantee uniformly protected classes (Pethick et al., 2023). Few-shot recognition makes the same point at the level of episodes: if deployment effectively samples a single episode, then a high mean over many episodes does not protect against a single low-accuracy realization, so the lower tail becomes operationally important (Fu et al., 2022).

These examples establish a recurring pattern: worst accuracy is not merely a fairness-style auxiliary statistic but often the quantity that captures the actual failure mode of interest.

3. Group robustness, spurious correlations, and subpopulation shift

Worst-group accuracy has become the canonical formulation for settings in which data are partitioned by known or latent subpopulations. For data $g\in G$6 with groups $g\in G$7, group-DRO minimizes the maximum group loss,

$g\in G$8

rather than the average loss (Idrissi et al., 2021, Sagawa et al., 2019). The motivation is that overparameterized networks can achieve near-perfect groupwise training performance while still exhibiting poor worst-group generalization at test time; one paper attributes this to group-dependent generalization gaps and shows that stronger $g\in G$9 regularization or early stopping is important for worst-group generalization in the overparameterized regime (Sagawa et al., 2019).

A more recent theoretical development connects worst-group error to representation geometry. In a Gaussian mixture model with core and spurious directions $\mathbb P_g$0 and $\mathbb P_g$1, worst-group error depends on a spurious term involving $\mathbb P_g$2 and a core term involving $\mathbb P_g$3; alignment with the spurious direction increases worst-group error, whereas alignment with the core direction decreases it (Park et al., 6 Nov 2025). The proposed "Spurious Correlation-Aware Embedding Regularization for Worst-Group Robustness" (Park et al., 6 Nov 2025) implements this insight with

$\mathbb P_g$4

combined with a worst-group classification loss. On Waterbirds, CelebA, MetaShift, CivilComments, and MultiNLI, the reported worst-group accuracies are 91.2, 91.4, 86.7, 74.0, and 76.8, respectively, each above the listed best prior by 0.9, 0.4, 0.8, 0.3, and 0.8 points; on highly biased ColorMNIST, the paper states that SCER is stable while preserving overall accuracy (Park et al., 6 Nov 2025).

Not all advances in worst-group accuracy require elaborate objectives. "Simple data balancing achieves competitive worst-group-accuracy" (Idrissi et al., 2021) shows that subsampling or reweighting by class or group can match state-of-the-art accuracy on several benchmarks, and that access to group information is most critical for model selection rather than for training. In the same problem family, "Spread Spurious Attribute: Improving Worst-group Accuracy with Spurious Attribute Estimation" (Nam et al., 2022) uses partial spurious-attribute supervision to infer pseudo-attributes and then run Group DRO, reporting that comparable performance to full spurious-attribute supervision can be achieved using only 0.6% to 1.5% annotated samples, depending on dataset. "Multitask Learning Can Improve Worst-Group Outcomes" (Kulkarni et al., 2023) further reports that an $\mathbb P_g$5-regularized multitask representation improves both worst-group and average outcomes in fine-tuning settings with few or no group annotations.

4. Worst accuracy under adversaries, transformations, smoothing, and hardware variation

In robustness research, worst accuracy is often defined over adversarial perturbations or other worst-case environments rather than over demographic groups. "WAT: Improve the Worst-class Robustness in Adversarial Training" (Li et al., 2023) formalizes worst-class robust risk as $\mathbb P_g$6, casts optimization as a two-player zero-sum game, and solves it with no-regret dynamics based on Hedge or Multiplicative Weights. On CIFAR-10 with ResNet-18 under PGD-100, the paper reports that TRADES attains average robust accuracy about 51.7% and worst robust accuracy about 25.2%, whereas WAT attains about 49.1% average robust accuracy and 36.6% worst robust accuracy, yielding $\mathbb P_g$7 (Li et al., 2023).

"Revisiting adversarial training for the worst-performing class" (Pethick et al., 2023) pushes the same axis further by optimizing

$\mathbb P_g$8

or its simplex relaxation over class weights. Its class-focused online learning method updates both model parameters and a class-sampling distribution. On CIFAR-10, ERM-AT reports robust accuracies 51.38 / 24.00 / 23.50 for average / 20%-tail / worst, while CFOL-AT reports 50.14 / 33.00 / 32.00; on STL-10, worst robust accuracy rises from 5.87 to 12.25 (Pethick et al., 2023).

Worst-case perturbations also arise outside $\mathbb P_g$9-style adversarial examples. "Invariance-inducing regularization using worst-case transformations suffices to boost accuracy and spatial robustness" (Yang et al., 2019) defines population worst-case risk over a compact set of spatial transformations $$E_{\mathrm{wge}}(f):=\max_{g\in G}\mathbb P_{(x,y)\sim\mathbb P_g}\bigl(f(x)\neq y\bigr),$$0 by

$$E_{\mathrm{wge}}(f):=\max_{g\in G}\mathbb P_{(x,y)\sim\mathbb P_g}\bigl(f(x)\neq y\bigr),$$1

with worst-case accuracy $$E_{\mathrm{wge}}(f):=\max_{g\in G}\mathbb P_{(x,y)\sim\mathbb P_g}\bigl(f(x)\neq y\bigr),$$2 when $$E_{\mathrm{wge}}(f):=\max_{g\in G}\mathbb P_{(x,y)\sim\mathbb P_g}\bigl(f(x)\neq y\bigr),$$3. On CIFAR-10, adding the invariance penalty to adversarially trained ResNet-32 raises robust grid accuracy from about 58.3% to about 77.5%; on SVHN, the method raises standard accuracy from about 94.0% to about 96.1% and robust accuracy from about 82.6% to about 92.7% (Yang et al., 2019).

Certified robustness introduces yet another variant. "Principal Eigenvalue Regularization for Improved Worst-Class Certified Robustness of Smoothed Classifiers" (Jin et al., 21 Mar 2025) derives a PAC-Bayesian worst-class bound whose leading empirical term is linear in the principal eigenvalue of the smoothed confusion matrix. The paper regularizes this eigenvalue during smooth training and reports, for CIFAR-10 at $$E_{\mathrm{wge}}(f):=\max_{g\in G}\mathbb P_{(x,y)\sim\mathbb P_g}\bigl(f(x)\neq y\bigr),$$4, overall / worst certified accuracy at radius $$E_{\mathrm{wge}}(f):=\max_{g\in G}\mathbb P_{(x,y)\sim\mathbb P_g}\bigl(f(x)\neq y\bigr),$$5 of 71.8% / 59.4%, compared with 71.4% / 58.6% for the cited baseline; it also reports class-wise standard deviation 0.118 versus 0.175 (Jin et al., 21 Mar 2025).

Hardware variation creates a non-data-distribution version of the same problem. "Compute-in-Memory based Neural Network Accelerators for Safety-Critical Systems: Worst-Case Scenarios and Protections" (Yan et al., 2023) defines worst-case accuracy over bounded weight perturbations induced by device deviations and searches for the perturbation that minimizes the number of correctly classified samples. The paper reports that worst-case accuracy in large networks can fall to nearly zero and proposes A-TRICE, a combination of adversarial training and right-censored Gaussian noise injection, improving worst-case accuracy under device variations by up to 33% (Yan et al., 2023).

5. Tasks, episodes, categories, and long-tailed recognition

Worst accuracy is also used when the relevant axis is neither group nor perturbation, but task, episode, or category. In multitask learning, "Balancing Average and Worst-case Accuracy in Multitask Learning" (Michel et al., 2021) contrasts average loss minimization with the worst-case objective $$E_{\mathrm{wge}}(f):=\max_{g\in G}\mathbb P_{(x,y)\sim\mathbb P_g}\bigl(f(x)\neq y\bigr),$$6. Its Lookahead-DRO method uses a one-step Taylor approximation with lookahead matrix

$$E_{\mathrm{wge}}(f):=\max_{g\in G}\mathbb P_{(x,y)\sim\mathbb P_g}\bigl(f(x)\neq y\bigr),$$7

and solves a saddle-point problem $$E_{\mathrm{wge}}(f):=\max_{g\in G}\mathbb P_{(x,y)\sim\mathbb P_g}\bigl(f(x)\neq y\bigr),$$8. The reported empirical result is a better trade-off between average and worst-case task accuracy on multitask CIFAR-100, and the best average relative perplexity on a multilingual language-modeling benchmark with worst-case relative perplexity close to Baselined-DRO (Michel et al., 2021).

Few-shot learning uses episode-level worst accuracy. "Worst Case Matters for Few-Shot Recognition" (Fu et al., 2022) defines $$E_{\mathrm{wge}}(f):=\max_{g\in G}\mathbb P_{(x,y)\sim\mathbb P_g}\bigl(f(x)\neq y\bigr),$$9 and motivates it by the fact that deployment may involve only one episode. The paper proposes stability regularization,

$$\mathrm{Acc}_{\mathrm{wg}}(f)=1-E_{\mathrm{wge}}(f).$$0

with

$$\mathrm{Acc}_{\mathrm{wg}}(f)=1-E_{\mathrm{wge}}(f).$$1

together with an ensemble and adaptability calibration. On miniImageNet 1-shot, LR-DC reports $$\mathrm{Acc}_{\mathrm{wg}}(f)=1-E_{\mathrm{wge}}(f).$$2, $$\mathrm{Acc}_{\mathrm{wg}}(f)=1-E_{\mathrm{wge}}(f).$$3, and $$\mathrm{Acc}_{\mathrm{wg}}(f)=1-E_{\mathrm{wge}}(f).$$4, while AC+EnSR reports $$\mathrm{Acc}_{\mathrm{wg}}(f)=1-E_{\mathrm{wge}}(f).$$5, $$\mathrm{Acc}_{\mathrm{wg}}(f)=1-E_{\mathrm{wge}}(f).$$6, and $$\mathrm{Acc}_{\mathrm{wg}}(f)=1-E_{\mathrm{wge}}(f).$$7; on CUB 1-shot, worst-case accuracy rises from 44.00% to 53.04% (Fu et al., 2022).

Long-tailed recognition reformulates the problem through per-class recall. "No One Left Behind: Improving the Worst Categories in Long-Tailed Learning" (Du et al., 2023) emphasizes the minimum recall $$\mathrm{Acc}_{\mathrm{wg}}(f)=1-E_{\mathrm{wge}}(f).$$8 and the harmonic mean

$$\mathrm{Acc}_{\mathrm{wg}}(f)=1-E_{\mathrm{wge}}(f).$$9

arguing that average accuracy can leave some classes behind. Its Geometric Mean Loss fine-tunes only the classifier layer of a pretrained model. On CIFAR100-LT with imbalance 100, PaCo reports $g=(y,a)$0, PaCo+GML reports $g=(y,a)$1, and PaCo+GML (Ensemble) reports $g=(y,a)$2, while harmonic mean rises from 36.42 to 41.02 (Du et al., 2023).

Zero-shot vision-LLMs add a diagnostic perspective. "Investigating the Limitation of CLIP Models: The Worst-Performing Categories" (Shao et al., 2023) defines the Class-wise Matching Margin

$g=(y,a)$3

where $g=(y,a)$4 is the empirical average similarity between class-$g=(y,a)$5 images and prompt $g=(y,a)$6. CMM is used to identify worst-performing categories and to score candidate prompt templates. On ImageNet with ViT-B/16, the baseline zero-shot classifier using class names achieves Worst@10 $g=(y,a)$7, while the paper’s CPE method raises Worst@10 to 5.2% without labeled validation images; Worst@100 improves from 14.7% to 23.96% (Shao et al., 2023).

6. Optimization trade-offs, validation, and practical interpretation

A recurring issue in worst-accuracy optimization is the relation between lower-tail improvement and average performance. Several papers make this trade-off explicit. WAT introduces

$g=(y,a)$8

a joint score that rewards worst-class gains and penalizes average losses (Li et al., 2023). CFOL likewise exposes a tunable balance through the exploration parameter $g=(y,a)$9; the paper reports that larger $\mathrm{WGA}(f)=\min_{g\in G}\mathrm{Acc}_g(f).$0 recovers almost no loss in average robust accuracy while still boosting the worst class (Pethick et al., 2023). By contrast, the transformation-robustness paper proves that under group-structure and label-constancy assumptions there is no unavoidable trade-off between standard and worst-case accuracy in the infinite-data regime (Yang et al., 2019).

Validation methodology is another practical fault line. The balancing study reports that all methods, including methods that do not use attributes for training, need attribute labels on the validation set to pick the model that maximizes validation WGA; if checkpoints are selected by average validation accuracy instead, average WGA drops by about 10–20 points (Idrissi et al., 2021). This makes worst-accuracy optimization partly a model-selection problem rather than only an objective-design problem.

Recent work also shows that worst-accuracy improvement can coexist with other constraints. Regularized multitask learning is reported to improve both average and worst-group outcomes in fine-tuning regimes (Kulkarni et al., 2023). "Adaptive Sampling for Private Worst-Case Group Optimization" (Cairney-Leeming et al., 11 Feb 2026) addresses the differential-privacy setting, where unequal weighting can weaken privacy for minority groups. Its ASC algorithm adaptively changes both group sampling rates and clipping thresholds so that each group’s update has the same per-step RDP guarantee, and it reports substantially higher worst-case group accuracy without sacrificing overall average accuracy on Unbalanced MNIST, CelebA, and Bank Fraud (Cairney-Leeming et al., 11 Feb 2026).

Taken together, these results support a consistent interpretation. Worst accuracy is the metric family used when the dominant concern is not aggregate competence but the performance floor: the least accurate group, class, task, episode, or perturbation regime. The literature treats that floor as both an evaluative object and an optimization target, and the main technical question is how to raise it without inducing unacceptable losses elsewhere.