WILDS Benchmark: OOD Generalization & UDA

• WILDS Benchmark is a suite of real-world datasets designed to evaluate machine learning models on out-of-distribution robustness and domain adaptation through diverse natural shifts.
• It standardizes data splits, evaluation metrics like worst-group accuracy, and baseline implementations across domains such as medical imaging, remote sensing, and wildlife monitoring.
• By supporting both domain generalization and unsupervised domain adaptation, WILDS drives practical insights for developing robust ML models in challenging real-world applications.

The WILDS benchmark is a suite of real-world machine learning datasets specifically curated to study out-of-distribution (OOD) generalization and domain adaptation under realistic distribution shifts. Unlike traditional i.i.d. benchmarks, WILDS focuses on complex, naturally occurring domain shifts encountered in high-impact application areas, ranging from medical imaging to remote sensing, social media, molecular graphs, and wildlife monitoring. WILDS standardizes not only data but also evaluation metrics, splits, and baseline implementations, providing the community with a robust framework for empirical progress in OOD robustness and unsupervised domain adaptation research (Koh et al., 2020, Sagawa et al., 2021).

1. Motivation and Scope

The WILDS benchmark is motivated by the fundamental challenge that machine learning models often experience severe accuracy drops when deployed on data whose distribution differs from the training set. Classic datasets rarely capture the kinds of shifts between source (training) and target (deployment) distributions that are typical in real deployments, where data may be partitioned by hospitals, sensors, countries, time periods, batches, or annotators. WILDS introduces carefully curated datasets where shifts are defined along such real-world axes—termed "domains" or "groups"—with the goal of driving progress in both domain generalization (DG) and unsupervised domain adaptation (UDA) (Koh et al., 2020, Sagawa et al., 2021, Robey et al., 2021).

2. Core Design and Datasets

The benchmark provides a unified suite of ten datasets, each with its own natural domains. The original releases prioritize diversity in data modalities, application areas, and shift structures, as shown in the table below.

Dataset	Domain	Task
Camelyon17-WILDS	Hospital (histology)	Cancer/normal classification
iWildCam-WILDS	Camera trap	Species identification
FMoW-WILDS	Region/year	Land-use multiclass
PovertyMap-WILDS	Country/region	Asset index regression
CivilComments-WILDS	Demographic subgroup	Toxicity classification
AmazonReviews-WILDS	Product category	Sentiment classification
PCQM4Mv2-WILDS	Molecule scaffold	HOMO-LUMO regression
DomainNet-WILDS	Visual style	345-way image classification

Each dataset includes labeled train/validation/test splits, where the test set contains examples from previously unseen domains. WILDS also introduced "worst-group" or "cross-domain" metrics (e.g., minimum accuracy across domains) to directly evaluate OOD robustness (Koh et al., 2020, Sagawa et al., 2021, Irie et al., 2021).

With the WILDS 2.0 extension, eight datasets are augmented with unlabeled data from the target domains, turning WILDS into a standard UDA benchmark. These unlabeled target-domain samples are intended to be realistically obtainable in practical deployments (e.g., unlabelled images from new hospitals or locations) (Sagawa et al., 2021).

3. Evaluation Protocols and Metrics

WILDS prescribes a rigorous evaluation methodology:

• Splits: Disjoint train/validation/test splits by domain, with explicit OOD versus in-distribution (ID) designation.
• Metrics:
  • Classification: Accuracy, macro-F1, or worst-group accuracy.
  • Regression: RMSE or Pearson r, with worst-group aggregation as appropriate.
• Model selection: Hyperparameters are tuned on the OOD validation split. Model performance is reported on both ID and OOD test splits.
• Performance reporting: The standard gap (ID metric minus OOD metric) operationalizes OOD robustness. Scripts for metric-wise checkpoint selection and grid search are recommended to ensure baseline strength (Irie et al., 2021, Koh et al., 2020).

4. Problem Formulations: Domain Generalization and UDA

Original WILDS targets the domain generalization setting: learn a predictor $f$ using labeled data from multiple source domains, aiming to generalize to new, unseen domains. Formally, let $\{\mathcal{D}_i\}$ be source domains, and evaluate $f$ on a disjoint target domain under the same label space (Koh et al., 2020, Robey et al., 2021).

With WILDS 2.0, the UDA protocol is formalized as follows: given labeled source samples $\{(x_i^S, y_i^S)\}$ drawn from $\mathcal{D}_S$ and unlabeled target samples $\{x_j^T\}$ from $\mathcal{D}_T$, learn $f$ to minimize

$$R_T(f) = \mathbb{E}_{(x, y) \sim \mathcal{D}_T} [\ell(f(x), y)]$$

using only source labels and target unlabeled data. Evaluation remains on the held-out target domains (Sagawa et al., 2021).

5. Methodological Baselines and Algorithmic Insights

WILDS datasets serve as a unified testbed for several families of robust learning algorithms. Representative baselines include:

• Empirical Risk Minimization (ERM): Standard supervised learning on labeled sources, often underperforms severely on OOD domains (Koh et al., 2020).
• GroupDRO, IRM, CORAL: Regularization or reweighting schemes to encourage invariance or minimax robustness across domains/groups (Koh et al., 2020, Shi et al., 2021).
• Inter-domain gradient matching (Fish): Aligns optimization trajectories from different domains by maximizing gradient agreement, leading to improved worst-domain performance on WILDS (Shi et al., 2021).
• Model-based DG (MBDG): Enforces functional invariance via a primal-dual formulation, using an explicit domain-shift generator. Substantial relative improvements, especially on histopathology (Robey et al., 2021).
• IID Representation Learning: Seeks a latent mapping imposing distributional symmetry (IID) across groups; demonstrates superiority on batch-effect and sampling-protocol shifts in biomedical data under WILDS protocols (Wu et al., 2022).
• Unsup. Domain Adaptation (UDA) Methods (WILDS 2.0):
  • Domain-invariant representation learning: Minimizes e.g., MMD between source and target representations.
  • Self-training (pseudo-labeling): Assigns high-confidence pseudo-labels to target data and retrains; delivers the largest UDA gains.
  • Self-supervised pretraining: Contrastive or reconstruction objectives on joint unlabeled data, followed by fine-tuning.
  • Empirical results indicate that pseudo-labeling is generally most effective, but all methods' gains are limited when the domain gap is very large or labels/styles correlate with domains (Sagawa et al., 2021).

6. Lessons, Best Practices, and Empirical Observations

Key findings and recommendations from large-scale WILDS evaluations:

• Tight baseline optimization—metric-wise checkpointing, frequent saving, and grid search over hyperparameters—yields significant performance gains and is crucial for fair benchmarking (Irie et al., 2021).
• OOD validation performance does not always correlate strongly with OOD test set performance, especially in datasets like iWildCam with strong domain-label biases.
• Practical data irregularities (e.g., some camera traps with only a handful of species) complicate the design and interpretation of domain-aware methods; strong label-domain correlations create hidden shortcuts (Irie et al., 2021).
• Incorporating geographic metadata via structured encoders can improve robustness to geo-shifts, outperforming vanilla domain-invariant approaches (Crasto, 3 Mar 2025).
• No single algorithm consistently dominates across all WILDS datasets or shifts. Effectiveness is highly context-dependent, and meaningful progress demands per-dataset scrutiny (Koh et al., 2020, Sagawa et al., 2021).

7. Open-Source Resources and Impact

WILDS is distributed as a Python package (installable via pip install wilds), with APIs for convenient data loading, standard architectures, and baseline scripts. A continuously updated leaderboard is maintained at https://wilds.stanford.edu, enabling open, standardized submission and comparison of new methods. The ecosystem encompasses tightly benchmarked UDA baselines, dataset versions (e.g., for unsupervised adaptation), and reproducibility scripts (Koh et al., 2020, Sagawa et al., 2021).

The WILDS benchmark has become a canonical resource for evaluating domain generalization and UDA, informing both algorithmic development and more practical deployment considerations across application domains. Its design principles—rich domain shift, real-world applicability, and standardized evaluation—have broadly influenced subsequent directions in robustness and fairness research.