AdapT-Bench: Unified TTA Benchmark

• AdapT-Bench is a comprehensive benchmark that systematically evaluates test-time adaptation methods to enhance model robustness under distribution shifts.
• It standardizes experimental protocols by fixing hyperparameters and integrating diverse datasets and backbone architectures, including CNNs and Vision Transformers.
• The framework enables reproducible, fair comparisons across 13 TTA algorithms spanning TTBA, OTTA, and TTDA paradigms with robust empirical metrics.

AdapT-Bench is a standardized, extensible benchmark designed to systematically evaluate and compare test-time adaptation (TTA) methodologies for image classification under distribution shifts. It addresses inconsistencies in experimental setup, reproducibility, and coverage in prior TTA evaluation efforts by providing a unified PyTorch framework, fixed hyperparameter search procedures, and a broad suite of datasets and backbone architectures (Yu et al., 2023).

1. Benchmark Definition and Goals

AdapT-Bench provides a comprehensive evaluation platform for TTA methods, which enhance model robustness by leveraging unlabeled test-time samples to adapt to previously unseen data distributions. The benchmark systematically covers all major TTA paradigms: Test-Time Batch Adaptation (TTBA), Online Test-Time Adaptation (OTTA), and Test-Time Domain Adaptation (TTDA). It standardizes experimental protocols across 13 representative TTA algorithms (plus multi-epoch variants), five widely adopted image classification datasets, and both convolutional neural network (CNN) and Vision Transformer (ViT) backbones.

By fixing key components—batch size, data ordering, hyperparameter search space, and source model weights—AdapT-Bench eliminates confounding variables prevalent in earlier studies, enabling fair, reproducible, and transparent assessment of TTA method effectiveness.

2. Datasets and Distribution Shift Taxonomy

AdapT-Bench encompasses both synthetically corrupted and natural domain-shift datasets, supporting a wide array of transfer and streaming adaptation scenarios:

• CIFAR-10-C / CIFAR-100-C: These are the canonical CIFAR-10/100 test sets augmented by 15 common corruptions (e.g., Gaussian noise, blur, brightness) at five severity levels. The framework fixes severity at the most challenging level (5). These datasets probe generalization under severe covariate shift.
• ImageNet-C: The ImageNet-1k validation set with the same 15 corruptions. Evaluations are performed on ResNet-50 and ViT-Base/16 architectures, expanding coverage to large-scale, high-capacity models.
• Office-Home: Comprises four domains (Artistic, Clip Art, Product, Real-World) and 65 classes, introducing realistic style transfer challenges.
• DomainNet126: A 126-class subset of DomainNet, spanning four domains (Clipart, Painting, Real, Sketch), simulating both feature and label distribution shifts.

TTBA and OTTA process samples as sequential or batched inputs (batch size typically 64), while TTDA aggregates the entire test set for offline, multi-epoch global domain adaptation. Each paradigm is instantiated according to the nature of the distribution shift—corruption or style/domain shift.

3. Adaptation Algorithms Evaluated

AdapT-Bench includes 13 core algorithms and 5 multi-epoch OTTA variants, categorized as follows:

• Test-Time Batch Adaptation (TTBA):
  • PredBN: BatchNorm in training mode using test batch statistics; no parameter updates.
  • PredBN+: Linear interpolation between source and batch BN statistics.
  • MEMO: Sample augmentation and stable statistics for entropy minimization; updates all parameters.
  • LAME: Laplacian-adjusted post-hoc logit refinement; parameter-free.
• Online Test-Time Adaptation (OTTA):
  • Tent: Minimizes entropy with respect to BN affine parameters $\gamma,\beta$ per test batch.
  • T3A: Feature-space prototype adjustment without backbone modification.
  • CoTTA: Teacher-student, parameter-averaging, stochastic weight restoration and augmentations; full parameter updates.
  • EATA: Entropy-based sample selection to restrict BN affine parameter updates.
  • SAR: Sharpness-Aware Minimization (SAM) for BN affine updates.
  • -E10 Variants: Each online method above is run for 10 full-epoch traversals over the test set, bridging online and offline regimes.
• Test-Time Domain Adaptation (TTDA):
  • SHOT: Information maximization, adapting feature extractors in multi-epoch, full offline mode.
  • NRC: Neighborhood clustering with contrastive loss and pseudo-labeling.
  • AdaContrast: Contrastive learning on test augmentations with diversity regularization.
  • PLUE: Pseudo-label refinement using uncertainty and entropy.

A summary of representative methods by paradigm is provided below.

Paradigm	Method	Update Scope
TTBA	PredBN	No gradients
TTBA	MEMO	All parameters
OTTA	EATA	BN affine only
TTDA	AdaContrast	All parameters

All algorithms inherit from a common TTA base in the implementation, standardizing their interface and evaluation (Yu et al., 2023).

4. Framework Implementation and Experimental Protocol

The unified PyTorch-based infrastructure used in AdapT-Bench ensures consistent evaluation across methods and datasets:

• Data modules with reproducible shuffling and batching (fixed seed).
• Model wrappers to load pre-trained ResNet, DenseNet, or ViT backbones from standardized checkpoints (e.g., RobustBench, torchvision).
• TTA base class exposing .adapt_batch() and .predict() APIs for seamless integration of new adaptation strategies.
• Experiment driver that loops over all dataset-method-backbone combinations, executing adaptation and prediction according to the algorithmic paradigm. For TTDA, the test set is repeatedly traversed; for TTBA, models are reset to the source state on each batch; for OTTA, state accumulates through the stream. Metrics and results are written to CSV for postprocessing.

A core pseudocode excerpt illustrates the unified protocol:

for dataset in Datasets: for method in Methods: model = load_pretrained(backbone, dataset) method.initialize(model) if method.paradigm == 'TTDA': for epoch in range(E_max): for x in loader: method.adapt_batch(model, x) preds = method.predict_all(model, loader) else: for (x, y) in loader: if method.paradigm == 'TTBA': model.reset_to_source() method.adapt_batch(model, x) preds = method.predict(model, x) record(preds, y) compute_and_store_metrics()

Hyperparameters are chosen via grid search on a single held-out validation corruption per dataset, with fixed search ranges. Each experiment is repeated across three random seeds for averaging and statistical assessment, and methods are ranked via the Friedman test.

5. Evaluation Metrics

AdapT-Bench quantifies adaptation via multiple metrics:

• Average Classification Accuracy:

$$\mathrm{Acc} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}\{\hat y_i = y_i\}$$

• Average Corruption Error (ACE), the mean error over 15 corruption types:

$$\mathrm{ACE} = \frac{1}{15} \sum_{k=1}^{15} (1 - \mathrm{Acc}_k)$$

• Adaptation Gain compared to the non-adapted source model:

$$\Delta = \mathrm{Acc}_{\mathrm{TTA}} - \mathrm{Acc}_{\mathrm{source}}$$

Results are averaged over three seeds; standard deviation is reported. Methodological significance is assessed through the Friedman ranking procedure.

6. Empirical Results and Comparative Analysis

AdapT-Bench provides robust empirical insights across the spectrum of distribution shift and backbone configurations:

• Corruptions (e.g. CIFAR-10-C, ImageNet-C):
  • PredBN achieves substantial gains over source (e.g., +22.6% on CIFAR-10-C).
  • EATA and SAR (OTTA, BN-affine only, entropy-filtered) achieve the top average ranks on streaming corrupted data (average ranks 5.7 and 6.0). SAR-E10 yields the highest mean accuracy (67.62%).
  • TTDA methods (e.g., AdaContrast, SHOT) excel on small/medium synthetic corruption datasets but underperform on large-scale or high-resolution datasets such as ImageNet-C. Conversely, contrastive TTDA methods outperform others under natural shift (style, domain) but not under severe synthetic corruptions.
• Natural-shift (Office-Home, DomainNet126):
  • TTDA algorithms (especially NRC and AdaContrast) are most effective, underscoring the utility of global self-supervised objectives under strong domain-style shifts.
  • TTBA methods contribute insignificantly or negatively in natural shifts, since BN statistic drift no longer dominates the transfer challenge.
  • T3A (OTTA) becomes competitive due to its independence from batch-based normalization.
• ViT Backbone:
  • Entropy-based OTTA (EATA-E10, SAR-E10) remain optimal (67.1% and 65.9% accuracy).
  • Methods that manipulate batch statistics (e.g., CoTTA) are occasionally less stable under snow/frost corruptions.
• Continual TTA (CTTA):
  • CoTTA demonstrates resilience to non-stationary corruption streams (+0.33% gain over OTTA).
  • Other OTTA algorithms exhibit negligible or slightly negative adaptation in continual settings, reflecting the inherent difficulty of dynamic shift adaptation.

Sample accuracy and rank data for prominent methods on corruption benchmarks:

Method	Avg. Acc. (%)	Rank
Source	42.73	19.0
PredBN	58.20	13.3
Tent	63.96	9.0
EATA-E10	67.62	2.3
SAR-E10	67.55	2.3
AdaContrast	59.75	7.7
NRC	60.33	9.0
SHOT	65.28	6.0

7. Best Practices and Guidance

AdapT-Bench provides the following empirically grounded recommendations for future TTA research and deployment:

• Use EATA or SAR (entropy-filtered BN-affine update) for streaming inference on severely corrupted data when computational overhead and latency are critical.
• Harness PredBN+, MEMO for large-batch, heavy corruption settings, as they offer high efficiency with minimal retraining.
• Deploy TTDA algorithms, particularly NRC or AdaContrast, for substantial natural style or domain shifts; global self-supervised objectives are essential for effective feature alignment in these circumstances.
• When partial test set access is permitted, multi-epoch OTTA variants (e.g. -E10) can approach TTDA performance with lower resource requirements.
• Always hold batch size and data ordering fixed, and restrict hyperparameter tuning to an independent, held-out shift for legitimate generalization estimates.
• For ViT architectures, replace BatchNorm adaptation with LayerNorm adaptation in entropy-based OTTA for stability.
• In continual adaptation (CTTA), stochastically reset parameters (CoTTA) or explicitly regularize drift (EATA) to mitigate catastrophic forgetting.

AdapT-Bench thereby consolidates a previously fragmented evaluation landscape, enabling reproducible, rigorous, and exhaustive assessment of TTA algorithms across corruption and natural distribution shifts, adaptation regimes, and backbone architectures (Yu et al., 2023).