Compound Domain Test-Time Adaptation

• Compound Domain TTA is an approach that leverages style-based sub-domain partitioning and specialized BN modules to continuously adapt to heterogeneous test data.
• It employs k-means clustering and exponential moving averages to update domain prototypes, thereby reducing overfitting and mitigating catastrophic forgetting.
• Empirical results on benchmarks like ImageNet-C and Cityscapes-C demonstrate enhanced performance under both cyclic and abrupt domain shifts.

Compound Domain Test-Time Adaptation (TTA) addresses the challenge of continuous, robust adaptation in visual recognition systems deployed in dynamic environments where data at inference time are drawn from a compound, heterogeneous mix of domains and are subject to non-stationary shifts. This paradigm is motivated by the realities of robotics, autonomous driving, and related applications, where it is infeasible to pre-train models covering all possible scenarios. Compound Domain TTA leverages principled partitioning of domain characteristics, expert subnetwork modules, and adaptive regularization to maintain performance while mitigating catastrophic forgetting and overfitting under streaming, shifting data distributions (Song et al., 2022).

1. Problem Definition and Context

In real-world deployment, machine learning models encounter streams of data drawn from an evolving mixture of sub-domains (e.g., sunny versus rainy weather, day versus night, clear versus corrupted sensors). Formally, if $p_T(x)$ represents the target data distribution at test time, it is composed as a mixture:

$p_T(x) = \sum_{k=1}^K \alpha_k \, p_k(x)$

where $p_k(x)$ is the $k$-th latent sub-domain marginal and $\alpha_k$ its unknown mixing weight. Neither sub-domain labels nor mixture weights are given. Furthermore, the system must generalize to open sub-domains—novel distributions not observed during (source-based) training.

Classic test-time adaptation (TTA) algorithms, such as TENT and CoTTA, adapt model parameters on incoming test samples. However, these methods use a single set of model parameters and are prone to forgetting and overfitting, especially under recurring or abruptly-changing domain shifts with heterogeneous sub-populations (Song et al., 2022).

The compound domain scenario introduces the need for mechanisms to track, identify, and exploit knowledge about multiple distinct, possibly recurring, sub-domains during lifelong adaptation.

2. Compound Domain Knowledge Management

Compound Domain TTA introduces a knowledge management module that partitions the input space into $K$ sub-domains, each modeled by a dedicated expert batch normalization (BN) module. Domain-distinctive features (ddf) are computed for each sample by extracting style statistics (mean and variance) from early layers of a ResNet backbone:

$$\text{ddf}(x) = \big[ \mu(\phi_0(x)), \sigma(\phi_0(x)), \mu(\phi_1(x)), \sigma(\phi_1(x)) \big]$$

where $\phi_0,\phi_1$ denote the output of the first two convolutional blocks. $K$-means clustering is performed in the ddf space on source data to define $K$ initial sub-domain prototypes $p_T(x) = \sum_{k=1}^K \alpha_k \, p_k(x)$0, and the BN parameters are copied $p_T(x) = \sum_{k=1}^K \alpha_k \, p_k(x)$1 times, yielding $p_T(x) = \sum_{k=1}^K \alpha_k \, p_k(x)$2 experts specializing to each region.

For each incoming test image $p_T(x) = \sum_{k=1}^K \alpha_k \, p_k(x)$3:

• The ddf is computed.
• Bhattacharyya distance is used to measure proximity to each prototype $p_T(x) = \sum_{k=1}^K \alpha_k \, p_k(x)$4:

$p_T(x) = \sum_{k=1}^K \alpha_k \, p_k(x)$5

• The closest expert $p_T(x) = \sum_{k=1}^K \alpha_k \, p_k(x)$6 is selected, and $p_T(x) = \sum_{k=1}^K \alpha_k \, p_k(x)$7 is routed through this BN module for both inference and adaptation.
• Each prototype $p_T(x) = \sum_{k=1}^K \alpha_k \, p_k(x)$8 is updated via exponential moving average: $p_T(x) = \sum_{k=1}^K \alpha_k \, p_k(x)$9, maintaining adaptive sub-domain boundaries.

This continual domain-matching ensures that each expert BN module preserves and evolves sub-domain-specific normalization statistics throughout deployment (Song et al., 2022).

3. Adaptation Objective and Regularization

Test-time adaptation is confined to the BN parameters $p_k(x)$0, with other network weights fixed. The adaptation loss for each batch is a filtered entropy minimization:

$p_k(x)$1

with $p_k(x)$2 and threshold $p_k(x)$3. To prevent overfitting, especially on inputs far from the original source distribution, the framework imposes a Fisher-weighted regularizer:

$p_k(x)$4

Here, $p_k(x)$5 is the Fisher information for $p_k(x)$6, and $p_k(x)$7 denotes the initial (source) BN parameters.

An additional domain-similarity weighting modulates adaptation strength on a per-sample basis. For $p_k(x)$8, its similarity to the source is measured as:

$p_k(x)$9

where $k$0 is the mean/variance from source BN layer $k$1, $k$2 is the corresponding statistic from $k$3, and $k$4. The total loss is:

$k$5

Adaptation steps are thus dampened for test samples whose style statistics deviate from the source, reducing noisy updates and guarding against abrupt overfitting (Song et al., 2022).

4. Algorithmic Workflow

The following outlines the compound domain TTA procedure:

1. Initialization:
   • Compute ddf vectors for all source data.
   • Apply $k$6-means (with $k$7) to define sub-domain prototypes.
   • Duplicate BN layers $k$8 times for expert branches.
2. Per-batch Adaptation:
   • For each incoming $k$9:
     1. Compute ddf.
     2. Assign expert index $\alpha_k$0.
     3. Update prototype $\alpha_k$1 with new ddf.
   • Partition the batch by assigned expert.
   • For each expert BN module with assigned data:
     1. Forward samples and compute softmax outputs.
     2. Calculate entropy loss, filter by threshold.
     3. Compute per-sample similarity score $\alpha_k$2.
     4. Calculate Fisher-regularized loss.
     5. Update BN parameters for the expert only.

This modular and adaptive approach ensures that each recurring or emergent domain maintains its own adaptation history, facilitating robust lifelong learning and retention in dynamic environments (Song et al., 2022).

5. Benchmarks and Empirical Findings

Compound Domain TTA has been evaluated in diverse scenarios:

Benchmark	Task	Notable Results
ImageNet-C	Classification	Top-1 error: 53.5% vs. 54.6% (CoTTA)
GTA5 → C-Driving	Segmentation	mIoU: 33.7% vs. 32.1% (CoTTA)
Cityscapes-C	Segmentation	mIoU: 61.5% vs. 60.8% (CoTTA)

In these evaluations, the approach attains consistent performance improvements over existing TTA algorithms, notably under cyclical and abrupt domain shifts, with minimal computational overhead (e.g., ∼8 minutes for full ImageNet-C adaptation). Empirically, the domain similarity weighting $\alpha_k$3 correlates strongly with actual mIoU variation ($\alpha_k$4), outperforming alternatives based purely on confidence or entropy (Song et al., 2022).

6. Limitations and Practical Considerations

Compound Domain TTA assumes that:

• Sub-domains are well-separated in shallow style-statistics space.
• The number $\alpha_k$5 of BN experts is fixed at deployment (cannot expand for entirely novel domains without explicit re-initialization).
• Initial source statistics and parameters are accessible for reference.

The memory overhead from multiple BN modules is marginal compared to full model duplication, and online computations of ddf and similarity metrics are tractable for real-time systems. In highly non-stationary or growing-domain settings, further mechanisms would be required for dynamically adding, merging, or pruning domain experts (Song et al., 2022).

7. Distinctiveness from Other Compound Domain Paradigms

While Open Compound Domain Adaptation (OCDA) (Liu et al., 2019) addresses the challenge of learning from unlabeled target data arising from a mixture of latent sub-domains and potentially open domains, it focuses on pre-deployment adaptation using a curriculum strategy and a memory module for enhanced transfer. Compound Domain TTA, in contrast, operates in a pure test-time, lifelong learning context, explicitly manages expert subnetworks per sub-domain, and modulates adaptation via domain similarity, offering a complementary approach to dynamic domain adaptation in real-world continuous-inference settings (Song et al., 2022). No metric or algorithm named "TTAA" is defined in these works.

---

Key References:

• "Test-time Adaptation in the Dynamic World with Compound Domain Knowledge Management" (Song et al., 2022)
• "Open Compound Domain Adaptation" (Liu et al., 2019)