CCT: Confidence-Consistency Test-Time Adaptation

• CCT is a test-time adaptation framework that filters out noisy or open-set samples using dynamic confidence-difference measures and short-term consistency regularization.
• It employs a two-stage adaptation process, evaluating per-sample confidence and enforcing local feature consistency to improve stability and reduce error accumulation.
• Empirical results demonstrate significant improvements, such as reduced CIFAR-10-C error rates and lower WER in ASR tasks, validating CCT's effectiveness in real-world conditions.

Confidence-Consistency Test-time Adaptation (CCT) is a methodological framework for test-time adaptation (TTA) of deep neural models under domain shift, specifically engineered to enhance stability, prevent confirmation bias, and manage noisy or open-set samples. CCT integrates confidence-based sample selection and short-term consistency regularization, and is applicable both to vision tasks and to foundation models for Automatic Speech Recognition (ASR) operating in wild, real-world acoustic environments (Liu et al., 2023, Lee et al., 2023).

1. Foundational Problem: Test-Time Adaptation under Covariate and Open-Set Shift

Classical TTA aims to adapt a source model, trained on a source domain with class set $C$, to a stream of unlabeled target inputs $\{x_i\}$ from a shifted domain, without recourse to source data or target labels. Key challenges include:

• Covariate shift, where test data distributions diverge from training (due to noise, environmental conditions, device mismatch, etc.).
• Open-set TTA, where unseen class labels ($\notin C$) may appear at test time; naive adaptation can degrade closed-set performance or mis-absorb open-set examples.
• Accumulated error signals from noisy or misclassified samples, especially when applying self-supervised adaptation at scale or in online/long-term deployment.

A core insight motivating CCT is that entropy-minimization (e.g., TENT, SAR) — the prevailing objective in vision TTA — is vulnerable to noise, confirmation bias, and error accumulation, particularly when naively applied to shifted or mixed closed/open data (Lee et al., 2023).

2. Confidence Measurement and "Wisdom of Crowds" Criterion

CCT formalizes a per-sample, dynamic confidence measure to filter adaptation signals. For each test sample $x_i$:

• Let the original (source) model $\theta_0$ produce output $$\tilde{y}_i=\text{softmax}(f(x_i;\theta_0)) \in \mathbb{R}^C$$.
• $c_i^s = \tilde{y}_i^{c_i^s}$ where $c_i^s = \arg\max_k \tilde{y}_i^k$.
• After $k$ TTA steps (model $\theta_a$): $\hat{y}_i = \text{softmax}(f(x_i;\theta_a))$, compute $c_i^t = \hat{y}_i^{c_i^s}$.
• Define the confidence-difference $\Delta c_i = c_i^t - c_i^s$ (Lee et al., 2023).

Empirically:

• Correct-class samples overwhelmingly show $\Delta c_i \geq 0$ (confidence increases or is maintained).
• Wrong or open-set samples typically show $\Delta c_i < 0$ (confidence decays).
• The effect is attributed to the "wisdom of crowds": correct samples' gradients align in prediction space and dominate the global model update, while wrong/open-set gradients are cancelled or repelled.

This difference forms the basis for a sample-selection indicator: $\Phi_i = \mathbb{I}(c_i^t \geq c_i^s)$ Only samples with $\Phi_i = 1$ (i.e., non-decreasing confidence under the adapting model) are used in subsequent adaptation steps.

3. CCT Loss Functions and Algorithmic Framework

CCT combines entropy minimization over filtered samples and optional regularization terms. For example, in semantic segmentation or classification: $$\mathcal{L}(\theta_a) = \frac{1}{\sum_i \Phi_i} \sum_{i=1}^n \Phi_i H\left(\hat y_i\right) - \lambda_{\max} H\left(\frac{1}{n} \sum_{i=1}^n \hat y_i \right)$$ where $H(p) = -\sum_{k=1}^C p^k \log p^k$ is the entropy over the output distribution.

For ASR/acoustic models (Liu et al., 2023), two components are integrated:

1. Confidence-enhanced adaptation (CEA):
   • Compute frame-level entropy $h_i = -\sum_{c=1}^C p_{i,c} \log p_{i,c}$.
   • Define per-frame confidence score $c_i = \sigma(h_i)$, with $\sigma$: logistic sigmoid; high-entropy frames get $c_i \approx 1$, low-entropy or silent frames $c_i \approx 0$.
   • Adaptation is weighted by $w_i = c_i \cdot \mathbb{I}_i$, masking out silent frames.
2. Short-term consistency regularization:
   • For last-layer features $z_{1:n}$ and their self-attention adjusted representations $z_{1:n}'$, enforce similarity within window $k$:

   $$L_{\rm cons} = \sum_{i=1}^{n-k} \|z_i' - z_{i+k}'\|_2^2 \; \mathbb{I}_i$$

The final test-time adaptation loss is: $$L_{\rm TTA} = \lambda_{\rm conf} L_{\rm conf} + \lambda_{\rm cons} L_{\rm cons}$$ Where $L_{\rm conf}$ is the confidence-enhanced entropy minimization, $L_{\rm cons}$ is the short-term consistency loss; hyperparameters (typical: $\lambda_{\rm conf}=1$, $\lambda_{\rm cons}=0.3$).

Adaptation targets only the affine parameters of all LayerNorms, plus (optionally) the feature-extractor block.

Algorithmic Pseudocode:

for each test utterance x_{1:n}: initialize Θ'_0 ← Θ' # reset per utterance for t in 0 .. T-1: # 1. forward pass logits p_i ← f_{Θ'_t}(x_{1:n}) # shape n×C h_i ← -∑_c p_{i,c} log p_{i,c} w_i ← sigmoid(h_i) ⋅ 𝟙_non_silent(i) z_{1:n} ← feature_extractor_{Θ'_t}(x_{1:n}) z'_{1:n} ← self_attention(z_{1:n}) # 2. losses L_conf ← ∑_{i=1}^n w_i ⋅ h_i L_cons ← ∑_{i=1}^{n-k} || z'_i - z'_{i+k} ||_2^2 ⋅ 𝟙_i L_TTA ← λ_conf ⋅ L_conf + λ_cons ⋅ L_cons # 3. update Θ'_{t+1} ← Θ'_t - η ⋅ ∇_{Θ'} L_TTA decode with adapted model f_{Θ'_T}(x_{1:n})

This structure is generally reflected in both acoustic and vision settings, with batch size and windowing adapted to the domain.

4. Empirical Results and Comparative Evaluation

CCT's performance has been systematically benchmarked against leading TTA approaches, revealing the following:

Image Classification and Semantic Segmentation (Lee et al., 2023)

• Long-term adaptation (50 rounds): On CIFAR-10-C, integrating CCT with TENT reduces closed-set error from 45.84% to 14.10%. Similar improvements are observed on CIFAR-100-C, and TinyImageNet-C.
• Open-set protection: CCT restricts error escalation on open-set samples (e.g., SVHN) under both short-term (1 round) and long-term settings.
• Semantic segmentation: CCT consistently improves mean IoU over standard TTA baselines (see sample values in Table below):

Method	CIFAR-10-C (Closed)	Semantic Seg. (mIoU, Cityscapes)	TinyImg-C (Open)
TENT	45.84	46.73	85.22
TENT + CCT	14.10	46.76	15.77
SWR	10.21	46.17	90.55
SWR + CCT	10.12	46.65	72.58

• Open-set detection AUROC: The use of $\Delta c$ far surpasses established OOD scores like MSP or max-logit (e.g., AUROC 88.24 vs. 51.87 on CIFAR10/SVHN-C).

Acoustic Foundation Models / ASR (Liu et al., 2023)

• Word Error Rate (WER) reductions: On LibriSpeech with Gaussian noise, WER is reduced from 41.6% to 28.3%. Significant improvements are also observed under real environmental sounds, accented speech, and sung speech. For DSing-dev, WER drops from 61.8% to 53.5% (Wav2vec2-base).
• Ablation studies: Removing confidence-enhanced adaptation or consistency regularization degrades WER by 1–2%, indicating both are necessary for optimal performance.

5. Domain-Specific Implementations: Vision vs. Acoustic Models

CCT is instantiated differently based on the underlying model and data modality:

• Vision Models: Filtering operates at the sample level using $\Delta c$; adaptation proceeds over minibatches, leveraging batchnorm for statistics (if present).
• Speech Models (e.g., Wav2vec2, HuBERT, Whisper): Sequence-based, transformer architectures without batchnorm; frame-level scoring (e.g., per-frame entropy/confidence) avoids discarding high-entropy but semantically vital frames. Only non-silent, high-entropy frames are adapted upon, addressing the predominance and importance of ambiguous phonetic content in noisy audio. Consistency regularization leverages phoneme-level coherence in short time windows. Adaptation is performed online, per utterance.

6. Practical Implications, Robustness, and Limitations

• Stability: Across batch sizes and learning rates, CCT improves robustness, cutting standard deviations in error-rate by over 50% compared to classic TTA baselines.
• Resource Overhead: Requires two forward passes per batch (for $\theta_0$ and $\theta_a$) but remains substantially lighter than competing robust adaptation strategies (such as SWR).
• Model-agnosticity: CCT provides gains on multiple architectures (ResNet50, WRN28) for vision classification and is not tied to a particular backbone.
• No static thresholds: Filtering based on empirical, per-sample confidence change, rather than fixed cutoffs, enables adaptation to evolving domains.
• Limitations: Some correct, low-confidence samples may be excluded, potentially causing rare but correct samples to be ignored. Future work may refine the $\Delta c$ criterion to admit some tolerances (e.g., negative margin).

A plausible implication is that CCT's crowd-based filtering principle provides a generic mechanism for error suppression during TTA, both for vision and sequence models, where self-supervised signals could otherwise accumulate detrimental drift in long-term adaptation.

7. Relationships to Prior and Contemporary Approaches

CCT generalizes and stabilizes approaches such as TENT, SAR, EATA, and SWR by introducing dynamic, data-driven selection mechanisms and, in the acoustic setting, specialized frame-level weighting and temporal consistency. Notably:

• Unlike vision-centric TTA methods that heuristically discard high-entropy/uncertain samples or rely on batchnorm statistics, CCT adapts responsively per-modal and per-task.
• In speech, CCT refrains from discarding noisy (uncertain) frames, preferring to "denoise" via learnable weighting, acknowledging their content-bearing role.

By explicitly filtering adaptation signals based on the direction of confidence change and regularizing short-term consistency, CCT enables both heuristic-free and source-free online adaptation under both closed-set and open-set wild domain shifts (Liu et al., 2023, Lee et al., 2023).