Reliable Neural Collapse Approximation for Open-World Test-Time Adaptation

Abstract: Test-Time Adaptation (TTA) methods aim to bridge the domain gap between the source and target domains. However, traditional TTA methods become ineffective when the label distribution shift occurs, a challenge commonly referred to as an open-world scenario. In this paper, we introduce a new method named Reliable Neural Collapse approximation (ReNC) for Open-World Test-Time Adaptation (OWTTA). Specifically, we leverage neural collapse as a structural prior for reliable target-domain adaptation. Guided by this prior, we justify that the pre-trained classifier weights can serve as the prototypes of the source domain. By measuring the similarity between samples and prototypes, we filter out the Out-Of-Distribution~(OOD) samples for reliable updates. Furthermore, we propose a neural collapse approximation mechanism to refine these prototypes, ensuring they can gradually adapt to the target domain while maintaining the neural collapse structure. Extensive experiments on several open-world benchmarks demonstrate the superiority of the proposed method. Our empirical analysis suggests that ReNC better preserves NC-related properties in the target domain, providing useful evidence for explaining reliable OWTTA and offering new insights for model design. Code is available at https://github.com/JiaqiLin-AI/ReNC.

• The paper introduces ReNC, a source-free method that uses neural collapse to initialize and adapt prototypes, adaptively filters OOD samples, and jointly optimizes entropy, balance, and feature-alignment losses.
• ReNC achieves state-of-the-art harmonic accuracy in nearly all tested settings, including gains from 43.35% to 65.16% on CIFAR100-C with noise and from 21.37% to 31.02% on ImageNet-C.
• The method improves continual and high-class-count adaptation while remaining computationally practical, but its reliability depends on approximate source-model collapse, separable ID/OOD scores, and near-balanced target classes.

Motivation and problem setting

Test-Time Adaptation (TTA) methods such as TENT and SHOT improve model generalization under data distribution shift by minimizing output entropy or aligning target features with source statistics, without access to source data or target labels. These methods, however, rest on the assumption that source and target share a label space. In open-world scenarios this assumption fails: the target stream contains both In-Distribution (ID) samples from seen classes and Out-of-Distribution (OOD) samples from unseen classes. The paper identifies two coupled difficulties in Open-World Test-Time Adaptation (OWTTA): label distribution shift makes entropy-based updates unreliable (OOD samples exhibit low entropy too, so indiscriminate entropy minimization degrades performance), and the residual domain gap complicates effective adaptation of ID samples.

Existing OWTTA approaches—OWT3, OSTTA, and UniEnt—follow a two-step recipe: filter OOD samples using similarity to prototypes, then apply TTA to the remainder. Their weakness is that prototypes are fixed, derived either from source-domain class means or from pre-trained classifier weights. Source prototypes are frequently unavailable (privacy, copyright, computational cost), and even when accessible they misalign with the shifted target geometry. ReNC addresses both issues by treating neural collapse as a structural prior that is progressively approximated on the target domain rather than assumed static.

Method: neural collapse as a structural prior

The methodological core rests on three established results from unconstrained-features neural collapse theory: convergence of centered class-means to a Simplex ETF ($\mathcal{NC}_2$), self-duality between classifier weights and class-means ($\mathcal{NC}_3$), and within-class variability collapse ($\mathcal{NC}_1$). From these, the paper derives two assumptions—that the pre-trained source model has collapsed into a Simplex ETF, and that the bias term is negligible relative to the weight–prototype inner product—and proves a lemma showing that normalized classifier weights coincide with normalized class prototypes. This justifies a fully source-free initialization: prototypes are read directly off the pre-trained classifier. Empirically, the bias-to-interaction ratio is between $1.1\times10^{-3}$ and $1.5\times10^{-2}$ across CIFAR10, CIFAR100, ImageNet, and VisDA, supporting the second assumption; measured $\mathcal{NC}_1$/$\mathcal{NC}_3$ values confirm approximate collapse of the source models, though ImageNet's $\mathcal{NC}_1$ of 2.33 is notably larger than CIFAR10's 0.17, indicating the collapse assumption is only approximately satisfied at scale.

OOD filtering as adaptive thresholding. Each test sample receives a similarity score $$s_i^t = 1 - \max_j \cos(\mathbf{z}_i^t, \mathbf{c}_j)$$ against the nearest prototype. Rather than fitting symmetric two-component GMMs—as UniEnt does—the authors observe empirically that ID and OOD score distributions are not Gaussian, and instead learn a per-batch threshold $\tau_t^*$ by minimizing intra-cluster variance of the two clusters. Entropy minimization is then applied only to the $\mathcal{NC}_3$0 ID samples.

Neural collapse approximation. A Proposition establishes that under distribution shift, source and target prototypes are distinct Simplex ETFs differing by scaling factors, so pushing target features toward frozen source prototypes is fundamentally mismatched. ReNC therefore optimizes a joint objective with three terms: filtered entropy minimization $\mathcal{NC}_3$1 (variability-collapse prior), a KL divergence between batch-averaged logits and the uniform distribution $\mathcal{NC}_3$2 (preserving the balanced ETF embedding structure), and an assignment-based attraction loss $\mathcal{NC}_3$3 pulling features toward their nearest prototype while prototypes co-adapt. Because mini-batches cover only a subset of classes when $\mathcal{NC}_3$4 exceeds batch size, prototype updates are stochastic—only classes present in the current batch are updated. Optimization alternates over parameters $\mathcal{NC}_3$5, auxiliary variables $\mathcal{NC}_3$6, and prototypes $\mathcal{NC}_3$7 via an ADMM-style decomposition; the $\mathcal{NC}_3$8 update admits a closed form equivalent to an exponential moving average whose momentum $\mathcal{NC}_3$9 adapts to per-class sample counts, and $\mathcal{NC}_1$0. Predictions for non-ID samples are assigned to a $\mathcal{NC}_1$1-th rejection label.

Empirical results

ReNC is evaluated on five open-world benchmarks (CIFAR10-C, CIFAR100-C, ImageNet-C, ImageNet-R, VisDA-C) crossed with five OOD sources (Gaussian noise, MNIST, SVHN, Tiny-ImageNet, and the complementary CIFAR set), at an OOD ratio of 1, using $\mathcal{NC}_1$2, $\mathcal{NC}_1$3, and their harmonic mean $\mathcal{NC}_1$4. It achieves the best $\mathcal{NC}_1$5 in nearly all settings against ten baselines including TENT, SHOT, EATA, RMT, CoTTA, OSTTA, UniEnt, and OWT3. Representative gains:

Benchmark	Best baseline $\mathcal{NC}_1$6	ReNC $\mathcal{NC}_1$7
CIFAR10-C + Noise	81.93 (OWT3)	88.35
CIFAR100-C + Noise	43.35 (RMT)	65.16
ImageNet-C + Noise	46.58 (RMT)	53.94
ImageNet-R + Noise	39.63 (TEST)	56.77
VisDA-C + Noise	69.80 (UniEnt)	71.67

The largest margins appear where fixed prototypes fail most: on ImageNet-C (1000 classes, so batches miss most classes), ReNC improves $\mathcal{NC}_1$8 from 21.37% to 31.02% over the strongest prior method, which the authors attribute directly to the stochastic prototype update mechanism. Notably, OSTTA and UniEnt frequently underperform plain TENT despite being OWTTA-specific, suggesting their GMM- and confidence-based partitioning is poorly matched to entropy optimization; ReNC avoids this by performing partitioning and adaptation in the same latent space. Per-sample runtime is competitive (0.0019–0.0041 s, roughly 2–3× TEST but faster than OWT3 on ImageNet-C). Robustness checks include stable performance across the trade-off parameter $\mathcal{NC}_1$9, consistent results for OOD ratios from 0.2 to 1.0, best average $1.1\times10^{-3}$0 (82.26 vs. 73.72 for OWT3) under continual OWTTA with sequentially changing OOD sets, consistent superiority with a ViT backbone, and a Friedman/Nemenyi test over 627 instances confirming statistically significant differences ($1.1\times10^{-3}$1).

Connection between TTA and neural collapse

A diagnostic analysis measures $1.1\times10^{-3}$2 and $1.1\times10^{-3}$3 after adaptation. Across corrupted domains, lower NC metrics generally accompany higher ID accuracy, and ReNC attains the lowest $1.1\times10^{-3}$4/$1.1\times10^{-3}$5 on most benchmarks—for example, $1.1\times10^{-3}$6 of 0.573 versus 0.738 for the unadapted model on CIFAR10-C + Noise. The correlation is not universal: on VisDA-C, SHOT achieves strong collapse metrics yet poor ID accuracy, sometimes below the unadapted model, which the authors attribute to the disjoint train/test splits of VisDA causing classifier–feature misalignment. The paper accordingly frames NC preservation as facilitating but not solely determining adaptation quality—an appropriately hedged claim.

Ablations and limitations

Ablations show that the alignment loss $1.1\times10^{-3}$7 degrades performance when prototypes remain fixed (e.g., dropping CIFAR100-C + MNIST $1.1\times10^{-3}$8), confirming the paper's central argument that prototype updating is essential rather than auxiliary. Conversely, entropy or balance losses alone with prototype updates also underperform the full objective, indicating the components are complementary. Several limitations deserve note. The method depends on Assumption 1 (the source model has collapsed), which holds only approximately for large-scale models as the ImageNet $1.1\times10^{-3}$9 measurement shows; behavior under poorly converged source models is unexamined. The adaptive threshold assumes ID/OOD similarity scores are separable into two clusters, which may fail when OOD samples semantically resemble ID classes. Evaluation uses a fixed OOD ratio of 1 in main comparisons, and the VLM experiment requires grafting an external ResNet-50 detector onto C-TPT, leaving open how native OOD rejection could be incorporated into vision-language backbones. The balance prior implicitly assumes near-uniform target class proportions, an assumption the paper does not stress-test under severe class imbalance.

Conclusion

ReNC contributes a theoretically grounded link between neural collapse and OWTTA: classifier weights serve as source prototypes under collapse, and reliable adaptation consists of filtering OOD samples via adaptive thresholding while jointly refining prototypes and parameters to approximate a target-domain collapse geometry. The empirical record—consistent state-of-the-art harmonic accuracy, particular strength in high-class-count and continual settings, preserved NC metrics, and statistical significance testing—supports the framework. The main open questions are robustness to weakly collapsed source models, non-Gaussian or semantically close OOD distributions, and integration with foundation-model backbones, which the authors identify as future work.