Ramen: Robust Test-Time Adaptation of Vision-Language Models with Active Sample Selection

Abstract: Pretrained vision-LLMs such as CLIP exhibit strong zero-shot generalization but remain sensitive to distribution shifts. Test-time adaptation adapts models during inference without access to source data or target labels, offering a practical way to handle such shifts. However, existing methods typically assume that test samples come from a single, consistent domain, while in practice, test data often include samples from mixed domains with distinct characteristics. Consequently, their performance degrades under mixed-domain settings. To address this, we present Ramen, a framework for robust test-time adaptation through active sample selection. For each incoming test sample, Ramen retrieves a customized batch of relevant samples from previously seen data based on two criteria: domain consistency, which ensures that adaptation focuses on data from similar domains, and prediction balance, which mitigates adaptation bias caused by skewed predictions. To improve efficiency, Ramen employs an embedding-gradient cache that stores the embeddings and sample-level gradients of past test images. The stored embeddings are used to retrieve relevant samples, and the corresponding gradients are aggregated for model updates, eliminating the need for any additional forward or backward passes. Our theoretical analysis provides insight into why the proposed adaptation mechanism is effective under mixed-domain shifts. Experiments on multiple image corruption and domain-shift benchmarks demonstrate that Ramen achieves strong and consistent performance, offering robust and efficient adaptation in complex mixed-domain scenarios. Our code is available at https://github.com/baowenxuan/Ramen .

• The paper introduces an active sample selection mechanism that selects support samples based on domain consistency and class balance to enable robust test-time adaptation.
• It leverages an efficient embedding-gradient cache to update normalization layers, providing near-real time adaptation without increased computational cost.
• Empirical results demonstrate significant accuracy gains on benchmarks such as CIFAR-100-C and ImageNet-C, confirming its scalability and robustness in mixed-domain settings.

Ramen: Robust Test-Time Adaptation of Vision-LLMs with Active Sample Selection

Problem Motivation and Background

Test-time adaptation (TTA) seeks to improve the robustness of pretrained models, particularly vision-LLMs (VLMs) such as CLIP, when exposed to distribution or domain shifts during inference. While VLMs exhibit strong zero-shot generalization, their predictions deteriorate under shifts such as image corruption or cross-domain variations. Most state-of-the-art TTA approaches assume a single, consistent domain during testing, which is inconsistent with practical settings where samples may stem from diverse, interleaved domains. Existing methods thus experience substantial performance degradation in these mixed-domain scenarios, as the adaptation process is confounded by the lack of domain homogeneity in the test stream.

Ramen Framework and Methodology

The proposed Ramen framework systematically addresses the core challenges of mixed-domain TTA in VLMs through an active sample selection mechanism. Instead of naive adaptation over potentially heterogeneous input batches, Ramen constructs for each test instance a tailored support set from prior test samples. The support set retrieval is guided by two crucial principles:

• Domain Consistency: Only samples that are similar in embedding space to the query (indicative of shared domain) are considered for adaptation. Empirical analyses support that embedding similarity strongly correlates with domain similarity (see Appendix).
• Prediction Balance: The support set is explicitly balanced across predicted classes, preventing adaptation bias towards any one class and preserving discriminative features necessary for robust learning.

To achieve computational tractability, Ramen employs an embedding-gradient cache, which retains both image embeddings and per-sample gradients for all observed test samples. For each new query, relevant support samples are retrieved using embedding similarity, and the corresponding cached gradients are aggregated to update the model's normalization layers. This approach sidesteps redundant forward/backward passes and preserves online inference speed comparable to near real-time adaptation.

Figure 1: Overview of the Ramen pipeline, illustrating sample embedding, memory update, support set retrieval, efficient gradient aggregation, and inference.

Theoretical Foundations

The analysis establishes that entropy minimization—common in TTA—updates normalization affine parameters by maximizing the importance of features with high class discriminativeness and within-domain variance. However, in the presence of mixed domains, high variance may stem from domain-specific (not class-specific) features, leading to suboptimal adaptation or outright performance collapse. Ramen's selective retrieval mechanism minimizes domain-induced variance for irrelevant features and maintains variance for class-relevant ones, enabling entropy minimization to amplify the correct feature space directions during adaptation.

Specifically:

• The per-dimension affine parameter update is proportional to $(t_{1h} - t_{0h})^2 \cdot M_{hh}$, where $(t_{1h} - t_{0h})$ is the discriminative power of the $h$-th feature in text space, and $M_{hh}$ is the probability-weighted variance in the support set.
• By actively enforcing domain consistency and class balance in the support set, non-discriminative, domain-variant features do not spuriously acquire importance, preventing overfitting to non-stationary or irrelevant domain noise.

Empirical Results

Ramen decisively outperforms competing TTA methods across a battery of large-scale benchmarks, including CIFAR-10-C, CIFAR-100-C, ImageNet-C, and DomainNet, under fully interleaved mixed-domain settings. On CIFAR-100-C (ViT-B/16), Ramen yields a +3.4% improvement in accuracy over the best baseline. On ImageNet-C (ViT-L/14), the gain is 2.5%. The performance is robust across encoder architectures (ViT, ResNet) and scales.

Figure 2: Performance of TTA methods under single-domain and mixed-domain shifts on CIFAR-100-C. Ramen sustains accuracy under domain mixture, unlike competing baselines.

Notably, Ramen achieves these gains without increased computational overhead—its embedding-gradient cache enables a 490× speedup relative to naive per-sample adaptation, with GPU memory usage remaining practical.

Further analyses demonstrate the mechanism of retrieval:

Figure 3: Proportion of retrieved support samples from each domain; 40.9% are from the same domain as the query (random baseline: 6.7%), verifying targeted selection.

Hyperparameter Robustness

Extensive experiments reveal that Ramen remains superior across a wide range of hyperparameter settings—including memory size ($K$), support set size ($k$), similarity scaling parameter ($\beta$), learning rate, and batch size—implying minimal tuning burden relative to baseline methods.

Figure 4: Sensitivity of Ramen and entropy minimization (EntMin) to hyperparameter choices; Ramen consistently maintains higher accuracy.

Ablation Analyses

Variants with either domain consistency or prediction balance ablated demonstrate significant performance drops or instabilities, confirming the necessity of both selection criteria.

Figure 5: Ablation results confirm that both domain consistency (DC) and prediction balance (PB) are critical for robust mixed-domain TTA.

Implications and Future Directions

Ramen provides an efficient, theoretically-underpinned methodology for robustly adapting VLMs in deployment environments where domain boundaries are unknown, unlabeled, and frequently crossed—reflecting realistic end-user scenarios (e.g., aggregated online image streams, robotics perception, or mobile applications). Its reliance on per-sample embedding–gradient caches suggests generalizability to broader self-supervised or federated TTA tasks, especially those involving streaming, non-i.i.d. test inputs.

The conceptual separation of domain- and class-related feature adaptation could inspire future research into finer-grained TTA objectives, e.g., support set construction that dynamically estimates and exploits latent hierarchical domain structures or leverages transductive clustering of support samples.

Conclusion

Ramen introduces a principled and scalable approach to TTA for pretrained VLMs under mixed-domain shift, leveraging active support set selection based on domain affinity and prediction balance, combined with a computationally-optimized cache architecture. Both theoretical and empirical analyses validate its superiority and robustness over prior approaches, especially in conditions reflecting practical, heterogeneous test-time environments. These findings may serve as a methodological blueprint for future advances in adaptive, self-supervised, and out-of-distribution robust perception systems.