ProtoTTA: Prototype-Guided Test-Time Adaptation

Abstract: Deep networks that rely on prototypes-interpretable representations that can be related to the model input-have gained significant attention for balancing high accuracy with inherent interpretability, which makes them suitable for critical domains such as healthcare. However, these models are limited by their reliance on training data, which hampers their robustness to distribution shifts. While test-time adaptation (TTA) improves the robustness of deep networks by updating parameters and statistics, the prototypes of interpretable models have not been explored for this purpose. We introduce ProtoTTA, a general framework for prototypical models that leverages intermediate prototype signals rather than relying solely on model outputs. ProtoTTA minimizes the entropy of the prototype-similarity distribution to encourage more confident and prototype-specific activations on shifted data. To maintain stability, we employ geometric filtering to restrict updates to samples with reliable prototype activations, regularized by prototype-importance weights and model-confidence scores. Experiments across four prototypical backbones on four diverse benchmarks spanning fine-grained vision, histopathology, and NLP demonstrate that ProtoTTA improves robustness over standard output entropy minimization while restoring correct semantic focus in prototype activations. We also introduce novel interpretability metrics and a vision-LLM (VLM) evaluation framework to explain TTA dynamics, confirming ProtoTTA restores human-aligned semantic focus and correlates reliably with VLM-rated reasoning quality. Code is available at: https://github.com/DeepRCL/ProtoTTA.

• The paper presents a novel prototype-guided framework that leverages internal prototype signals to restore semantic interpretability under domain shifts.
• It employs reliable sample selection, a prototype-guided loss, and consensus aggregation to optimize adaptation while mitigating spurious activations.
• Experimental results across vision and NLP tasks demonstrate improved accuracy, efficiency, and preservation of human-aligned reasoning compared to standard TTA methods.

Prototype-Guided Test-Time Adaptation for Interpretable Deep Models

Introduction

ProtoTTA introduces a prototype-driven framework for test-time adaptation (TTA) in deep neural networks that employ interpretable prototype-based mechanisms. The motivation arises from the vulnerability of prototype models to distribution shift: while these architectures offer localized, human-interpretable evidence for predictions, their prototype selection is notably brittle when test samples differ from training data. Prior TTA work has largely addressed black-box models, updating only normalization statistics or output logits. ProtoTTA is the first to leverage the internal, semantically-grounded signals present in prototype activations, optimizing adaptation not just for robustness but also to explicitly restore interpretable, class-aligned focus in the adapted model under covariate shift.

Methodology

Prototype networks, e.g., ProtoPNet, ProtoViT, or ProtoLens, compute class evidence through learned prototype representations compared to input features. Under domain shift, such comparisons degrade: correct prototype matches are suppressed, and spurious activations are amplified, severing the chain of semantic reasoning.

ProtoTTA tackles this by minimizing the binary entropy of the (normalized) prototype similarity scores, $\{S_{ip}\}$, for the prototypes $p$ associated with each test input $x_i$. The process comprises several distinct components:

• Reliable Sample Selection (Geometric Filtering): Only samples with decisive, high-confidence prototype activations (max similarity exceeding threshold $T$) and optionally low-entropy in output predictions are used for parameter updates, avoiding adaptation to ambiguous or corrupted instances that could inject instability.
• Prototype-Guided Loss: For each reliable sample and predicted class, ProtoTTA computes a weighted binary entropy over the set of associated prototypes. Weights are a function of both sample confidence and the class-prototype importances from the trained classification head. The loss:

$$\mathcal{L}_\text{ProtoTTA} = \frac{1}{|R|}\sum_{i\in R} c_i \sum_{p\in P_{t(i)}} w_p H(S_{ip})$$

where $H(\cdot)$ denotes binary entropy, $P_{t(i)}$ are prototypes for the pseudo-label $t(i)$, $w_p$ is the prototype's class weight, and $c_i$ the sample confidence.

• Consensus Aggregation: For models with sub-prototypes, Top-K Mean pooling aggregates the most relevant sub-prototype activations, avoiding the sensitivity to outliers of max pooling.
• Structural Targeted Updates: Only normalization statistics and structural add-ons (attention biases, 1×1 convolutions) are updated, preventing catastrophic forgetting and ensuring semantic recalibration rather than wholesale parameter drift.

These operations are architecture-agnostic, applicable to both CNN- and Transformer-based prototype models.

Experimental Results

Benchmarks and Evaluation

ProtoTTA was evaluated on fine-grained visual classification (CUB-200-C, Stanford Dogs-C), medical histopathology (SICAPv2-C), and NLP sentiment tasks (Amazon-C). Prototype backbones included ProtoViT, ProtoPNet, ProtoPFormer, and ProtoLens. Test distributions involved both synthetic corruptions (e.g., Gaussian/shot noise, blur, compression artifacts) and real-world shifts.

Evaluation prioritized not only classification accuracy under shift but also interpretability preservation, employing three key metrics:

• Prototype Activation Consistency (PAC): Cosine similarity between clean and adapted prototype activations, measuring semantic stability.
• Weighted Prototype Alignment (PCA-W): Degree to which the top activated prototypes correspond to ground-truth class prototypes, weighted by activation and head importance.
• Prediction Stability: Consistency of predictions pre- and post-adaptation.

Quantitative Findings

ProtoTTA consistently outperformed output-based TTA baselines (Tent, SAR, MEMO, EATA) on both accuracy and interpretability criteria across all domains. Notably:

• On CUB-200-C with ProtoViT, ProtoTTA achieved $p$0% accuracy (mean across corruptions), a $p$1-point improvement over EATA. Crucially, this was obtained with less than $p$2 of samples adapted, indicating superior adaptation efficiency.
• In Amazon-C NLP shifts with ProtoLens, ProtoTTA maintained both highest accuracy ($p$3%) and PCA-W prototype alignment, confirming transferability to sequence tasks.
• On SICAPv2-C (ProtoPNet) and Stanford Dogs-C (ProtoPFormer), a hybrid of ProtoTTA and standard logit-based adaptation (ProtoTTA+) yielded best-in-class accuracy, underscoring the complementarity of prototype- and logit-level adaptation mechanisms.

ProtoTTA was uniquely robust against catastrophic forgetting during continuous adaptation—per-batch performance in long corruption sequences showed no degradation, attributable to geometric filtering and judicious parameter selection.

Interpretability and Diagnostic Analysis

A distinguishing aspect of ProtoTTA is the restoration and traceability of human-aligned reasoning under shift:

• Visualizations of top activated prototypes and their spatial attentions confirmed that, post-adaptation, ProtoTTA reactivates discriminative, ground-truth class prototypes, closely mirroring the clean baseline. Competing methods frequently exhibited semantic hallucinations, manifest as erroneous matches to irrelevant or non-discriminative prototypes.
• VLM-based Evaluation: Using large vision-LLMs to rate focus relevance, prototype match quality, and overall adaptation quality, ProtoTTA achieved the highest scores across all axes. Notably, improvement in focus and prototype match metrics (+0.12 and +0.20 over EATA/unadapted, respectively) corroborates the claimed restoration of semantic alignment.
• Correlation between PCA-W and VLM-derived interpretability scores strengthened under ProtoTTA (r=0.68), indicating that suppression of noise-induced activations yields activations that genuinely reflect visual evidence, closing the "semantic gap" induced by spurious adaptation.

Practical and Theoretical Implications

ProtoTTA demonstrates that prototype-based interpretability is not only compatible with robust test-time adaptation but can directly enhance it. By operating on the intermediate, interpretable signals intrinsic to prototype architectures, adaptation is rendered not only more diagnostically transparent but also more precise, avoiding robustness-interpretability trade-offs characteristic of standard output-based adaptation.

Practically, this approach is particularly impactful in critical applications such as healthcare, where explanation fidelity under distribution shift is paramount. The language-based evaluation protocol offers a mechanism for automated, scalable auditing of model reasoning, tractable for clinical or regulatory acceptance.

Theoretically, the work motivates extension of TTA research to leverage intermediate model representations—such as structured concepts and part-based alignments—instead of being confined to the statistical properties of output layers. The findings also imply that many prior results on the fragility of interpretable models under shift can be mitigated with minimal, interpretable adaptation.

Limitations and Future Directions

While ProtoTTA exhibits state-of-the-art performance on diverse corruptions, some tasks—particularly blur—expose the limits of patch-based prototype matching, as critical local features are obfuscated. Future developments may integrate generative rather than solely discriminative adaptation signals, or blend black-box and prototype-guided strategies for extreme domain drift.

Integration with existing TTA methods is already demonstrated to be beneficial (ProtoTTA+), and future TTA research should systematically explore the joint space of adaptive mechanisms at both prototype and logit levels. Automatic calibration of geometric filtering thresholds, and deployment in resource-constrained or streaming environments, are practical follow-ons.

Furthermore, as foundation models in vision and language expand, prototype-centric adaptation promises a compatible path for explainable and controllable deployment, especially when paired with LLM-based reasoning audits.

Conclusion

ProtoTTA establishes a novel paradigm for robust, interpretable test-time adaptation by leveraging prototype similarity distributions and human-aligned reasoning signals as the update targets, rather than output logits alone. Experimental evidence on vision and language tasks demonstrates simultaneous improvements in accuracy, efficiency, and interpretability. The framework enables detailed, automatic diagnosis of adaptation behavior and aligns machine reasoning with both mathematical and human proxy evaluations. This work articulates a new direction for explainable adaptation in deep learning and sets a baseline for future advancements at the intersection of robustness and interpretability in critical domains (2604.15494).