RETTA: Retrieval-Enhanced Test-Time Adaptation

• RETTA is a family of test-time adaptation methods that use retrieval techniques to supply auxiliary evidence for more robust model predictions under distribution shifts.
• These methods retrieve diverse substrates—such as prior target samples, external images, and specialist parameters—to refine predictions and update adaptation objectives in real time.
• Empirical results demonstrate that RETTA improves accuracy and stability in areas like image classification, recommendation systems, and clinical QA, despite challenges in memory reliability.

Searching arXiv for papers on retrieval-enhanced test-time adaptation and closely related methods. Retrieval-Enhanced Test-Time Adaptation, or “RETTA” (Editor’s term), denotes a family of inference-time adaptation methods in which retrieval is used to supply auxiliary evidence, memory, or reusable adaptation state for a model facing distribution shift. In this family, retrieval may operate over a growing support set of prior test samples, an external unlabeled pool, a bank of domain-specialized parameter states, a hybrid cache of historical and query-local augmentations, a training-derived collaborative memory, or libraries of skills and experience. The retrieved objects are then used to denoise pseudo-labels, shape contrastive objectives, route inputs to specialists, refine predictions, or drive lightweight per-instance parameter updates. Early and representative instances include nearest-neighbor support retrieval in TAST (Jang et al., 2022), external-sample retrieval in T$^3$AR (Zancato et al., 2023), hybrid cache retrieval for CLIP adaptation in BoostAdapter (Zhang et al., 2024), model-state retrieval in ReservoirTTA (Vray et al., 20 May 2025), and dual-memory retrieval with test-time reasoning adaptation in TARSE (Wang et al., 1 Mar 2026).

1. Conceptual scope and problem formulation

RETTA inherits the standard test-time adaptation setting: a pretrained model is deployed under target-domain shift, target labels are unavailable, and adaptation must occur during inference. In the classical source-free formulation, the model has access only to unlabeled target data and not to source training data or labels at deployment. Retrieval augments this regime by introducing an additional nonparametric channel through which the model can consult stored examples, prototypes, specialists, or trajectories at test time (Jang et al., 2022).

Within this umbrella, “retrieval” is narrower and more heterogeneous than in conventional retrieval-augmented generation. TAST retrieves nearest support examples from an online memory of prior target samples rather than from an external knowledge base (Jang et al., 2022). T$^3$AR retrieves real external images from an auxiliary unlabeled pool and uses them to alter the adaptation objective (Zancato et al., 2023). ReservoirTTA retrieves a domain-specialized adaptation state by comparing the current batch’s style descriptor to a bank of learned style centroids (Vray et al., 20 May 2025). ReAd retrieves collaborative item signals from a training-derived memory of historical sequence representations and target item embeddings (Tang et al., 7 Apr 2026). TARSE retrieves two distinct resources—skills and verified experience traces—and uses only the latter for gradient-based adaptation (Wang et al., 1 Mar 2026).

This broader view implies that RETTA is not defined by a single mechanism such as $k$-NN over features. A method belongs to this family when retrieval materially changes the adaptation dynamics or inference rule. By contrast, some recent TTA methods are relevant to RETTA mainly as adjacent baselines or composable components rather than as retrieval-based methods themselves: EPOTTA supplies a calibration-oriented, sampling-free adaptation objective without retrieval (Han et al., 26 May 2025), and TTT-NTP supplies a prompt-derived fast-weight adaptation rule without any retrieval module (Ouyang et al., 19 Jun 2026). RTTC occupies an intermediate position, because it adaptively selects among direct inference, RAG, and TTT, but its defining contribution is reward-guided strategy selection rather than a fixed retrieve-and-adapt pipeline (Muñoz et al., 7 Aug 2025).

2. Memory substrates and retrieval targets

RETTA systems differ most sharply in what they store and what they retrieve. The stored object may be an unlabeled target feature, a pseudo-labeled cache entry, an external exemplar, a parameter state, or a structured reasoning artifact. This diversity is visible across vision, retrieval, recommendation, multimodal misinformation detection, and reasoning-agent settings (Zhang et al., 2024).

Method	Retrieved object	Immediate use
TAST	Prior target support features	Neighbor pseudo-labels and prediction averaging
T$^3$AR	External unlabeled images	Contrastive negatives for adaptation
BoostAdapter	Historical samples and regional views	Cache logits for CLIP prediction
ReservoirTTA	Domain specialists via style centroids	Routing and specialist adaptation
ReAd	Historical item embeddings	Augmentation embedding and score fusion
TARSE	Experience traces and skill rules	Test-time training and step verification

A first major substrate is the target-stream memory. TAST stores normalized backbone features in a class-partitioned support set $$\mathbb{S}_t=\{\mathbb{S}_t^1,\dots,\mathbb{S}_t^K\}$$, initialized from normalized classifier weights and updated online with entropy filtering (Jang et al., 2022). BoostAdapter and ReTA similarly maintain dynamic caches over historical test samples for CLIP-like models, while RADAR keeps a FIFO bank of recent target videos from which semantically similar and low-entropy references are retrieved (Zhang et al., 2024, Liang et al., 13 Jul 2025, Lang et al., 17 Jan 2026).

A second substrate is the external auxiliary pool. T$^3$AR indexes about 2M unlabeled images drawn from datasets such as ImageNet-1k, iNaturalist 2019, Food-101, Logo 2k+, NWPU-RESISC45, and iMaterialist Product, then retrieves nearest neighbors per target image using a fixed encoder, typically CLIP (Zancato et al., 2023). The retrieved images are not used as labeled supervision and need not share the target label space.

A third substrate is the specialist reservoir. ReservoirTTA stores one domain-specialized trainable parameter set per discovered domain, $\{\theta_t^1,\dots,\theta_t^{K_t}\}$, together with style centroids $$\mathbf{C}_t=[\mathbf{c}_t^1,\dots,\mathbf{c}_t^{K_t}]$$ and a style reservoir maintained by reservoir sampling (Vray et al., 20 May 2025). Retrieval in this setting is effectively key–value routing: the query is the current batch style vector, the key is a centroid, and the retrieved value is a specialist parameter state.

A fourth substrate is the structured library. TARSE separates procedural memory into a skills library of executable clinical rules and an experience library of verified reasoning trajectories $\mathcal{E}=\{(q_i,y_i,\mathbf{z}_i)\}_{i=1}^N$, where $\mathbf{z}_i$ is a step-indexed logical chain (Wang et al., 1 Mar 2026). This separation is atypical within TTA and makes retrieval itself heterogeneous inside a single system.

3. Adaptation mechanisms induced by retrieval

The defining property of RETTA is not storage alone but the way retrieval changes adaptation targets, losses, or prediction rules. TAST provides the cleanest nearest-neighbor formulation. For a test example $^3$0, it retrieves nearby support examples $^3$1 in frozen backbone space, forms a pseudo-label distribution from a histogram of the neighbors’ prototype-based argmax labels, and trains lightweight adaptation modules $^3$2 to match that neighbor-induced pseudo-label distribution rather than the model’s own direct prediction (Jang et al., 2022). The relevant update target is

$^3$3

and adaptation minimizes cross-entropy to the module’s current prototype prediction. Retrieval is then used again at inference by averaging prototype-based class distributions over the retrieved neighbors.

Other RETTA mechanisms are structurally different. T$^3$4AR retrieves external images and inserts them into a retrieval-augmented contrastive loss, thereby changing the negative/noise distribution used for feature adaptation (Zancato et al., 2023). BoostAdapter performs training-free retrieval over a joint cache of historical target samples and current-image regional augmentations, using similarity-weighted pseudo-label values to produce adapted logits (Zhang et al., 2024). ReTA likewise uses a visual cache for similarity-weighted label aggregation, but further modifies cache construction through Consistency-aware Entropy Reweighting, $^3$5, and augments final prediction with Diversity-driven Distribution Calibration over class-wise text distributions (Liang et al., 13 Jul 2025).

ReservoirTTA changes adaptation through retrieval and routing rather than neighbor labeling. The current batch style descriptor $^3$6 is compared against style centroids by

$^3$7

the top specialist $^3$8 is adapted, and a soft parameter ensemble $^3$9 is used for prediction (Vray et al., 20 May 2025). Retrieval here changes which parameters are updated and which parameters are used.

DART instantiates retrieval-enhanced adaptation in dense reranking. The top-ranked documents serve as pseudo-positives, the bottom-ranked documents as pseudo-negatives, and a bilinear scoring matrix $k$0 is updated for each query using a confidence-weighted margin loss,

$k$1

where $k$2 and $k$3 are confidence-weighted aggregate scores over retrieved documents (Liu et al., 31 May 2026). In this case the retrieved list is both the task output and the adaptation supervision.

ReAd and TARSE show two further endpoints. ReAd turns retrieved item embeddings into an augmentation embedding through cross-attention and fuses retrieval-derived and backbone predictions through entropy-aware weighting (Tang et al., 7 Apr 2026). TARSE uses retrieved experience traces as supervised adaptation data,

$k$4

then performs step-aware skills retrieval on provisional reasoning transitions $k$5 to verify or repair intermediate logic (Wang et al., 1 Mar 2026).

4. Empirical behavior across domains

In visual classification under source-free or source-light shift, RETTA methods typically show their largest gains when standard pseudo-labeling or entropy minimization becomes unreliable. TAST improved average accuracy on ERM-trained domain-generalization backbones from $k$6 to $k$7 on ResNet-18 and from $k$8 to $k$9 on ResNet-50, and improved over T3A by $^3$0 and $^3$1 on those two backbones, respectively (Jang et al., 2022). T$^3$2AR was strongest in low-target-data TTA regimes: on DomainNet-126 it achieved $^3$3 average accuracy at $^3$4 target data, and on VisDA-C $^3$5, with the largest gains at $^3$6 and $^3$7 (Zancato et al., 2023).

Cache-centric CLIP methods show a progression toward richer retrieval design. BoostAdapter, which combines historical target memory with instance-aware regional bootstrapping, reached a ViT-B/16 OOD average of $^3$8 versus $^3$9 for TDA and a cross-domain average of $$\mathbb{S}_t=\{\mathbb{S}_t^1,\dots,\mathbb{S}_t^K\}$$0 versus $$\mathbb{S}_t=\{\mathbb{S}_t^1,\dots,\mathbb{S}_t^K\}$$1 (Zhang et al., 2024). ReTA, which targets reliability failures in cache construction and decision calibration, reached $$\mathbb{S}_t=\{\mathbb{S}_t^1,\dots,\mathbb{S}_t^K\}$$2 average on cross-dataset generalization for CLIP-ViT-B/16 and $$\mathbb{S}_t=\{\mathbb{S}_t^1,\dots,\mathbb{S}_t^K\}$$3 average on natural distribution shifts, outperforming DPE, BoostAdapter, and TDA in those reported settings (Liang et al., 13 Jul 2025).

In prolonged nonstationary streams, retrieval of parameter states rather than exemplars becomes central. ReservoirTTA was designed for recurring continual structure change, recurring continual dynamic change, and continuously changing corruptions. On CIFAR-10-C recurring CSC, TENT degraded from $$\mathbb{S}_t=\{\mathbb{S}_t^1,\dots,\mathbb{S}_t^K\}$$4 error at visit 1 to $$\mathbb{S}_t=\{\mathbb{S}_t^1,\dots,\mathbb{S}_t^K\}$$5 at visit 20, whereas TENT + ReservoirTTA stayed around $$\mathbb{S}_t=\{\mathbb{S}_t^1,\dots,\mathbb{S}_t^K\}$$6 at visit 20; on ImageNet-C recurring CSC, EATA improved from $$\mathbb{S}_t=\{\mathbb{S}_t^1,\dots,\mathbb{S}_t^K\}$$7 to $$\mathbb{S}_t=\{\mathbb{S}_t^1,\dots,\mathbb{S}_t^K\}$$8 with ReservoirTTA (Vray et al., 20 May 2025). These results support the claim that retrieving domain-specialized states can mitigate catastrophic forgetting under recurrence.

Outside image classification, RETTA retains the same logic but changes substrate. DART improved dense retrieval reranking on five of six BEIR datasets, raising average NDCG@10 from $$\mathbb{S}_t=\{\mathbb{S}_t^1,\dots,\mathbb{S}_t^K\}$$9 to $^3$0, a mean per-dataset relative gain of $^3$1, with under 10 ms additional latency per query (Liu et al., 31 May 2026). ReAd achieved the best reported performance across five sequential recommendation datasets when attached to DuoRec, for example reaching Office HR@10 $^3$2 and ML-1M HR@20 $^3$3, while outperforming TTA and RaSeRec in the main table (Tang et al., 7 Apr 2026). RADAR, under unseen-topic fake-news video detection, reported average gains of $^3$4 Macro-F1 under random batch sampling and $^3$5 under event-wise batch sampling, with ablations showing that removing semantic retrieval or entropy-based stable-reference selection substantially degrades performance (Lang et al., 17 Jan 2026). TARSE, in clinical QA, reached MedQA $^3$6 with Qwen2.5-7B and $^3$7 with Qwen2.5-14B, substantially above CoT, MedRAG, i-MedRAG, and rStar baselines in the reported comparison (Wang et al., 1 Mar 2026).

5. Failure modes, reliability issues, and common misconceptions

A recurring misconception is that retrieval in RETTA always means external document retrieval. The surveyed methods contradict this directly. Retrieval may target prior test samples, support-set features, style centroids, specialist parameters, regional self-augmentations, recommender memories, or reasoning trajectories (Jang et al., 2022, Vray et al., 20 May 2025, Zhang et al., 2024, Wang et al., 1 Mar 2026). What unifies these systems is not the source of retrieved data but the fact that retrieval changes adaptation or prediction under shift.

A second misconception is that more retrieval is always better. Several papers report the opposite. TAST found on CIFAR-10C that smaller $^3$8 often works better, suggesting that too many neighbors dilute label purity (Jang et al., 2022). ReAd shows the same qualitative pattern with retrieval depth $^3$9: moderate retrieval enriches collaborative evidence, whereas larger $\{\theta_t^1,\dots,\theta_t^{K_t}\}$0 introduces noise (Tang et al., 7 Apr 2026). DART likewise shows that naive per-query adaptation can hurt, as the FiQA base online update dropped to $\{\theta_t^1,\dots,\theta_t^{K_t}\}$1 before EMA and optimizer choices recovered performance (Liu et al., 31 May 2026).

Reliability of the memory itself is a major bottleneck. TAST states that good prototypes and reliable neighborhoods are required, and reports a severe cold-start problem on ImageNet-C: under Gaussian noise it took 768 out of 782 batches before all prototypes were updated at least once (Jang et al., 2022). ReTA is built around the observation that raw entropy is unreliable under visual shift and can admit confidently wrong samples into the cache, producing error accumulation (Liang et al., 13 Jul 2025). RADAR similarly assumes that low-entropy target instances are more stable and source-close, but explicitly notes that some queries may have no stable references after entropy filtering (Lang et al., 17 Jan 2026).

Retriever quality and coverage are equally critical. T$\{\theta_t^1,\dots,\theta_t^{K_t}\}$2AR reports that performance falls as the domain gap between the external pool $\{\theta_t^1,\dots,\theta_t^{K_t}\}$3 and the adaptation domain increases, and that retriever quality matters, with CLIP retrieval outperforming DINO and random retrieval in its ablations (Zancato et al., 2023). ReservoirTTA assumes that low-level style statistics from early VGG-19 layers capture domain identity well enough for routing; if style is ambiguous, routing may fail, and scalability remains bounded by $\{\theta_t^1,\dots,\theta_t^{K_t}\}$4 (Vray et al., 20 May 2025). DART depends on the initial dense ranking being good enough that top-ranked documents are plausible positives and bottom-ranked documents within top-$\{\theta_t^1,\dots,\theta_t^{K_t}\}$5 are plausible negatives, so it remains exposed to confirmation bias (Liu et al., 31 May 2026).

These observations suggest a general RETTA design principle: retrieval quality, reliability estimation, and memory management are first-order components, not implementation details. Several methods attempt to address exactly this point through entropy filtering, consistency checks, style clustering, confidence weighting, or bounded caches, but none remove the dependence on retriever quality altogether (Zhang et al., 2024, Liang et al., 13 Jul 2025).

6. Relation to adjacent directions and emerging design patterns

RETTA sits at the intersection of TTA, memory-based inference, and retrieval augmentation, but adjacent work indicates that retrieval is only one axis of the design space. EPOTTA shows that a stronger adaptation objective can be obtained by replacing entropy minimization with a sampling-free energy-ratio preference objective, yet the method remains non-retrieval-based and uses only a small source replay buffer (Han et al., 26 May 2025). TTT-NTP shows that in long-context LLMs, test-time adaptation may benefit from aligning the fast-weight write target with next-token prediction via the next same-layer contextual hidden state, again without retrieval (Ouyang et al., 19 Jun 2026). RTTC shows that retrieval and adaptation can also be treated as alternative test-time compute strategies selected by a reward model rather than as obligatorily coupled modules (Muñoz et al., 7 Aug 2025).

From the retrieval-centered papers themselves, several cross-cutting patterns emerge. One is the move from single-memory to hybrid-memory systems: BoostAdapter combines historical and instance-local memory (Zhang et al., 2024), ReTA combines visual cache retrieval with text-side distribution calibration (Liang et al., 13 Jul 2025), and TARSE combines experience and skills (Wang et al., 1 Mar 2026). Another is the move from sample retrieval to state retrieval: ReservoirTTA retrieves adaptation states rather than instances (Vray et al., 20 May 2025). A third is increasing query locality: DART recycles the current query’s ranked list, and ReAd retrieves collaborative signals keyed by the current sequence representation (Liu et al., 31 May 2026, Tang et al., 7 Apr 2026).

A plausible implication is that future RETTA systems will combine three ingredients more systematically: instance- or step-conditioned retrieval, reliability-aware memory curation, and lightweight but task-aligned adaptation objectives. The surveyed work already supplies exemplars of each component—retrieval-conditioned pseudo-labeling in TAST (Jang et al., 2022), reliability-aware cache construction in ReTA (Liang et al., 13 Jul 2025), reward-gated retrieval/adaptation selection in RTTC (Muñoz et al., 7 Aug 2025), and reasoning-trace adaptation in TARSE (Wang et al., 1 Mar 2026). What remains uneven across the literature is the degree of integration: many systems still use fixed retrieval spaces, simple thresholding, limited memory management, or narrow adaptation targets.

Taken together, RETTA is best understood not as a single algorithm but as a research program: use retrieval to expose nonparametric evidence or reusable adaptation state at test time, and use that retrieved object to stabilize or specialize inference under shift. The main conceptual advance across the cited papers is that test-time adaptation need not rely solely on a model’s own uncertain predictions; it can instead be grounded in retrieved neighbors, cached target experience, external exemplars, specialist parameters, collaborative memories, or structured procedural traces, depending on the deployment regime (Jang et al., 2022, Zancato et al., 2023, Vray et al., 20 May 2025, Wang et al., 1 Mar 2026).