Test-Time Memory Decay in Inference Systems

• Test-Time Memory Decay is a phenomenon where memory usefulness degrades during inference due to error accumulation, staleness, and redundancy.
• It spans various subfields such as test-time adaptation, continual learning, and agent systems, each using different operational formulations and metrics.
• Effective mitigation involves strategies like retention filtering, memory refresh, and compressed state consolidation to maintain system performance.

Test-time memory decay is a non-unified research concept describing the degradation of memory usefulness during inference or deployment, rather than during offline training alone. Across recent work, it refers to several closely related phenomena: error accumulation in test-time adaptation memory banks, staleness and redundancy in fixed-capacity buffers, loss of accessibility in long-horizon agent memory, overwrite of compressed fast weights, degradation of trustworthiness in self-evolving agent memory, and divergence in runtime memory behavior even among functionally correct programs. The literature therefore does not present a single canonical decay law; instead, it offers a family of operational formulations in which memory can become stale, contaminated, overconfident, redundant, misaligned, or inaccessible unless the test-time system explicitly controls retention, retrieval, update, or replay (Yu et al., 2024, Alhuwaider et al., 19 May 2026, Rana et al., 31 Mar 2026, P, 26 Feb 2026, Cheng et al., 3 Feb 2026).

1. Scope and conceptual variants

The term has different meanings in different subfields. In test-time adaptation for vision, the core issue is usually not explicit forgetting by a parametric decay law, but progressive degradation of replay quality under online unlabeled adaptation. In this setting, memory decay appears as error accumulation, contamination by unreliable or outlier samples, class imbalance, or instability under long online adaptation (Yu et al., 2024). In practical test-time adaptation under non-i.i.d. streams, the central problem is that recency alone is not enough: a recent sample may already be stale in an informational sense if it is a near-duplicate, while an older sample may remain valuable if it preserves class or feature-space coverage (Alhuwaider et al., 19 May 2026).

In continual learning with test-time data, the concept shifts from replay hygiene to memory refresh. The claim is that forgetting should be controlled not only during supervised updates, but also during deployment, because representative unlabeled test-time data can refresh prior-task competence when old-task samples reappear (Singh et al., 2024). In continual test-time adaptation, a related usage treats memory decay as gradual loss of source knowledge and accumulation of unsupervised errors over long horizons, motivating mechanisms that preserve source-domain information without replay buffers (Song et al., 2023).

In long-horizon agent systems and LLMs, memory decay is often formulated more explicitly. Oblivion defines forgetting as “decay-driven reductions in accessibility—not explicit deletion,” so memories remain stored but become less prominent in retrieval unless reinforced by actual use (Rana et al., 31 Mar 2026). DART-VLN uses the term directly for a read-side memory reweighting rule that suppresses stale and redundant evidence during navigation without rewriting stored content (Zhang et al., 1 Jul 2026). SR-TTT describes decay as overwrite of compressed fast weights during inference, where unique or surprising tokens are forgotten by subsequent token gradient updates unless routed to an exact residual cache (P, 26 Feb 2026). In agent benchmarks such as TAME and Evo-Memory, the degradation is less about loss of recall than about drift of evolving memory toward unsafe, noisy, or procedurally unhelpful content (Cheng et al., 3 Feb 2026, Wei et al., 25 Nov 2025).

A compact way to organize these usages is the following.

Domain	Primary notion of decay	Representative papers
Test-time adaptation	Error accumulation, contamination, staleness, redundancy	(Yu et al., 2024, Alhuwaider et al., 19 May 2026, Zhou et al., 2024)
Continual learning / continual TTA	Forgetting of prior tasks or source knowledge; memory refresh	(Singh et al., 2024, Song et al., 2023)
LLM agents and long-context reasoning	Accessibility decay, stale retrieval, noisy accumulation, search-state loss	(Rana et al., 31 Mar 2026, Lei et al., 9 Apr 2026, Wei et al., 25 Nov 2025)
Compressed inference memory	Overwrite of fast weights or unstable runtime memory profiles	(P, 26 Feb 2026, Rajput et al., 3 Jan 2026)
Embodied navigation	Stale historical evidence at memory readout	(Zhang et al., 1 Jul 2026)

This suggests that “test-time memory decay” is best treated as an umbrella term for inference-time degradation of memory quality, accessibility, representativeness, or operational stability, rather than as a single mathematically standardized object.

2. Memory decay in online test-time adaptation

In source-free test-time adaptation, a pretrained source model is adapted online using only unlabeled target data. The immediate challenge is that adaptation gradients are derived from uncertain predictions, so any unreliable batch can reinforce incorrect beliefs. STAMP makes this motivation explicit by noting that optimization on unreliable samples can “result in error accumulation over time,” especially when the test stream includes outliers or unknown classes (Yu et al., 2024). Its stable memory replay formulation therefore treats memory decay as progressive degradation of replay quality due to noisy admissions, class imbalance, and long-term parameter drift.

STAMP stores reliable test samples with pseudo-labels in a bounded memory bank $\mathcal M$, initialized as $\mathcal M=\emptyset$, together with a class-frequency vector $\{\xi_c\}_{c=1}^{|\mathcal C_S|}$. A sample is admitted only if it passes both consistency filtering,

$$m_{con}(x) = \mathbb{I}\!\left(\arg\max_c [\hat p(x)]_c = \arg\max_c [f_{\theta_s}(x)]_c \right),$$

and entropy filtering,

$$m_{ent}(x) = \mathbb{I}\!\left(\mathcal H(\hat p(x)) < \mathcal H_{thr}\right), \qquad \mathcal H(q) = -\sum_c q_c \log q_c.$$

This is not called “memory decay” in the paper, but it is the paper’s clearest mechanism for preventing it: only low-entropy, label-consistent, presumably in-distribution samples are allowed to persist in replay (Yu et al., 2024).

ResiTTA takes a related but more explicitly maintenance-oriented approach through its Entropy-driven Memory Bank. Each stored item is

$(x,\hat{y}, \alpha, e),$

where $x$ is the raw sample, $\hat y$ the inferred label, $\alpha$ the sample age, and $e$ the entropy. EntroBank increments age over time, defines outdated samples by

$\mathcal M=\emptyset$0

and long-persisted over-confident samples by

$\mathcal M=\emptyset$1

Its default thresholds are $\mathcal M=\emptyset$2 and $\mathcal M=\emptyset$3, so persistent low-entropy samples are typically targeted before they become formally outdated. The ablation shows that removing the outdated-sample strategy changes CIFAR100-C average error from $\mathcal M=\emptyset$4 to $\mathcal M=\emptyset$5, removing the long-persisted over-confident strategy changes it to $\mathcal M=\emptyset$6, and removing sample uncertainty changes it to $\mathcal M=\emptyset$7, indicating that persistent over-confidence and uncertainty filtering are more consequential than hard age expiration in the reported setting (Zhou et al., 2024).

EcoTTA addresses a different but adjacent form of decay: continual degradation of model state under long-horizon unsupervised adaptation. Its central claim is that continual TTA tends to induce catastrophic forgetting and error accumulation, and that these must be controlled under tight runtime memory budgets (Song et al., 2023). Instead of a replay bank, EcoTTA freezes the original model, adapts only lightweight meta networks, and uses self-distilled regularization

$\mathcal M=\emptyset$8

In the 100-round long-term experiment on CIFAR100-C with WRN-40, TENT reaches its best error of $\mathcal M=\emptyset$9 after 8 rounds but worsens to $\{\xi_c\}_{c=1}^{|\mathcal C_S|}$0 after 100 rounds, while EcoTTA with self-distilled regularization remains stable; EcoTTA without regularization also eventually overfits (Song et al., 2023). Here, test-time memory decay is effectively parameter drift rather than replay corruption.

3. Retention, eviction, and representativeness in memory buffers

A major recent shift is the explicit study of memory policy as a first-class component of test-time adaptation. GoTTA argues that prior work often bundled memory together with a specific adaptation loss, making it hard to isolate which memory design choices matter. It reformulates memory-based TTA as

$\{\xi_c\}_{c=1}^{|\mathcal C_S|}$1

where $\{\xi_c\}_{c=1}^{|\mathcal C_S|}$2 is the memory policy. In this view, decay is implicit in how a fixed-capacity buffer admits, refreshes, and evicts samples, rather than in a standalone forgetting coefficient (Alhuwaider et al., 19 May 2026).

The paper’s central insight is that under temporally correlated streams, “old” and “stale” are not the same thing. FIFO implements pure recency decay by always discarding the oldest samples. Reservoir sampling has no explicit recency bias. PBRS slows decay of minority classes by partitioning memory by pseudo-label. CSTU and GOTTA use score-based eviction involving age and uncertainty. GOTTA adds a second criterion—feature-space diversity—so that memory can also decay by redundancy rather than by age alone (Alhuwaider et al., 19 May 2026).

The shared eviction score inherited from CSTU is

$\{\xi_c\}_{c=1}^{|\mathcal C_S|}$3

where $\{\xi_c\}_{c=1}^{|\mathcal C_S|}$4 is age, $\{\xi_c\}_{c=1}^{|\mathcal C_S|}$5 is uncertainty, $\{\xi_c\}_{c=1}^{|\mathcal C_S|}$6 is the number of classes, and $\{\xi_c\}_{c=1}^{|\mathcal C_S|}$7 is memory capacity. Intra-class diversity is then enforced by feature-space filters. In the FPS variant, a candidate competes with its nearest same-class neighbor and is inserted only if it is sufficiently different: $\{\xi_c\}_{c=1}^{|\mathcal C_S|}$8 with $\{\xi_c\}_{c=1}^{|\mathcal C_S|}$9 by default. In FPSD, stored representations, pseudo-labels, and uncertainties are periodically refreshed every $$m_{con}(x) = \mathbb{I}\!\left(\arg\max_c [\hat p(x)]_c = \arg\max_c [f_{\theta_s}(x)]_c \right),$$0 steps, which is the paper’s most explicit anti-staleness mechanism. The empirical finding is that recency-only decay is insufficient: on continual temporal CIFAR-10-C streams with $$m_{con}(x) = \mathbb{I}\!\left(\arg\max_c [\hat p(x)]_c = \arg\max_c [f_{\theta_s}(x)]_c \right),$$1, Norm+FPS reaches $$m_{con}(x) = \mathbb{I}\!\left(\arg\max_c [\hat p(x)]_c = \arg\max_c [f_{\theta_s}(x)]_c \right),$$2, compared with FIFO at $$m_{con}(x) = \mathbb{I}\!\left(\arg\max_c [\hat p(x)]_c = \arg\max_c [f_{\theta_s}(x)]_c \right),$$3, Reservoir at $$m_{con}(x) = \mathbb{I}\!\left(\arg\max_c [\hat p(x)]_c = \arg\max_c [f_{\theta_s}(x)]_c \right),$$4, PBRS at $$m_{con}(x) = \mathbb{I}\!\left(\arg\max_c [\hat p(x)]_c = \arg\max_c [f_{\theta_s}(x)]_c \right),$$5, and CSTU at $$m_{con}(x) = \mathbb{I}\!\left(\arg\max_c [\hat p(x)]_c = \arg\max_c [f_{\theta_s}(x)]_c \right),$$6 (Alhuwaider et al., 19 May 2026).

ResiTTA and STAMP provide complementary design patterns. ResiTTA couples age, entropy, and class-balance-aware replacement. STAMP uses class-frequency-based eviction: $$m_{con}(x) = \mathbb{I}\!\left(\arg\max_c [\hat p(x)]_c = \arg\max_c [f_{\theta_s}(x)]_c \right),$$7 with $$m_{con}(x) = \mathbb{I}\!\left(\arg\max_c [\hat p(x)]_c = \arg\max_c [f_{\theta_s}(x)]_c \right),$$8, and when memory is full discards one sample from the class with the highest $$m_{con}(x) = \mathbb{I}\!\left(\arg\max_c [\hat p(x)]_c = \arg\max_c [f_{\theta_s}(x)]_c \right),$$9. This is not stale-sample removal in a temporal sense; it is a distributional anti-drift policy to prevent a frequent class from dominating replay (Yu et al., 2024). Across these papers, the emerging principle is that practical test-time memory decay is often caused more by unrepresentative retention and redundancy than by chronological age alone.

4. Explicit decay-driven inference-time memory control

Some recent systems define test-time memory decay directly as a runtime control rule. Oblivion is the clearest example in agentic memory. It defines forgetting as a reduction in accessibility, not deletion, through the cluster-level retention score

$$m_{ent}(x) = \mathbb{I}\!\left(\mathcal H(\hat p(x)) < \mathcal H_{thr}\right), \qquad \mathcal H(q) = -\sum_c q_c \log q_c.$$0

where $$m_{ent}(x) = \mathbb{I}\!\left(\mathcal H(\hat p(x)) < \mathcal H_{thr}\right), \qquad \mathcal H(q) = -\sum_c q_c \log q_c.$$1 is the number of turns since cluster $$m_{ent}(x) = \mathbb{I}\!\left(\mathcal H(\hat p(x)) < \mathcal H_{thr}\right), \qquad \mathcal H(q) = -\sum_c q_c \log q_c.$$2 was last accessed, $$m_{ent}(x) = \mathbb{I}\!\left(\mathcal H(\hat p(x)) < \mathcal H_{thr}\right), \qquad \mathcal H(q) = -\sum_c q_c \log q_c.$$3 is a learned utility proxy, $$m_{ent}(x) = \mathbb{I}\!\left(\mathcal H(\hat p(x)) < \mathcal H_{thr}\right), \qquad \mathcal H(q) = -\sum_c q_c \log q_c.$$4 is an access-frequency proxy, and $$m_{ent}(x) = \mathbb{I}\!\left(\mathcal H(\hat p(x)) < \mathcal H_{thr}\right), \qquad \mathcal H(q) = -\sum_c q_c \log q_c.$$5 is a decay temperature (Rana et al., 31 Mar 2026). Low-retention memories become less prominent in retrieval ranking and more likely to be evicted from the bounded working-memory buffer, but they are “never deleted from $$m_{ent}(x) = \mathbb{I}\!\left(\mathcal H(\hat p(x)) < \mathcal H_{thr}\right), \qquad \mathcal H(q) = -\sum_c q_c \log q_c.$$6.” The paper’s decay-temperature analysis shows a sawtooth pattern: retention drops by exponential decay and spikes upon reinforcement; $$m_{ent}(x) = \mathbb{I}\!\left(\mathcal H(\hat p(x)) < \mathcal H_{thr}\right), \qquad \mathcal H(q) = -\sum_c q_c \log q_c.$$7 leads to rapid decay, $$m_{ent}(x) = \mathbb{I}\!\left(\mathcal H(\hat p(x)) < \mathcal H_{thr}\right), \qquad \mathcal H(q) = -\sum_c q_c \log q_c.$$8 gives the best balance, and $$m_{ent}(x) = \mathbb{I}\!\left(\mathcal H(\hat p(x)) < \mathcal H_{thr}\right), \qquad \mathcal H(q) = -\sum_c q_c \log q_c.$$9 causes buffer saturation and behavior close to always-on retrieval (Rana et al., 31 Mar 2026).

DART-VLN introduces Test-Time Memory Decay as a training-free, read-side memory reweighting rule for discrete vision-language navigation. Each memory slot $(x,\hat{y}, \alpha, e),$0 is assigned metadata for age $(x,\hat{y}, \alpha, e),$1, visit count $(x,\hat{y}, \alpha, e),$2, and novelty $(x,\hat{y}, \alpha, e),$3, where novelty is computed from feature change: $(x,\hat{y}, \alpha, e),$4 with $(x,\hat{y}, \alpha, e),$5. The readout weight is then

$(x,\hat{y}, \alpha, e),$6

using $(x,\hat{y}, \alpha, e),$7, $(x,\hat{y}, \alpha, e),$8, $(x,\hat{y}, \alpha, e),$9, and $x$0 (Zhang et al., 1 Jul 2026). The crucial point is that stored slots are not rewritten; only their contribution at readout is decayed. On REVERIE val unseen, decay-only changes TL from $x$1 to $x$2, OSR from $x$3 to $x$4, SR from $x$5 to $x$6, SPL from $x$7 to $x$8, RGS from $x$9 to $\hat y$0, RGSPL from $\hat y$1 to $\hat y$2, and runtime from $\hat y$3 s to $\hat y$4 s (Zhang et al., 1 Jul 2026).

MemCoT does not define a decay law, but it addresses the same operational failure modes in long-context reasoning—catastrophic forgetting, semantic dilution, and contextual fragmentation—through an iterative stateful search loop. Long-term memory is perceived via Zoom-In and Zoom-Out retrieval,

$\hat y$5

and short-term memory evolves as

$\hat y$6

This preserves semantic findings and search trajectories across iterations instead of relying on a single large context. On LoCoMo with GPT-4o-mini, MemCoT reaches overall $\hat y$7 versus CompassMem at $\hat y$8, with especially large gains in Open-domain ($\hat y$9 vs $\alpha$0) and Temporal ($\alpha$1 vs $\alpha$2) categories (Lei et al., 9 Apr 2026). This suggests that many apparent long-context decay effects are failures of dynamic memory access and state persistence, not just window length.

5. Memory refresh, consolidation, and compressed test-time memory

A separate line of work treats test-time memory decay as the loss of previously acquired competence unless deployment-time data are used to refresh it. DoSAPP studies class-incremental learning where after each supervised task the model receives an unlabeled stream $\alpha$3 and updates a student model on pseudo-labels chosen by a more confident expert between teacher and student: $\alpha$4

$\alpha$5

The student then updates only a sparse parameter subset,

$\alpha$6

while the teacher follows by dual-rate EMA (Singh et al., 2024). The paper interprets this as online unsupervised memory refresh: forgetting is reduced when representative old-task samples reappear at test time. Empirically, DoSAPP achieves $\alpha$7 Acc and $\alpha$8 Forgetting on Aircraft, $\alpha$9 Acc and $e$0 Forgetting on Cars, $e$1 Acc and $e$2 Forgetting on CIFAR100, $e$3 Acc and $e$4 Forgetting on CUB, and $e$5 Acc and $e$6 Forgetting on GTSRB, substantially improving over SPU (Singh et al., 2024).

Mela instead treats test-time memory as a neural memory module whose weights are updated online, with explicit forgetting and momentum: $e$7 where $e$8 is the forgetting factor and $e$9 is a learnable decay factor for past surprise (Chen, 11 May 2026). Its Hierarchical Memory Module separates a fast low-level episodic trace from a slower high-level semantic trace, motivated by the transformation hypothesis. The empirical signature is a strong reduction in long-context degradation beyond the 4K training length: for 400M models, Transformer++ goes from $\mathcal M=\emptyset$00 perplexity at 4K to $\mathcal M=\emptyset$01 at 32K, whereas Mela goes from $\mathcal M=\emptyset$02 to $\mathcal M=\emptyset$03; for 1.2B, Transformer++ goes from $\mathcal M=\emptyset$04 to $\mathcal M=\emptyset$05, whereas Mela goes from $\mathcal M=\emptyset$06 to $\mathcal M=\emptyset$07 (Chen, 11 May 2026). This suggests an architectural route to slowing test-time memory decay by transforming transient detail into more stable semantic structure.

SR-TTT provides a compressed-memory perspective. In vanilla TTT, fast weights $\mathcal M=\emptyset$08 offer $\mathcal M=\emptyset$09 memory but overwrite surprising tokens under continued self-supervised updates, causing exact-recall failure. SR-TTT routes incompressible tokens to a residual exact-attention cache using the per-token reconstruction loss

$\mathcal M=\emptyset$10

with routing threshold

$\mathcal M=\emptyset$11

The final output is

$\mathcal M=\emptyset$12

On Needle-in-a-Haystack at sequence length 2048, exact match improves from $\mathcal M=\emptyset$13 to $\mathcal M=\emptyset$14 at depth $\mathcal M=\emptyset$15 and from $\mathcal M=\emptyset$16 to $\mathcal M=\emptyset$17 at depth $\mathcal M=\emptyset$18 (P, 26 Feb 2026). Here, test-time memory decay is overwrite of compressed state, and the mitigation is to preserve exactly what cannot be compressed.

6. Evidence, trade-offs, and unresolved questions

Across the literature, empirical evidence for test-time memory decay is mostly indirect but consistent. In TTA, degradation appears as error accumulation under long online adaptation, instability under non-i.i.d. streams, and collapse under poorly curated memory. STAMP’s CIFAR100-C with SVHN-C ablation shows a clear cumulative pattern: entropy minimization baseline has ACC $\mathcal M=\emptyset$19, AUC $\mathcal M=\emptyset$20, H-score $\mathcal M=\emptyset$21; adding self-weighting gives ACC $\mathcal M=\emptyset$22, AUC $\mathcal M=\emptyset$23; adding reliable class-balanced memory gives ACC $\mathcal M=\emptyset$24, AUC $\mathcal M=\emptyset$25; full STAMP reaches ACC $\mathcal M=\emptyset$26, AUC $\mathcal M=\emptyset$27, H-score $\mathcal M=\emptyset$28 (Yu et al., 2024). GoTTA shows that diversity-aware memories matter most under constrained budgets and non-i.i.d. streams, while ResiTTA shows that removing uncertainty-aware retention sharply worsens average error (Alhuwaider et al., 19 May 2026, Zhou et al., 2024).

In agent memory systems, the evidence increasingly emphasizes that naive accumulation is actively harmful. Evo-Memory reports that “naive memory accumulation introduces noise and disrupts subsequent retrieval,” and shows that baseline methods drop clearly when exposed to unfiltered failures, whereas ReMem remains robust because it can prune and refine memory during inference (Wei et al., 25 Nov 2025). MemoPilot likewise shows that Full History is harmful relative to structured, rewritten memory: with Qwen2.5-14B-Instruct as player, RPS@5 is $\mathcal M=\emptyset$29 for No Memory, $\mathcal M=\emptyset$30 for Full History, and $\mathcal M=\emptyset$31 for Memory w/ MemoPilot; LHE@5 is $\mathcal M=\emptyset$32, $\mathcal M=\emptyset$33, and $\mathcal M=\emptyset$34, respectively (Cai et al., 7 Jun 2026). This suggests that test-time memory decay in agents is often a failure of memory updating policy rather than of raw storage.

The same theme appears in self-evolving memory and trustworthiness. TAME defines Agent Memory Misevolution as the regime where

$\mathcal M=\emptyset$35

so capability improves while trustworthiness declines (Cheng et al., 3 Feb 2026). That is not decay of recall, but it is deployment-time memory degradation in a broader sense: what accumulates in memory becomes increasingly utility-centric and normatively harmful. In Science and Tool-use, the benchmark reports pronounced trustworthiness degradation for several standard methods, while TAME maintains much higher aggregated trustworthiness through dual-memory filtering and refinement (Cheng et al., 3 Feb 2026).

Two unresolved issues recur. First, most papers do not provide a literal time-series diagnosis of memory decay, such as age distributions, retention curves, or stale-sample attribution. STAMP explicitly notes that it provides no plots of memory contents or stale-sample analysis, and MemCoT does not quantify retention as a function of reasoning depth (Yu et al., 2024, Lei et al., 9 Apr 2026). Second, many methods rely on heuristic thresholds, retrieval policies, or fixed-capacity designs, while theoretical treatments remain mostly separate from applied systems. Mathematical papers on nonlocal memory and decay laws do supply explicit kernels—such as the scalar relaxation

$\mathcal M=\emptyset$36

for entropy decay under memory kernels, or the Ebbinghaus-style

$\mathcal M=\emptyset$37

for temporal memory dynamics—but these formalisms are not yet the dominant design language in practical test-time learning systems (Agresti et al., 2019, D'Agostino, 9 Feb 2025).

The most stable cross-paper conclusion is therefore narrow but robust: test-time memory decay is usually not a single event of forgetting, but a cumulative degradation process produced by poor retention, noisy updates, stale accessibility, redundancy, overwrite, or unsafe evolution. Methods that work best do not merely add memory; they control admission, retention, retrieval, consolidation, or reinforcement so that memory remains representative, relevant, and usable as inference unfolds (Yu et al., 2024, Alhuwaider et al., 19 May 2026, Rana et al., 31 Mar 2026, Chen, 11 May 2026).