STALE: State Tracking and Latent Evaluation

• STALE is a benchmark that defines latent-state tracking by evaluating LLMs' ability to detect when stored memories become invalid.
• It distinguishes between Type I co-referential conflicts and Type II propagated conflicts, highlighting the need for both explicit and propagated state updates.
• The framework employs 400 expert-validated scenarios with extensive dialogue contexts to rigorously test memory retrieval, state adjudication, and policy adaptation.

STALE most specifically denotes State Tracking And Latent Evaluation, a benchmark for testing whether LLM agents can determine when a previously stored memory is no longer valid (Chao et al., 7 May 2026). In a broader technical sense, the term also belongs to a wider research vocabulary of stale information: outdated memories, delayed model updates, saturated embeddings, obsolete profiles, superseded facts, and temporally misaligned sensor inputs that remain semantically relevant but no longer reflect the current state of the world (Askin et al., 1 Nov 2025, Maboud et al., 2024, Ayupov et al., 2024, Soleymani et al., 2018, Yadav, 25 Jun 2026). Within agent memory research, STALE formalizes the claim that long-term memory should be evaluated not as static fact retrieval, but as latent user-state tracking under evolving evidence (Chao et al., 7 May 2026).

1. Latent-state tracking and implicit conflict

The benchmark models an interaction history as a temporally ordered sequence of user and assistant messages,

$\mathcal{H} = \{(m_t, r_t)\},$

over user attributes

$\mathcal{A} = \{a_1, a_2, \dots, a_k\}.$

The user’s latent state at time $t$ is

$S_t = \{v_t(a) \mid a \in \mathcal{A}\},$

where $v_t(a)$ is the current value of attribute $a$, such as location, health, commute method, schedule, or preferences (Chao et al., 7 May 2026). In this formulation, memory is not a bag of recoverable spans; it is a partially observed evolving state.

The central failure mode is Implicit Conflict. An earlier message $m_o$ supports a belief $v_o(a)$, while a later message $m_n$ implies that the old belief is no longer valid,

$m_n \models_{\mathcal{K}} \neg v_o(a),$

but no later utterance explicitly states that the prior belief is false or outdated: $\mathcal{A} = \{a_1, a_2, \dots, a_k\}.$0 The later observation therefore invalidates the earlier one only through contextual inference and commonsense reasoning (Chao et al., 7 May 2026).

STALE distinguishes two mutually exclusive conflict types. In Type I: Co-referential conflict, both observations concern the same underlying attribute $\mathcal{A} = \{a_1, a_2, \dots, a_k\}.$1, and the later one implies an incompatible value,

$\mathcal{A} = \{a_1, a_2, \dots, a_k\}.$2

In Type II: Propagated conflict, the later observation updates a different attribute $\mathcal{A} = \{a_1, a_2, \dots, a_k\}.$3, and through a dependency

$\mathcal{A} = \{a_1, a_2, \dots, a_k\}.$4

that change invalidates an earlier belief about attribute $\mathcal{A} = \{a_1, a_2, \dots, a_k\}.$5, where $\mathcal{A} = \{a_1, a_2, \dots, a_k\}.$6 (Chao et al., 7 May 2026). The benchmark repeatedly emphasizes that Type II is harder because the agent must propagate a state change across related memories rather than merely replace one value with another.

2. Benchmark structure and probing dimensions

STALE contains 400 expert-validated conflict scenarios, 1,200 evaluation queries, 3 probing dimensions, over 100 everyday topics, and contexts up to 150K tokens (Chao et al., 7 May 2026). Appendix details specify 10 high-level ontology categories, 104 fine-grained seed attributes in the generation ontology, a final benchmark covering all 10 categories and 103 fine-grained attribute types, exactly 50 sessions per instance, around 593 dialogue turns per instance on average, and average formatted context length 151,784 tokens with median 151,829 and maximum 164,919 (Chao et al., 7 May 2026).

The ontology spans Spatiotemporal Context, Role and Identity, Social Network, Capability and Resource, Routine, Belongings and Possessions, Preference and Value, Physical and Mental Health, Current Focus, and Digital Footprint (Chao et al., 7 May 2026). The benchmark examples include moving cities, injuries that invalidate commute assumptions, schedule changes that alter social availability, and environmental shifts that implicitly contradict earlier place-based beliefs.

The three-dimensional probing framework separates explicit recognition from downstream behavioral use.

Probe	Question tested	Successful behavior
State Resolution (SR)	Is the earlier belief outdated?	Explicitly identifies the stale state
Premise Resistance (PR)	Can the model reject a stale presupposition?	Resists or corrects the false premise
Implicit Policy Adaptation (IPA)	Does updated state govern downstream behavior?	Acts consistently with the new state

State Resolution uses direct evaluator-style questions such as whether the user still commutes by cycling. Premise Resistance uses adversarial third-person requests built only from the old state, with zero leakage from the new state. Implicit Policy Adaptation uses natural first-person requests that mention neither the old nor the new evidence, requiring the model to retrieve and apply the updated latent state proactively (Chao et al., 7 May 2026). This separation is one of the benchmark’s central methodological contributions: retrieving updated evidence, rejecting stale presuppositions, and acting on revised state are treated as distinct capabilities.

3. Construction pipeline and evaluation protocol

The construction pipeline begins with base state formulation. A seed attribute $\mathcal{A} = \{a_1, a_2, \dots, a_k\}.$7 is sampled from the ontology, and an LLM generates a hypothetical persona, a realistic scenario, and an old observation $\mathcal{A} = \{a_1, a_2, \dots, a_k\}.$8 supporting some value $\mathcal{A} = \{a_1, a_2, \dots, a_k\}.$9 (Chao et al., 7 May 2026). An adversarial Logic Attacker then generates $t$0: for Type I, it implies an incompatible new value of the same attribute without naming the attribute directly; for Type II, it updates an upstream attribute while never directly mentioning the target belief or the dependency chain (Chao et al., 7 May 2026).

Quality control uses strict type-specific criteria: independent plausibility, true state-level conflict, and implicitness. Failed instances are regenerated. Human experts manually review candidates and label them Accept, Weak Reject, Wrong Type, or Reject, checking whether $t$1 supports a stable belief, whether $t$2 is genuinely the source of invalidation, whether the conflict type is correct, whether the three probes match their intended purpose, and whether there is no leakage from $t$3 into PR or IPA (Chao et al., 7 May 2026).

The evidence is then embedded into realistic long-horizon dialogue. The old and new evidence are wrapped into dialogue sessions $t$4 and $t$5, generated via role-playing, and inserted into a chronological haystack with distractor sessions sampled from LongMemEval. The filtering rule is that distractors must not contradict the target state, elaborate on it, or accidentally introduce another update to the same attribute. Timestamping enforces

$t$6

so the updated state should still plausibly govern the query (Chao et al., 7 May 2026).

Evaluation covers both plain LLMs and memory frameworks. The paper evaluates GPT-4o-mini, GPT-5.4-nano, GPT-5.4, Gemini-3.1-flash-lite, Gemini-3.1-pro, Llama-3.3-70B-Instruct, Qwen3.5-9B, Qwen3.5-27B, and MiniMax-M2.5, alongside LightMem, Zep, LiCoMemory, A-mem, and mem-0, all memory frameworks using GPT-4o-mini as backbone (Chao et al., 7 May 2026). If a model cannot fit the full context, the protocol uses evidence-preserving truncation, keeping $t$7 and $t$8 while partially removing distractors (Chao et al., 7 May 2026).

Responses are scored by an LLM judge, Gemini-3.1-flash-lite, against the underlying state logic rather than by string matching. The reported agreement with humans is 95.8% overall, with Cohen’s

$t$9

(Chao et al., 7 May 2026). The judge is described as slightly conservative, with very low false positive rate.

4. Empirical findings and the CUPMem prototype

The main empirical result is that current systems perform poorly on stale-memory reasoning. The best evaluated plain model, Gemini-3.1-pro, achieves 55.2% overall. Other plain models are much lower: Qwen3.5-27B reaches 31.3%, Gemini-3.1-flash-lite 22.4%, and GPT-5.4 15.7%. Memory frameworks mostly do not solve the problem: LightMem reaches 17.8%, Zep 6.0%, LiCoMemory 7.6%, A-mem 5.1%, and mem-0 8.3% (Chao et al., 7 May 2026).

The paper emphasizes three diagnostic findings. First, recognition does not imply application. For Qwen3.5-27B, Type I State Resolution is 76.0%, but Type I Implicit Policy Adaptation is only 39.0%; on Type II, 42.0% SR becomes 23.0% IPA (Chao et al., 7 May 2026). Second, Premise Resistance is the weakest dimension by far. For Gemini-3.1-pro, Type I SR is 92.0% while Type I PR is 30.0%; for Qwen3.5-27B, Type I SR is 76.0% while Type I PR is 4.0% (Chao et al., 7 May 2026). Third, Type II propagated conflicts are harder across nearly all systems. For Gemini-3.1-pro, SR drops from 92.0% to 69.0%, PR from 30.0% to 14.0%, and IPA from 71.0% to 55.0% when moving from Type I to Type II (Chao et al., 7 May 2026).

The paper’s analysis of LightMem is especially revealing. Updated evidence appears in top-20 retrieval for 77.5% of SR/PR cases and 67.8% of IPA cases, yet failures remain common: 56.1% SR failures despite new evidence, 99.0% PR failures despite new evidence, and 78.6% IPA failures despite new evidence (Chao et al., 7 May 2026). When new evidence arrives, the old evidence appears among top-3 recalled entries in 60.5% of cases, but only 3.3% of those old entries are judged as requiring an update (Chao et al., 7 May 2026). The benchmark therefore isolates a current-state adjudication gap, not merely a retrieval failure.

To establish a stronger baseline, the paper introduces CUPMem (Current-state Updating and Propagation-aware Memory). Its write-time adjudicator decides

$S_t = \{v_t(a) \mid a \in \mathcal{A}\},$0

Memory is represented with a typed two-level schema

$S_t = \{v_t(a) \mid a \in \mathcal{A}\},$1

and items

$S_t = \{v_t(a) \mid a \in \mathcal{A}\},$2

where $S_t = \{v_t(a) \mid a \in \mathcal{A}\},$3 and an UNKNOWN_CURRENT marker is used when the system knows that an old default is no longer safe but cannot infer a new replacement (Chao et al., 7 May 2026). To handle Type II, CUPMem expands stale-state search via

$S_t = \{v_t(a) \mid a \in \mathcal{A}\},$4

CUPMem, using a GPT-4o-mini backbone, achieves 68.0% overall. Its per-condition scores are 91.0% Type I SR, 78.0% Type I PR, 32.0% Type I IPA, 89.0% Type II SR, 75.0% Type II PR, and 43.0% Type II IPA (Chao et al., 7 May 2026). The largest gains are on Premise Resistance, which supports the paper’s claim that explicit write-time state adjudication is a promising direction, while Implicit Policy Adaptation remains difficult.

5. Stale information as a broader research motif

Outside agent memory, the same core difficulty recurs across machine learning and systems: information can remain semantically rich after it has become temporally misaligned. This suggests that STALE belongs to a larger research family in which the problem is not mere noise, but valid-once, invalid-now state.

Domain	Stale object	Representative result
Asynchronous federated learning	Client model updates	FedRevive revives stale updates through data-free knowledge distillation; up to 32.1% faster training and up to 21.5% higher final accuracy (Askin et al., 1 Nov 2025)
Dense retrieval	Cached target embeddings	Corrector networks match strong baselines with 4–80x reduction in re-embedding computational cost (Monath et al., 2024)
Post-link optimization	Binary profiles from older revisions	Stale profile matching in BOLT recovers up to 0.8 of the maximum BOLT benefit (Ayupov et al., 2024)
Retrieval memory over evolving knowledge	Superseded facts	MemStrata reaches 0.95–1.00 on evolving knowledge where RAG reaches 0.20–0.47, and drives stale-fact-error to ~0% (Yadav, 25 Jun 2026)
Control under outdated state	Age of information	Control cost is traded against information age through an AoI-aware dynamic program (Soleymani et al., 2018)

In asynchronous federated learning, stale client updates are defined by delay in server rounds, and directly applying them can destabilize optimization. FedRevive instead treats stale local models as teachers and transfers their predictive knowledge into the current global model through server-side data-free knowledge distillation, rather than trusting or discarding old parameter deltas outright (Askin et al., 1 Nov 2025). In dense retrieval, stale cached target embeddings degrade hard-negative mining and softmax approximation; the proposed remedy is a small corrector network that maps stale cached embeddings toward current ones during training (Monath et al., 2024). In compiler optimization, stale profiles arise because binary revisions drift between profiling and release, and matching plus flow-based inference can recover much of the optimization value that exact-match policies would discard (Ayupov et al., 2024). In retrieval memory over evolving knowledge, similarity alone cannot reliably distinguish contradictions from duplicates: on a calibrated dataset, cosine similarity yields AUROC 0.5926, and contradicted facts are on average more embedding-similar to the original than duplicates are; MemStrata addresses this with deterministic supersession and a bi-temporal ledger (Yadav, 25 Jun 2026). In stochastic control, stale information is formalized by Age of Information, with the controller acting on $S_t = \{v_t(a) \mid a \in \mathcal{A}\},$5 rather than $S_t = \{v_t(a) \mid a \in \mathcal{A}\},$6, and the optimal queuing policy explicitly balances information staleness against control performance (Soleymani et al., 2018).

These cases are technically heterogeneous, but they share a common structure: an outdated representation can still encode useful signal, yet naïve use of that signal can be harmful because validity has shifted.

6. Significance, misconceptions, and future directions

Several misconceptions are rejected explicitly by the STALE benchmark. It is not standard QA, because the task is not simply to answer from a retrieved passage. It is not standard contradiction detection, because the invalidating evidence usually does not explicitly negate the earlier one. It is not standard retrieval evaluation, because success requires more than retrieving updated evidence. It is not just fact replacement, because Type II requires propagation across related attributes. And it is not enough for a system merely to recognize outdated information under direct questioning, because downstream behavior may still follow the stale state (Chao et al., 7 May 2026).

A broader misconception across stale-information research is that stale signals are uniformly harmful and should simply be suppressed. The literature is less uniform. In some cases, stale updates or stale models remain informative but need to be revived, corrected, retired, or reinterpreted rather than directly applied (Askin et al., 1 Nov 2025). This suggests that the central design question is often not whether stale information should be used, but how its temporal validity should be represented and adjudicated.

The future directions named in the STALE benchmark are multi-step cascading updates, coupled attribute changes, and schema-free open-domain state tracking (Chao et al., 7 May 2026). A plausible implication is that memory systems for agents will need stronger write-time state consolidation, explicit stale-state adjudication, propagation-aware revision, and constrained readout grounded only in authorized current state. Within the wider stale-information literature, parallel developments such as deterministic supersession rules, validity intervals, server-side stale-update transformation, and temporal correction of cached representations suggest convergent pressure toward architectures that model currentness as a first-class property rather than leaving it to similarity or retrieval alone (Yadav, 25 Jun 2026).