---
title: 'MemTraceBench: Memory Evaluation Benchmark'
url: https://www.emergentmind.com/topics/memtracebench
type: topic
---

# MemTraceBench: Memory Evaluation Benchmark

Searching arXiv for the cited MemTrace/MemTraceBench papers and closely related memory-evaluation work.
MemTraceBench denotes a benchmark name used in two distinct but related 2026 lines of work on long-term memory in large language model systems. In one usage, introduced alongside "MemTrace: Probing What Final Accuracy Misses in Long-Term Memory," it is a benchmark for evaluating persistent user-fact memory at the level of the knowledge point rather than the question row, with controlled variation over memory age, question type, and evidence condition [2606.17328]. In another usage, introduced in "MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systems," it is a benchmark for error tracing and attribution in memory pipelines, built from executable memory evolution graphs and annotated failed cases across representative memory paradigms [2605.28732]. The shared name reflects a common concern: aggregate end-task accuracy is insufficient for characterizing how memory systems store, retrieve, propagate, or misuse information over time.

## 1. Dual usage and scope

The benchmark name appears in two technically different evaluation programs. The first centers on behavioral probing of long-term memory fidelity in conversational agents that retain user facts across sessions. Its basic claim is that row-wise QA accuracy cannot reveal how a single fact behaves under changing temporal and evidential conditions, because several questions may probe the same fact while being scored independently [2606.17328]. MemTrace therefore adopts the "knowledge point" as the atomic unit.

The second centers on causal diagnosis of failures inside memory-augmented pipelines. It starts from the observation that systems such as Long-Context, RAG, Mem0, and EverMemOS may succeed or fail on end-task accuracy, but flat logs make it difficult to determine which operation caused a failure. MemTraceBench in this sense instruments memory systems into executable graphs, collects failed runs, and labels faulty operations and error types to support attribution [2605.28732].

These two uses are complementary rather than redundant. A plausible implication is that one benchmark family measures *what* a memory system gets wrong at the behavioral level, while the other measures *where* in the internal pipeline the failure originates. Both reject final accuracy as a sufficient diagnostic statistic.

## 2. Knowledge-point benchmark design

In the long-term memory evaluation formulation, MemTrace defines the unit of measurement as a single typed fact about the user, termed a knowledge point [2606.17328]. Let

$$
K = \{\,k_1, k_2, \dots, k_{|K|}\}
$$

be the set of all knowledge points. Each $k \in K$ has a type $\tau(k)\in\{\text{static}, \text{dynamic}, \text{preference}\}$ and a canonical correct answer $a_k$.

Question rows are then generated by varying three controlled dimensions over each knowledge point. The space of question rows is defined as

$$
Q = \bigl\{(k, w, t, c)\bigr\}\subseteq K \times W\times T\times C,
$$

where $W$ is the set of memory windows, $T$ the set of question-type categories, and $C$ the set of evidence conditions [2606.17328]. This design shifts evaluation from isolated question-answer pairs to a structured probe over the same underlying fact.

The benchmark’s central distinction is therefore between knowledge-point evaluation and question-row evaluation. Traditional QA scoring treats each row independently. MemTrace instead groups multiple probes around one fact and examines performance as context changes. This suggests that it is intended to expose latent instability that pooled accuracy averages away.

## 3. Controlled dimensions and metrics

MemTrace varies three dimensions: memory age, question type, and evidence condition [2606.17328].

Memory age is defined by session indices $1,2,\dots,S$. If a knowledge point $k$ is first observed in session $s_{\mathrm{obs}(k)}$, and evaluation occurs at a checkpoint window $w$ ending at session $s_{\mathrm{eval}(w)}$, then memory age is

$$
a(k,w) \;=\; s_{\mathrm{eval}(w)\;-\;s_{\mathrm{obs}(k)\;\in\{0,1,2,\ldots\}.
$$

MemTrace chooses eight ordered windows $W = \{w_1,\dots,w_8\}$ from early to late, with $w_1,w_2$ designated "fresh" and $w_7,w_8$ designated "saturated" [2606.17328].

Question type is divided into current state, historical state, and trajectory of change. These correspond to asking for the current value of a fact ($t=\text{Curr}$), the earlier value as of first mention ($t=\text{Hist}$), or how the fact changed over time ($t=\text{Traj}$) [2606.17328]. The inclusion of trajectory explicitly tests whether systems can compose temporal evolution rather than merely recover one state.

Evidence condition is divided into Present, Missing, and Conflict. Present means the history through the evaluation window contains one or more explicit mentions of the knowledge point. Missing is a boundary probe in which the fact was never mentioned in any session up to evaluation, so the correct behavior is refusal or "I don’t know." Conflict is a false-premise probe in which the question begins with a false premise that contradicts the known state, and the system must detect and correct that premise [2606.17328].

Each response is scored by a triple

$$
(g,\,v,\,r)
\quad\text{where}\quad
g\in\{0,1\},
\;v\in[0,1],
\;r\in\{\mathrm{Answer},\mathrm{Abstain},\mathrm{Hallucinate}\}.
$$

Here $g$ denotes Gist, or semantic correctness; $v$ denotes Verbatim completeness; and $r$ denotes response type [2606.17328].

Aggregation is then defined over subsets of question rows:

$$
\mathrm{Acc}(\mathcal{Q}') \;=\;\frac{1}{|\mathcal{Q}'|}\sum_{(k,w,t,c)\in\mathcal{Q}'}
\mathbf{1}\bigl[g(k,w,t,c)=1\bigr].
$$

For Present-evidence questions, MemTrace defines fresh and saturated summaries and their gap $\Delta_t$:

$$
\text{Fresh}_t
= \frac{1}{2}\Bigl(\mathrm{Acc}(\{w_1,w_2\}\times\{t\}\times\{\mathrm{Present}\})\Bigr),\quad
\text{Sat}_t
= \frac{1}{2}\Bigl(\mathrm{Acc}(\{w_7,w_8\}\times\{t\}\times\{\mathrm{Present}\})\Bigr),
$$

$$
\Delta_t \;=\; \text{Fresh}_t \;-\;\text{Sat}_t.
$$

These definitions formalize temporal degradation at the level of fact usage rather than raw answer frequency [2606.17328].

## 4. Evaluated memory paradigms and system configurations

The knowledge-point MemTrace study evaluates 13 complete system configurations across four paradigms, with the final generator held constant as gpt-4o-mini except where otherwise noted [2606.17328].

| Paradigm | Configurations |
|---|---|
| Long-context models | Qwen3.5-35B; Gemini-3-Flash; GPT-5-nano |
| Retrieval-augmented generation | BM25 + generator; Text-emb-3-small + generator; Qwen3-Emb + generator; HippoRAG-v2 |
| External-memory stores | Mem0; SimpleMem; REMem; AMem |
| Agentic-memory architectures | MIRIX; Mem-T |

For each paradigm, the study varies what gets written to storage, how retrieval or context construction is done, and how the retrieved context is integrated into the final answer [2606.17328]. This decomposition matters because the benchmark is explicitly concerned not only with whether evidence exists, but with how memory-system design mediates its eventual use.

The error-attribution MemTraceBench is organized differently. It instruments four representative memory systems: Long-Context, RAG, Mem0, and EverMemOS [2605.28732]. Long-Context appends each user-assistant turn to a rolling context window. RAG stores past messages in a vector database and retrieves top-$K$ at query time. Mem0 extends RAG with explicit add/update/delete operations on memory units. EverMemOS adds an LLM-based sufficiency-checker and query-refinement step atop RAG [2605.28732]. These are replayed on sample trajectories from LoCoMo, LongMemEval, and RealMem, with execution tracing performed by smartcomment [2605.28732].

Taken together, the two benchmark instantiations cover both cross-paradigm behavioral evaluation and cross-paradigm operational diagnosis.

## 5. Principal empirical findings

The central empirical finding of the knowledge-point benchmark is that evidence-use is the dominant bottleneck [2606.17328]. In "reach vs. use" replay, a simple retriever, Text-emb-3-small, is run over the same question rows, and the study computes $P(R=0)$, $P(R=1,U=0)$, and $P(R=1,U=1)$, where $R=1$ means the retriever reaches the gold evidence and $U=1$ means the original system answers correctly. Empirically, $P(R=0)\approx 7\%$ while $P(R=1,U=0)\approx 73\%$, indicating that failures are approximately 10 times more often due to reachable but unused evidence than to unreachable evidence [2606.17328].

An "oracle open-book" test reinforces that interpretation. On all-systems-fail subsets, supplying gold evidence directly to the generator raises Gist accuracy from as low as $0\%$-$33\%$ to over $80\%$ [2606.17328]. This is used to argue that many errors reflect failures to exploit evidence rather than failures to store or retrieve it.

A second finding is that pooled accuracy hides distinct failure modes. Trajectory questions are systematically harder than current-state and historical-state questions. For example, Qwen3.5-35B has Fresh Trajectory = $49.0\%$ and Saturated Trajectory = $6.7\%$, a 42.3 percentage point drop, while its Current state gap is 36.1 percentage points and Historical gap is 21.9 percentage points. HippoRAG-v2 leads RAG systems in saturated Current ($45.4\%$) and Historical ($50.9\%$) performance but achieves only $13.4\%$ on saturated Trajectory [2606.17328]. The benchmark therefore demonstrates that recovering single-state facts does not imply the ability to track change over time.

A third finding concerns safety behavior. Some external-memory systems, specifically Mem0, AMem, and REMem, show very high refusal rates on missing-evidence probes, with $>\!95\%$ boundary abstention, but rarely resolve false-premise probes [2606.17328]. The reported example is Mem0, with conflict Gist = $14.6\%$, conflict hallucination $\approx 2.7\%$, and conflict abstention $\approx 82.5\%$ [2606.17328]. The interpretation given is that these systems treat contradicted premises as absent information rather than using memory to refute them.

The error-attribution MemTraceBench yields a different set of results. It collects over 1,500 actual errors across LoCoMo, LongMemEval, and RealMem, and distills them into 160 fully annotated failure cases, 40 per system, after filtering out annotation and judge errors [2605.28732]. It defines two attribution metrics: Error-Type Accuracy (ETA) and Operation Identification Accuracy (OIA). With GPT-4.1 mini and GPT-5.4 as agent backbones, MemTrace exceeds MemTrace-OBS by +16.46 points and +8.63 points on ETA, and reaches best overall OIA of 46.25% with GPT-5.4 [2605.28732]. On Mem0, attribution-guided prompt optimization produces a 7.62% absolute boost in end-task accuracy [2605.28732].

## 6. Error tracing, attribution, and memory evolution graphs

The error-attribution line formalizes any non-parametric memory system $\mathcal{M}$ as a stateful machine that processes messages $\mathbf{m}_j$, updates a memory state $\mathcal{S}_j$, reads a context $\mathcal{C}$ at query time, and produces an answer $\bar{a}$:

$$
\mathcal{S}_{j} \;=\; \mathcal{U}_{\mathcal{M}}\bigl(\mathcal{S}_{j-1},\,m_{j}\bigr)\quad(1\le j\le n),
\qquad
\mathcal{C}\;=\;\mathcal{R}_{\mathcal{M}}(\mathcal{S}_{n},q),
\qquad
\bar{a}\;=\;\mathcal{Q}\bigl(q,\mathcal{C}\bigr).
$$

To capture fine-grained causality, each update, retrieval, tool call, and LLM inference is intercepted as an operation $o$, and each intermediate artifact as a variable $v$. The execution graph is

$$
\mathcal{G}=(\mathcal{V},\mathcal{O},\mathcal{E}),
$$

where $\mathcal{V}$ are variable nodes, $\mathcal{O}$ are operation nodes, and directed edges connect variables and operations [2605.28732]. Task outcome is given by a binary variable $Z(\mathcal{G})\in\{0,1\}$ indicating success or failure.

The benchmark annotates seven error types: Extraction Error, Update Error, Deletion Error, Retrieval Error, Response Error, Annotation Error, and LLM-Judge Error [2605.28732]. Each case includes the question, gold answer, full execution graph, faulty operation id, error type, and a human-written explanation.

Automatic attribution seeks a minimal decisive error set $\mathcal{O}^*$. A candidate $O_c\subseteq\mathcal{O}$ is valid if every operation in it is faulty, all strictly upstream ancestors are correct, and replacing its outputs with correct values rescues the run, that is, $Z\bigl(\mathcal{G}^{(O_c,*)}\bigr)=0$ [2605.28732]. In the sequential systems studied, the decisive error set is always a singleton.

MemTrace performs graph-guided search. It uses hybrid retrieval, dense plus BM25 fused by reciprocal-rank fusion, on the concatenation $[q;\,a]$ to identify likely evidence messages, initializes a priority queue of variables ordered by timestamp, then iteratively inspects incident operations and their local subgraphs with an LLM agent [2605.28732]. MemTrace-OBS is a lighter variant that flattens operation subgraphs into a searchable log and uses a regex-driven global search tool, reducing token usage and runtime by approximately 70% at some cost to diagnostic accuracy [2605.28732].

This framework turns failure analysis into an explicit causal-search problem over memory evolution rather than a post hoc reading of logs. A plausible implication is that it provides an operational counterpart to the behavioral probing of the knowledge-point benchmark.

## 7. Interpretation, limitations, and research directions

Across both meanings of MemTraceBench, the recurring thesis is that memory evaluation should move beyond aggregate answer correctness. In the knowledge-point benchmark, this takes the form of fact-centric probing across temporal and evidential dimensions. In the graph-based benchmark, it takes the form of operation-level attribution over stateful memory pipelines [2606.17328; 2605.28732].

Several misconceptions are directly addressed by the reported results. One is that improved retrieval alone solves long-term memory. The "reach vs. use" replay indicates that only about $7\%$ of failures arise from unreachable evidence, whereas approximately $73\%$ arise when evidence is retrievable but not used correctly [2606.17328]. Another is that safe abstention implies robust epistemic behavior. The false-premise probes show that boundary refusal can coexist with poor contradiction correction, as in Mem0’s low conflict Gist and high conflict abstention [2606.17328]. A third is that end-task failure can be localized by inspection of final prompts or outputs; the graph-based benchmark argues instead for operation-level tracing because failures stem from issues such as information loss, retrieval misalignment, corrupted updates, or response-stage misuse of correct evidence [2605.28732].

The future directions explicitly proposed in the knowledge-point study include architectures for temporal summarization or "diffing" of multi-session facts, prompt or model modules specifically trained to detect and correct false premises, retrieval interfaces that expose not only where a fact was mentioned but how it changed over time, and expansion to more domains, interactive settings, and larger user populations [2606.17328]. The graph-based study recommends semantically meaningful instrumentation granularity, semantic metadata on operations and variables, versioned identity strategies for in-place updates, hybrid retrieval seeds for graph exploration, and extensions from singleton decisive-error assumptions to minimal multi-operation cut-sets in parallel or aggregated systems [2605.28732].

In aggregate, MemTraceBench names a broader methodological shift in long-term memory research for LLMs: evaluation and diagnosis should be structured around temporally evolving evidence, explicit state transitions, and identifiable causal failure points, rather than around undifferentiated final accuracy alone [2606.17328].

Source: https://www.emergentmind.com/topics/memtracebench