StructMemEval: Organized Memory Benchmark
- The paper introduces StructMemEval, a benchmark focusing on organizing conversational evidence into task-appropriate memory structures rather than mere factual recall.
- It evaluates performance across three task families—tree-based, state-tracking, and count-based—using structured representations like graphs, ledgers, and state trackers.
- Empirical findings reveal that retrieval-augmented LLMs struggle with latent structure maintenance while memory agents improve accuracy significantly with the aid of hints.
StructMemEval is a benchmark for long-term memory in LLM agents that targets organization rather than mere factual recall. The benchmark is motivated by the claim that many established long-term memory evaluations stress recall, multi-hop lookup, and temporal updates, yet often do not isolate whether a system can maintain a task-appropriate memory structure. StructMemEval therefore assembles tasks that humans typically solve by explicitly organizing notes into structures such as transaction ledgers, state trackers, and trees, while remaining architecture-agnostic and judging only final task performance (Shutova et al., 11 Feb 2026).
1. Motivation and benchmark objective
The central argument behind StructMemEval is that prior long-term memory benchmarks, including LOCOMO and LongMemEval, primarily test whether an agent can recover facts from long histories, not whether it can represent and maintain the right structure for the task. The paper further argues that these abilities can often be achieved with simple retrieval-augmented baselines, citing recent work such as EMem / EMem-G as evidence that complex memory systems do not necessarily gain a decisive advantage on recall-centric benchmarks (Shutova et al., 11 Feb 2026).
StructMemEval is designed to expose a different capability boundary. Its target is the ability to turn conversational evidence into an organized latent memory representation that supports operations such as shortest-path reasoning in a relation graph, current-state maintenance under repeated updates, and netted accounting over many transactions. The benchmark deliberately avoids requiring a canonical internal schema, because memory frameworks vary from markdown note stores to vector indices, graphs, temporal databases, and hybrid systems. Instead, it relies on tasks that are straightforward once the memory has been structured correctly and difficult when the agent relies only on local snippet retrieval (Shutova et al., 11 Feb 2026).
This design gives StructMemEval a functional interpretation of long-term memory quality. Success is treated as evidence that the system organized information in a usable way, even though the benchmark does not inspect the internal memory state directly. A plausible implication is that StructMemEval is less a probe of static retention than a probe of schema induction and maintenance policy under long-horizon interaction.
2. Operational notion of memory structure
The paper does not define memory structure as a fixed data model. Rather, it treats memory structure operationally as the organization of stored long-term knowledge into a form that facilitates solving a target task (Shutova et al., 11 Feb 2026). This choice is deliberate: a benchmark intended to compare heterogeneous frameworks cannot require one specific representation.
The paper’s examples make the intended concept concrete. A family-relationship conversation should induce a graph- or tree-like representation; a stream of expenses should induce a ledger with netting logic; a sequence of relocations and interactions should induce an explicit current-state tracker rather than a bag of semantically similar utterances. In this sense, raw conversational messages are not sufficient memory objects. The benchmark is designed so that the agent must transform those messages into a structure aligned with the downstream inference.
This notion is reflected in the benchmark’s diagnostic use of optional memory organization hints. These hints tell the model how a human would organize the material, for example by keeping a ledger or maintaining a tree. If performance improves substantially under the hint condition, the failure is interpreted as a failure of structure recognition rather than of low-level storage or retrieval mechanics. The paper notes that the main benchmark condition is without hints, and uses the hint condition primarily to disambiguate whether the model chose the wrong structure or failed to execute a known structural strategy (Shutova et al., 11 Feb 2026).
3. Dataset construction and task families
StructMemEval contains 73 unique evaluation scenarios (conversations) and 544 evaluation questions. The data are synthetic: initial scenarios were created manually, LLM augmentation was used to produce variants and additional questions, and humans verified and corrected hallucinations and incorrect references. The paper also states that 3 different LLMs were used for augmentation to reduce style contamination that might otherwise advantage a particular evaluated model (Shutova et al., 11 Feb 2026).
The current release evaluates three implemented subsets and explicitly distinguishes them from future extensions.
| Task family | Implemented content | Structural capability stressed |
|---|---|---|
| Tree-based | family trees; corporate hierarchies | graph/tree construction and traversal |
| State-tracking | static to 5 relevant transitions | current-state maintenance under updates |
| Count-based | accounting/settlement; recommendation/statistics | ledgers, netting, and running aggregates |
The paper also lists future extensions to ordered lists for to-do queues and top- selection, assignment maps, indexes, DAGs, and tasks involving multiple structures simultaneously. It is explicit that to-do lists are discussed as future work rather than part of the current main experiments (Shutova et al., 11 Feb 2026).
Tree-based problems
Tree-based problems are motivated by tasks requiring hierarchical or relational organization, including family trees and corporate hierarchies. The construction pipeline generates a reference graph over people, ensures all names are unique, selects node pairs whose unique shortest path has 10 hops with 11 nodes visited, and then creates graph subsets of 10, 20, 30, 40, 50, 100, 150, and 200, 250 links encoded as messages. Each subset is required to contain the target shortest path, and the agent is asked to recover that path after processing the messages (Shutova et al., 11 Feb 2026).
These problems are structurally difficult because relevant relations may be indirect and because some relations are implied rather than stated directly. The paper gives the example that if “C is B’s wife” and “A is B’s stepdaughter,” then “C is A’s parent” is implied. This pushes the agent toward graph construction and relation composition rather than top- semantic retrieval (Shutova et al., 11 Feb 2026).
State-tracking problems
State-tracking problems require the model to maintain the current state of entities under repeated updates. The paper gives examples involving changing city of residence, changing neighbors, KANBAN-like task movement, and code refactoring. Difficulty levels range from 0 transitions (“static”) to 5 relevant state changes (Shutova et al., 11 Feb 2026).
The appendix reports the following dataset statistics for the state-tracking subset:
| Difficulty | Files | Sessions | Total Msgs |
|---|---|---|---|
| static | 7 | 15 | 150 |
| 1tr | 7 | 35 | 219 |
| 2tr | 7 | 36 | 188 |
| 3tr | 7 | 49 | 238 |
| 4tr | 7 | 59 | 296 |
| 5tr | 7 | 73 | 362 |
The paper highlights that 2-transition conversations are shorter on average than 1-transition ones, yet models still find them harder, which is presented as evidence that difficulty is structural rather than merely length-based (Shutova et al., 11 Feb 2026).
Count-based problems
The count-based subset groups tasks that require cumulative accounting or aggregate statistics. One subtype is accounting / settlement, where messages describe transactions and the model must compute the final settlement after netting, including cancellation of circular debts. The paper states that unrelated messages are injected at approximately 10%, 30%, and 50% noise, with 5 unique scenarios each, for a total of 15 accounting scenarios (Shutova et al., 11 Feb 2026).
A second subtype covers recommendation/statistics tasks in which a user accumulates experiences such as movies watched, books read, or activities attended, and the assistant must determine which activities were tried more often or liked more on average, with exact numeric answers expected. The benchmark’s interpretation is that a retrieval pipeline over a limited cannot reliably recover such statistics once the relevant sample exceeds the retrieval window; the agent must instead maintain running aggregates (Shutova et al., 11 Feb 2026).
4. Evaluation protocol, systems, and reproducibility
StructMemEval evaluates three families of memory systems. The first is a retrieval-augmented LLM baseline using OpenAI embeddings, specifically text-embedding-3-large, together with retrieval baseline code from Mem0. The main setup uses top-15 retrieval for tree-based tasks, top-10 for count-based tasks, and top-5 for state tracking (Shutova et al., 11 Feb 2026).
The second is mem-agent, described as a markdown-file-based memory framework that can create, modify, and retrieve arbitrary markdown files. The evaluation feeds conversation shards through mem_agent.chat(input_shard) and then clears recent history with:
1 |
mem_agent.messages = mem_agent.messages[:1] |
This is intended to prevent the model from relying on immediate conversational context and to simulate messages arriving across long time spans and different sessions (Shutova et al., 11 Feb 2026).
The third is Mem0 in an agentic configuration. The paper emphasizes that this setup uses the same search module as the retrieval baseline, but now the LLM can decide what memories to store, how to formulate retrieval queries, and how to manage memory over time. The benchmark therefore compares passive retrieval against agentic memory management rather than against a wholly different serving substrate (Shutova et al., 11 Feb 2026).
Main experiments use gemini-2.5-pro and gemini-3-pro, with gemini-3-pro used for count-based tasks because gemini-2.5-pro hallucinated too much there. Additional appendix results include gemini-3-flash and gpt-4o-mini (Shutova et al., 11 Feb 2026).
Evaluation uses exact match where possible, including yes/no and numeric questions, and LLM-as-a-judge for questions without a clear answer format. The paper explicitly does not present formal mathematical formulas for the benchmark metrics; reported results are primarily accuracy (Shutova et al., 11 Feb 2026).
For reproducibility, the appendix provides a concrete Mem0 configuration with Qdrant as the vector store, 3072 embedding dimensions, a fixed user_id, and infer=False during ingestion. The benchmark is released at https://github.com/yandex-research/StructMemEval (Shutova et al., 11 Feb 2026).
5. Empirical findings
The paper’s main empirical conclusion is that retrieval-augmented LLMs degrade sharply when the task requires maintaining a latent structure rather than recovering a few locally relevant snippets. The failure modes differ by task family, but the overall pattern is consistent: retrieval helps on small instances, larger helps somewhat, and then performance collapses once the relevant state exceeds what top- retrieval can surface (Shutova et al., 11 Feb 2026).
State tracking
The appendix gives detailed state-tracking results for both gpt-4o-mini and gemini-2.5-pro. The most striking pattern is the large gap between retrieval and agentic memory, particularly under the hint condition.
| Backbone | System | Average accuracy |
|---|---|---|
| gpt-4o-mini | retrieval (top-5) | 24% |
| gpt-4o-mini | retrieval (top-20) | 40% |
| gpt-4o-mini | mem-agent (with hint) | 74% |
| gpt-4o-mini | Mem0 agent (with hint) | 60% |
| gemini-2.5-pro | retrieval (top-5) | 21% |
| gemini-2.5-pro | retrieval (top-20) | 26% |
| gemini-2.5-pro | mem-agent (with hint) | 79% |
| gemini-2.5-pro | Mem0 agent (with hint) | 81% |
The paper also reports gemini-3-flash with Mem0 agent (hint) at 95% average accuracy on this subset (Shutova et al., 11 Feb 2026). These numbers support the paper’s claim that state tracking exposes a genuine memory-organization gap: retrieval often returns stale mentions, whereas structured memory agents can maintain the current latent state.
Count-based tasks
For accounting/counting, the appendix shows that gemini-2.5-pro performs poorly because of hallucinations, which motivated the switch to gemini-3-pro for the main count-based evaluation. Under gemini-3-pro, retrieval remains competitive only on the smallest transaction sets and then collapses at higher counts.
| Method | Avg @10 tx | Avg @20 tx | Avg @50 tx |
|---|---|---|---|
| Retrieval (top-5) | 0.2 | 0.0 | 0.0 |
| Retrieval (top-10) | 1.0 | 0.2 | 0.0 |
| Retrieval (top-20) | 1.0 | 0.73 | 0.0 |
| mem-agent (w/o hint) | 0.73 | 0.67 | 0.20 |
| mem-agent (with hint) | 0.67 | 0.47 | 0.27 |
The benchmark interprets this as a direct demonstration that larger retrieval budgets are not sufficient when the problem requires a maintained ledger or running aggregate. Even top-20 retrieval reaches 0.0 at 50 transactions, whereas mem-agent remains nonzero and improves slightly under hints at that scale (Shutova et al., 11 Feb 2026).
Tree-based tasks
For tree-based tasks, the provided text does not include exact appendix tables, but the qualitative result is consistent with the other subsets: retrieval can handle smaller instances, degrades as graph size and indirectness increase, and is outperformed by memory agents. The paper also states that performance improves substantially when a memory-organization hint is given, which reinforces the interpretation that part of the challenge is recognizing that the conversation should be turned into a graph-like representation (Shutova et al., 11 Feb 2026).
Interpretive conclusion
One of the paper’s most consequential findings is that the gap between with hint and without hint is often larger than the gap between Mem0 and mem-agent. The paper therefore cautions against reading these experiments as definitive rankings of memory frameworks, especially because it uses default hyperparameters and minimal tuning. The stronger claim is that modern LLMs often possess enough procedural capacity to use structured memory once prompted, but do not reliably infer the required structure on their own (Shutova et al., 11 Feb 2026).
6. Failure modes, limitations, and open extensions
The paper’s error analysis identifies three recurring accounting failures in agentic memory files: omission of transactions, duplication of transactions, and hallucination of transactions. It gives an example in which a nonexistent transaction labeled “sandwich” appears and a lunch transaction is duplicated. This is important because it shows that even when the agent adopts the right high-level structure, maintenance fidelity can still drift over long update sequences (Shutova et al., 11 Feb 2026).
The paper suggests several reasons why LLMs fail to infer the right structure. One is a training mismatch: LLMs may know abstract structures such as trees, heaps, and state machines in coding or textbook contexts, yet have not been trained to apply those abstractions to their own memory management. Another is that natural conversational data are semantically messy, with irrelevant chatter, interleaved updates, and partial implications, making latent structure induction nontrivial. A third is that long update chains induce drift, especially in ledger-like tasks (Shutova et al., 11 Feb 2026).
Several benchmark limitations are explicit. Current task coverage is restricted to tree-based, state-tracking, and count-based subsets. Future work is needed for to-do/ordered lists, DAGs, assignment maps, indexes, and multi-structure tasks. The benchmark is synthetic, which aids controllability and privacy but may not capture all real-user noise. Some evaluations rely on LLM-as-a-judge. Results are also sensitive to the backbone model, as shown by the need to replace gemini-2.5-pro with gemini-3-pro on counting tasks (Shutova et al., 11 Feb 2026).
A plausible implication is that future progress may require both better benchmark coverage and architectural support for typed or scaffolded memory primitives, so that the burden of discovering ledgers, trees, or state registers does not fall entirely on the backbone model.
7. Position within long-term memory evaluation research
StructMemEval occupies a distinct niche within the 2026 landscape of LLM memory evaluation. Its contribution is not lifecycle-aware serving analysis, interactive scenario design, or dependency-aware world-state maintenance per se, but an explicit benchmark for whether an agent can choose and maintain an appropriate memory structure (Shutova et al., 11 Feb 2026).
This focus differs from Neuromem, which studies external memory under a streaming, lifecycle-aware, interleaved insertion-and-retrieval protocol and decomposes the memory lifecycle into memory data structure, normalization, consolidation, query formulation, and context integration. Neuromem argues that static snapshot evaluation misses the main deployment difficulty because memory is an evolving service, and it reports that the memory data structure dominates the attainable quality frontier (Zhang et al., 15 Feb 2026). StructMemEval is narrower but more targeted: it does not benchmark the serving lifecycle, but instead probes whether structured organization itself is required for success.
It also differs from StructMem, which evaluates a structure-enriched hierarchical memory framework on LoCoMo QA rather than proposing a separate benchmark. StructMem operationalizes structured memory through event-level binding, temporal anchoring, and cross-event consolidation, with strong results on overall LoCoMo performance and temporal reasoning, but its evaluation remains tied to long-horizon dialogue QA (Xu et al., 23 Apr 2026). StructMemEval abstracts away from one particular memory architecture and asks whether task families such as ledgers, trees, and state trackers can be solved at all without explicit organization.
Relative to MEME, StructMemEval targets a different structural axis. MEME formalizes persistent memory along single vs. multi-entity and static vs. evolving dimensions, introduces tasks such as Cascade, Absence, and Deletion, and shows that current systems collapse on dependency-aware post-change reasoning (Jung et al., 12 May 2026). MEME therefore evaluates coherent evolving world state and dependency propagation, whereas StructMemEval evaluates whether the agent recognizes and maintains a task-specific organizational form.
Finally, StructMemEval shares with MemGround a dissatisfaction with static recall-style testing, but the two frameworks diverge in methodology. MemGround is natively grounded in interactive, gamified scenarios and evaluates Surface State Memory, Temporal Associative Memory, and Reasoning-Based Memory using metrics such as QA Overall, MFU, MFCO, and ETD (Ding et al., 23 Mar 2026). StructMemEval is less process-heavy and more minimalistic: it emphasizes architecture independence, final-answer evaluation, and task families that become simple once the correct memory schema is in place.
Taken together, these comparisons situate StructMemEval as a benchmark for a specific but consequential failure mode in agent memory systems: the inability to spontaneously recognize that long-term memory should sometimes be maintained as a ledger, state tracker, or tree, even when the model can use such structures effectively once they are suggested.