---
title: 'StructMemEval: Organized Memory Benchmark'
url: https://www.emergentmind.com/topics/structmemeval
type: topic
---

# StructMemEval: Organized Memory Benchmark

StructMemEval is a benchmark for long-term memory in LLM agents that targets **organization** rather than mere **factual recall**. The benchmark is motivated by the claim that many established long-term memory evaluations stress recall, multi-hop lookup, and temporal updates, yet often do not isolate whether a system can maintain a task-appropriate memory structure. StructMemEval therefore assembles tasks that humans typically solve by explicitly organizing notes into structures such as **transaction ledgers**, **state trackers**, and **trees**, while remaining architecture-agnostic and judging only final task performance [2602.11243].

## 1. Motivation and benchmark objective

The central argument behind StructMemEval is that prior long-term memory benchmarks, including **LOCOMO** and **LongMemEval**, primarily test whether an agent can recover facts from long histories, not whether it can **represent and maintain the right structure** for the task. The paper further argues that these abilities can often be achieved with simple retrieval-augmented baselines, citing recent work such as **EMem / EMem-G** as evidence that complex memory systems do not necessarily gain a decisive advantage on recall-centric benchmarks [2602.11243].

StructMemEval is designed to expose a different capability boundary. Its target is the ability to turn conversational evidence into an organized latent memory representation that supports operations such as shortest-path reasoning in a relation graph, current-state maintenance under repeated updates, and netted accounting over many transactions. The benchmark deliberately avoids requiring a canonical internal schema, because memory frameworks vary from markdown note stores to vector indices, graphs, temporal databases, and hybrid systems. Instead, it relies on tasks that are straightforward once the memory has been structured correctly and difficult when the agent relies only on local snippet retrieval [2602.11243].

This design gives StructMemEval a functional interpretation of long-term memory quality. Success is treated as evidence that the system organized information in a usable way, even though the benchmark does not inspect the internal memory state directly. A plausible implication is that StructMemEval is less a probe of static retention than a probe of **schema induction and maintenance policy** under long-horizon interaction.

## 2. Operational notion of memory structure

The paper does not define memory structure as a fixed data model. Rather, it treats memory structure operationally as **the organization of stored long-term knowledge into a form that facilitates solving a target task** [2602.11243]. This choice is deliberate: a benchmark intended to compare heterogeneous frameworks cannot require one specific representation.

The paper’s examples make the intended concept concrete. A family-relationship conversation should induce a **graph- or tree-like representation**; a stream of expenses should induce a **ledger with netting logic**; a sequence of relocations and interactions should induce an explicit **current-state tracker** rather than a bag of semantically similar utterances. In this sense, raw conversational messages are not sufficient memory objects. The benchmark is designed so that the agent must transform those messages into a structure aligned with the downstream inference.

This notion is reflected in the benchmark’s diagnostic use of optional **memory organization hints**. These hints tell the model how a human would organize the material, for example by keeping a ledger or maintaining a tree. If performance improves substantially under the hint condition, the failure is interpreted as a failure of **structure recognition** rather than of low-level storage or retrieval mechanics. The paper notes that the main benchmark condition is **without hints**, and uses the hint condition primarily to disambiguate whether the model chose the wrong structure or failed to execute a known structural strategy [2602.11243].

## 3. Dataset construction and task families

StructMemEval contains **73 unique evaluation scenarios (conversations)** and **544 evaluation questions**. The data are synthetic: initial scenarios were created manually, LLM augmentation was used to produce variants and additional questions, and humans verified and corrected hallucinations and incorrect references. The paper also states that **3 different LLMs** were used for augmentation to reduce style contamination that might otherwise advantage a particular evaluated model [2602.11243].

The current release evaluates **three implemented subsets** and explicitly distinguishes them from future extensions.

| Task family | Implemented content | Structural capability stressed |
|---|---|---|
| Tree-based | family trees; corporate hierarchies | graph/tree construction and traversal |
| State-tracking | static to 5 relevant transitions | current-state maintenance under updates |
| Count-based | accounting/settlement; recommendation/statistics | ledgers, netting, and running aggregates |

The paper also lists future extensions to **ordered lists** for to-do queues and top-\(K\) selection, **assignment maps**, **indexes**, **DAGs**, and tasks involving **multiple structures simultaneously**. It is explicit that to-do lists are discussed as future work rather than part of the current main experiments [2602.11243].

### Tree-based problems

Tree-based problems are motivated by tasks requiring hierarchical or relational organization, including **family trees** and **corporate hierarchies**. The construction pipeline generates a reference graph over people, ensures all names are unique, selects node pairs whose unique shortest path has **10 hops** with **11 nodes visited**, and then creates graph subsets of **10, 20, 30, 40, 50, 100, 150, and 200, 250 links** encoded as messages. Each subset is required to contain the target shortest path, and the agent is asked to recover that path after processing the messages [2602.11243].

These problems are structurally difficult because relevant relations may be indirect and because some relations are implied rather than stated directly. The paper gives the example that if “C is B’s wife” and “A is B’s stepdaughter,” then “C is A’s parent” is implied. This pushes the agent toward graph construction and relation composition rather than top-\(k\) semantic retrieval [2602.11243].

### State-tracking problems

State-tracking problems require the model to maintain the **current state** of entities under repeated updates. The paper gives examples involving changing city of residence, changing neighbors, KANBAN-like task movement, and code refactoring. Difficulty levels range from **0 transitions (“static”)** to **5 relevant state changes** [2602.11243].

The appendix reports the following dataset statistics for the state-tracking subset:

| Difficulty | Files | Sessions | Total Msgs |
|---|---:|---:|---:|
| static | 7 | 15 | 150 |
| 1tr | 7 | 35 | 219 |
| 2tr | 7 | 36 | 188 |
| 3tr | 7 | 49 | 238 |
| 4tr | 7 | 59 | 296 |
| 5tr | 7 | 73 | 362 |

The paper highlights that **2-transition conversations are shorter on average than 1-transition ones, yet models still find them harder**, which is presented as evidence that difficulty is structural rather than merely length-based [2602.11243].

### Count-based problems

The count-based subset groups tasks that require cumulative accounting or aggregate statistics. One subtype is **accounting / settlement**, where messages describe transactions and the model must compute the **final settlement after netting**, including cancellation of circular debts. The paper states that unrelated messages are injected at approximately **10%, 30%, and 50% noise**, with **5 unique scenarios each**, for a total of **15 accounting scenarios** [2602.11243].

A second subtype covers **recommendation/statistics** tasks in which a user accumulates experiences such as movies watched, books read, or activities attended, and the assistant must determine which activities were tried more often or liked more on average, with exact numeric answers expected. The benchmark’s interpretation is that a retrieval pipeline over a limited \(k\) cannot reliably recover such statistics once the relevant sample exceeds the retrieval window; the agent must instead maintain running aggregates [2602.11243].

## 4. Evaluation protocol, systems, and reproducibility

StructMemEval evaluates three families of memory systems. The first is a **retrieval-augmented LLM baseline** using **OpenAI embeddings**, specifically **text-embedding-3-large**, together with retrieval baseline code from **Mem0**. The main setup uses **top-15** retrieval for tree-based tasks, **top-10** for count-based tasks, and **top-5** for state tracking [2602.11243].

The second is **mem-agent**, described as a markdown-file-based memory framework that can create, modify, and retrieve arbitrary markdown files. The evaluation feeds conversation shards through `mem_agent.chat(input_shard)` and then clears recent history with:

```python
mem_agent.messages = mem_agent.messages[:1]
```

This is intended to prevent the model from relying on immediate conversational context and to simulate messages arriving across long time spans and different sessions [2602.11243].

The third is **Mem0** in an agentic configuration. The paper emphasizes that this setup uses the same search module as the retrieval baseline, but now the LLM can decide **what memories to store**, **how to formulate retrieval queries**, and **how to manage memory over time**. The benchmark therefore compares passive retrieval against agentic memory management rather than against a wholly different serving substrate [2602.11243].

Main experiments use **gemini-2.5-pro** and **gemini-3-pro**, with **gemini-3-pro** used for count-based tasks because **gemini-2.5-pro hallucinated too much** there. Additional appendix results include **gemini-3-flash** and **gpt-4o-mini** [2602.11243].

Evaluation uses **exact match** where possible, including yes/no and numeric questions, and **LLM-as-a-judge** for questions without a clear answer format. The paper explicitly does **not** present formal mathematical formulas for the benchmark metrics; reported results are primarily **accuracy** [2602.11243].

For reproducibility, the appendix provides a concrete Mem0 configuration with **Qdrant** as the vector store, **3072** embedding dimensions, a fixed `user_id`, and `infer=False` during ingestion. The benchmark is released at `https://github.com/yandex-research/StructMemEval` [2602.11243].

## 5. Empirical findings

The paper’s main empirical conclusion is that retrieval-augmented LLMs degrade sharply when the task requires maintaining a latent structure rather than recovering a few locally relevant snippets. The failure modes differ by task family, but the overall pattern is consistent: retrieval helps on small instances, larger \(k\) helps somewhat, and then performance collapses once the relevant state exceeds what top-\(k\) retrieval can surface [2602.11243].

### State tracking

The appendix gives detailed state-tracking results for both **gpt-4o-mini** and **gemini-2.5-pro**. The most striking pattern is the large gap between retrieval and agentic memory, particularly under the hint condition.

| Backbone | System | Average accuracy |
|---|---|---:|
| gpt-4o-mini | retrieval (top-5) | 24% |
| gpt-4o-mini | retrieval (top-20) | 40% |
| gpt-4o-mini | mem-agent (with hint) | 74% |
| gpt-4o-mini | Mem0 agent (with hint) | 60% |
| gemini-2.5-pro | retrieval (top-5) | 21% |
| gemini-2.5-pro | retrieval (top-20) | 26% |
| gemini-2.5-pro | mem-agent (with hint) | 79% |
| gemini-2.5-pro | Mem0 agent (with hint) | 81% |

The paper also reports **gemini-3-flash** with **Mem0 agent (hint)** at **95%** average accuracy on this subset [2602.11243]. These numbers support the paper’s claim that state tracking exposes a genuine memory-organization gap: retrieval often returns stale mentions, whereas structured memory agents can maintain the current latent state.

### Count-based tasks

For accounting/counting, the appendix shows that **gemini-2.5-pro** performs poorly because of hallucinations, which motivated the switch to **gemini-3-pro** for the main count-based evaluation. Under **gemini-3-pro**, retrieval remains competitive only on the smallest transaction sets and then collapses at higher counts.

| Method | Avg @10 tx | Avg @20 tx | Avg @50 tx |
|---|---:|---:|---:|
| Retrieval (top-5) | 0.2 | 0.0 | 0.0 |
| Retrieval (top-10) | 1.0 | 0.2 | 0.0 |
| Retrieval (top-20) | 1.0 | 0.73 | 0.0 |
| mem-agent (w/o hint) | 0.73 | 0.67 | 0.20 |
| mem-agent (with hint) | 0.67 | 0.47 | 0.27 |

The benchmark interprets this as a direct demonstration that larger retrieval budgets are not sufficient when the problem requires a maintained ledger or running aggregate. Even **top-20** retrieval reaches **0.0** at **50 transactions**, whereas mem-agent remains nonzero and improves slightly under hints at that scale [2602.11243].

### Tree-based tasks

For tree-based tasks, the provided text does not include exact appendix tables, but the qualitative result is consistent with the other subsets: retrieval can handle smaller instances, degrades as graph size and indirectness increase, and is outperformed by memory agents. The paper also states that performance improves substantially when a memory-organization hint is given, which reinforces the interpretation that part of the challenge is recognizing that the conversation should be turned into a graph-like representation [2602.11243].

### Interpretive conclusion

One of the paper’s most consequential findings is that the gap between **with hint** and **without hint** is often larger than the gap between **Mem0** and **mem-agent**. The paper therefore cautions against reading these experiments as definitive rankings of memory frameworks, especially because it uses **default hyperparameters** and **minimal tuning**. The stronger claim is that modern LLMs often possess enough procedural capacity to use structured memory once prompted, but do not reliably infer the required structure on their own [2602.11243].

## 6. Failure modes, limitations, and open extensions

The paper’s error analysis identifies three recurring accounting failures in agentic memory files: **omission of transactions**, **duplication of transactions**, and **hallucination of transactions**. It gives an example in which a nonexistent transaction labeled **“sandwich”** appears and a lunch transaction is duplicated. This is important because it shows that even when the agent adopts the right high-level structure, maintenance fidelity can still drift over long update sequences [2602.11243].

The paper suggests several reasons why LLMs fail to infer the right structure. One is a **training mismatch**: LLMs may know abstract structures such as trees, heaps, and state machines in coding or textbook contexts, yet have not been trained to apply those abstractions to their own memory management. Another is that natural conversational data are semantically messy, with irrelevant chatter, interleaved updates, and partial implications, making latent structure induction nontrivial. A third is that long update chains induce drift, especially in ledger-like tasks [2602.11243].

Several benchmark limitations are explicit. Current task coverage is restricted to **tree-based**, **state-tracking**, and **count-based** subsets. Future work is needed for **to-do/ordered lists**, **DAGs**, **assignment maps**, **indexes**, and multi-structure tasks. The benchmark is **synthetic**, which aids controllability and privacy but may not capture all real-user noise. Some evaluations rely on **LLM-as-a-judge**. Results are also sensitive to the backbone model, as shown by the need to replace **gemini-2.5-pro** with **gemini-3-pro** on counting tasks [2602.11243].

A plausible implication is that future progress may require both better benchmark coverage and architectural support for typed or scaffolded memory primitives, so that the burden of discovering ledgers, trees, or state registers does not fall entirely on the backbone model.

## 7. Position within long-term memory evaluation research

StructMemEval occupies a distinct niche within the 2026 landscape of LLM memory evaluation. Its contribution is not lifecycle-aware serving analysis, interactive scenario design, or dependency-aware world-state maintenance per se, but an explicit benchmark for whether an agent can **choose and maintain an appropriate memory structure** [2602.11243].

This focus differs from **Neuromem**, which studies external memory under a **streaming, lifecycle-aware, interleaved insertion-and-retrieval protocol** and decomposes the memory lifecycle into memory data structure, normalization, consolidation, query formulation, and context integration. Neuromem argues that static snapshot evaluation misses the main deployment difficulty because memory is an evolving service, and it reports that the memory data structure dominates the attainable quality frontier [2602.13967]. StructMemEval is narrower but more targeted: it does not benchmark the serving lifecycle, but instead probes whether structured organization itself is required for success.

It also differs from **StructMem**, which evaluates a **structure-enriched hierarchical memory framework** on **LoCoMo QA** rather than proposing a separate benchmark. StructMem operationalizes structured memory through event-level binding, temporal anchoring, and cross-event consolidation, with strong results on overall LoCoMo performance and temporal reasoning, but its evaluation remains tied to long-horizon dialogue QA [2604.21748]. StructMemEval abstracts away from one particular memory architecture and asks whether task families such as ledgers, trees, and state trackers can be solved at all without explicit organization.

Relative to **MEME**, StructMemEval targets a different structural axis. MEME formalizes persistent memory along **single vs. multi-entity** and **static vs. evolving** dimensions, introduces tasks such as **Cascade**, **Absence**, and **Deletion**, and shows that current systems collapse on dependency-aware post-change reasoning [2605.12477]. MEME therefore evaluates coherent evolving world state and dependency propagation, whereas StructMemEval evaluates whether the agent recognizes and maintains a task-specific organizational form.

Finally, StructMemEval shares with **MemGround** a dissatisfaction with static recall-style testing, but the two frameworks diverge in methodology. MemGround is natively grounded in interactive, gamified scenarios and evaluates **Surface State Memory**, **Temporal Associative Memory**, and **Reasoning-Based Memory** using metrics such as **QA Overall**, **MFU**, **MFCO**, and **ETD** [2604.14158]. StructMemEval is less process-heavy and more minimalistic: it emphasizes architecture independence, final-answer evaluation, and task families that become simple once the correct memory schema is in place.

Taken together, these comparisons situate StructMemEval as a benchmark for a specific but consequential failure mode in agent memory systems: the inability to spontaneously recognize that long-term memory should sometimes be maintained as a **ledger**, **state tracker**, or **tree**, even when the model can use such structures effectively once they are suggested.

Source: https://www.emergentmind.com/topics/structmemeval