StructMem: Hierarchical Memory for Conversational Agents

• StructMem is a hierarchical memory framework that enriches conversational agents with structured, event-level memory to improve temporal reasoning.
• It employs dual-perspective extraction to capture both factual and relational details from each utterance, anchoring them with a timestamp.
• Periodic semantic consolidation synthesizes cross-event connections, enabling efficient memory retrieval with fewer tokens, API calls, and runtime.

StructMem is a structure-enriched hierarchical memory framework for long-horizon conversational agents. It is designed for settings in which memory must capture relationships between events rather than isolated facts, with the central memory unit being a temporally grounded relational event rather than a flat fact or a graph triplet. The framework combines event-level bindings, temporal anchoring, dual-perspective extraction, and periodic semantic consolidation to preserve local event structure while inducing cross-event connections without explicit graph construction. On \texttt{LoCoMo}, StructMem is reported to improve temporal reasoning and overall long-context question answering while using substantially fewer build tokens, API calls, and runtime than several prior memory systems (Xu et al., 23 Apr 2026).

1. Problem formulation and conceptual position

StructMem addresses persistent memory for long-term conversational agents that interact over many sessions and must answer questions requiring temporal dependencies, causal chains, interpersonal relations, and multi-hop connections across turns and sessions. The motivating benchmark context is \texttt{LoCoMo}, where conversations average about 588 turns and 16,618 tokens and span up to 32 sessions. In this regime, the framework argues that long-horizon dialogue memory should not be modeled as a bag of disconnected facts, because facts are often meaningful only within an event context and many questions require reconstructing cross-event structure (Xu et al., 23 Apr 2026).

The framework is explicitly positioned between two prior paradigms. Flat memory is described as simple and efficient, but as storing facts or summaries as independent units, which weakens temporal reasoning and multi-hop question answering. Graph memory is described as preserving entities and relations explicitly, but as incurring expensive and fragile graph construction, cascading extraction pipelines, entity-resolution overhead, and rigid schemas. StructMem is proposed as a middle ground: it preserves event-level bindings and induces cross-event structure, but represents that structure in natural language plus temporal anchoring rather than as an explicit symbolic graph (Xu et al., 23 Apr 2026).

This design choice aligns with a broader line of work that treats memory organization itself as a first-class problem. For example, StructMemEval argues that long-term memory competence should be evaluated by an agent’s ability to organize knowledge into task-appropriate structures rather than merely retrieve facts, and MemIR argues that flat untyped storage causes provenance-role collapse by conflating evidence, retrieval cues, and claims (Shutova et al., 11 Feb 2026, Jin et al., 25 May 2026). This suggests that StructMem belongs to a broader shift from flat archival memory toward explicitly organized memory substrates.

2. Memory hierarchy and representation

StructMem has a two-level memory organization. At the first level, it stores event-level memory entries extracted from each utterance. For each utterance $m_i$, the system produces factual entries $\Phi_i$ and relational entries $\Psi_i$, and stores each extracted entry $x$ together with its embedding $\mathbf{e}_x$ and timestamp $\tau_i$:

$$\mathcal{M} \leftarrow \bigcup_{i=1}^{N} \left\{ \langle x, \mathbf{e}_x, \tau_i \rangle \mid x \in \Phi_i \cup \Psi_i \right\}.$$

This event-level store is the core episodic memory. It is lightweight and text-centric: the explicitly described data structures are entry text $x$, embedding $\mathbf{e}_x$, timestamp $\tau$, the factual set $\Phi_i$0, the relational set $\Phi_i$1, the unconsolidated buffer $\Phi_i$2, retrieved semantic seeds $\Phi_i$3, reconstructed events $\Phi_i$4, and consolidated memory $\Phi_i$5 (Xu et al., 23 Apr 2026).

At the second level, StructMem maintains cross-event synthesis memory. This is produced periodically rather than per event, and it functions as a higher-level abstraction over semantically related and temporally reconstructed event clusters. During inference, the framework retrieves from both levels. The appendix implementation details state that question answering retrieves 60 entries and 5 synthesis memories, so the effective inference-time memory is dual-source: atomic event entries plus synthesized consolidations (Xu et al., 23 Apr 2026).

A key representational property is that event-level bindings are preserved implicitly by shared timestamps rather than explicit edges. If two entries share $\Phi_i$6, they belong to the same event-level unit. Cross-event relations are likewise not stored as graph edges; they are induced through semantic retrieval, timestamp-based event reconstruction, and natural-language synthesis. This gives StructMem a hierarchical structure without requiring explicit graph maintenance (Xu et al., 23 Apr 2026).

3. Dual perspectives and temporal anchoring

The phrase “temporally anchoring dual perspectives” refers to the combination of two operations. First, each utterance is processed from two perspectives: a factual perspective and a relational perspective. Second, all extracted entries are attached to the same originating timestamp. The dual-perspective extraction is defined as:

$\Phi_i$7

where $\Phi_i$8 is the factual extraction prompt, $\Phi_i$9 is the relational extraction prompt, and $\Psi_i$0 is the LLM (Xu et al., 23 Apr 2026).

The factual extractor produces entries denoted

$\Psi_i$1

while the relational extractor produces

$\Psi_i$2

The factual channel captures event content—what happened, who did what, what was stated. The relational channel captures interactional dynamics, causal influence, interpersonal relation, and temporal dependency. The framework’s claim is that factual entries alone do not preserve enough connective tissue for questions involving co-participation, influence, cause, or relative time expressions such as “last year” (Xu et al., 23 Apr 2026).

Temporal anchoring is then realized by assigning every entry extracted from utterance $\Psi_i$3 the same timestamp $\Psi_i$4, using the storage equation above. The timestamp is operational rather than merely metadata. It supports event reconstruction, chronological ordering, and grounded synthesis. The full event associated with a retrieved seed entry $\Psi_i$5 is reconstructed by:

$\Psi_i$6

The appendix case study illustrates the effect with the question “When did Caroline and Melanie go to a pride festival together?” StructMem answers correctly because it stores not only factual mentions but also relational context and synthesis memory that make implicit co-participation explicit (Xu et al., 23 Apr 2026).

4. Semantic consolidation and retrieval

StructMem performs consolidation periodically rather than after every utterance. The reported implementation uses a 1 hour threshold. Recent unconsolidated entries are placed in a buffer $\Psi_i$7, then sorted by timestamp:

$\Psi_i$8

The buffered entries are concatenated into an aggregated query, embedded, and used to retrieve semantically similar historical seed entries. The implementation detail given in the appendix is top-15 retrieval seeds for cross-event consolidation. For each retrieved seed $\Psi_i$9, the system reconstructs the entire historical event by timestamp using $x$0, then forms a cross-event context:

$x$1

Finally, the LLM synthesizes a consolidated memory:

$x$2

The paper describes this output as a complementary abstraction layer rather than a replacement for raw episodic memory (Xu et al., 23 Apr 2026).

This procedure is the mechanism by which StructMem induces cross-event links without explicit graphs. Semantic retrieval links buffered events to related historical seeds; timestamp reconstruction restores whole events around those seeds; synthesis then writes a natural-language summary that can express repeated preferences, causal progression, temporal continuity, or shared participation. The paper emphasizes that this is not generic window summarization. It is synthesis over semantically reconstructed event clusters (Xu et al., 23 Apr 2026).

At inference time, StructMem uses a dual-circuit retrieval procedure that combines retrieved atomic event entries and retrieved synthesis memories. The paper does not provide a formal inference-time ranking equation beyond semantic ranking by cosine similarity, and it does not specify a learned consolidation loss or merge objective. Consolidation remains a prompt-based LLM synthesis process rather than a trained optimization module (Xu et al., 23 Apr 2026).

5. Experimental results, fidelity, and efficiency

StructMem is evaluated on the \texttt{LoCoMo} question answering task, using 10 long-term conversations with average 588 turns, average 16,618 tokens, and up to 32 sessions. The evaluation categories are Single-hop (841 questions), Multi-hop (282 questions), Temporal (321 questions), and Open-domain (96 questions). The experimental setup uses gpt-4o-mini as the backbone LLM and text-embedding-3-small as the embedding model. Evaluation is performed with LLM-as-a-judge, and robustness is checked across gpt-4o-mini, Qwen2.5-32B-Instruct, and DeepSeek-V3.2, with reported inter-judge agreement of Fleiss’ $x$3, pairwise Pearson $x$4, and $x$5 (Xu et al., 23 Apr 2026).

On the main benchmark table, StructMem reports 76.82 overall, with 68.77 on multi-hop, 46.88 on open-domain, 81.09 on single-hop, and 81.62 on temporal reasoning. These results place it above the reported overall scores of OpenAI (71.82), FullContext (73.83), Mem0 (66.88), Zep (75.14), and Memobase (75.78). It is especially strong on temporal reasoning, where only Memobase is higher at 85.05. In the paradigm comparison, StructMem exceeds the reported “Flat Memory,” “Graph Memory,” and “w/o Cross-Event” variants on both single-session and temporal reasoning, supporting the claim that cross-event consolidation adds value beyond event-level structure alone (Xu et al., 23 Apr 2026).

The efficiency profile is one of the framework’s strongest reported properties. StructMem uses 1.501M build input tokens and 0.436M output tokens, for a total of 1.937M build tokens, with 1056 API calls and 22854 s runtime. The paper highlights the contrast with graph-heavy alternatives such as Mem0$x$6, which reports 35.825M total build tokens, 53,514 API calls, and 115670 s runtime. The argument is that graph memory requires multiple cascading LLM operations per event—entity extraction, entity deduplication, relation extraction, and relation deduplication—whereas StructMem shifts cross-event organization into periodic batched consolidation (Xu et al., 23 Apr 2026).

The paper also reports fidelity audits. Event-level extraction has an average hallucination rate of 2.36% across conversations. For cross-event consolidation, the default constrained setting reports spurious link rates of 0.61% under the GPT judge, 3.41% under the Qwen judge, and 3.63% under the DeepSeek judge. In an unconstrained variant, the corresponding hallucinated cross-event link rates rise to 7.45%, 20.00%, and 15.14%, respectively. The authors interpret this as evidence that temporal and grounding constraints in the synthesis prompt materially improve faithfulness (Xu et al., 23 Apr 2026).

6. Broader significance, limitations, and relation to later structured-memory work

StructMem’s broader significance lies in its reformulation of conversational memory around the temporally grounded relational event. Rather than choosing between flat retrieval and explicit graph construction, it preserves factual and relational content together at event level and then induces higher-order connections through periodic synthesis. A plausible implication is that its main contribution is not merely improved recall, but a specific memory unit and update schedule: event-local binding first, cross-event abstraction second (Xu et al., 23 Apr 2026).

The paper also notes limitations. The quality of dual-perspective extraction depends heavily on prompt quality, and the framework currently lacks an explicit mechanism for revising outdated facts, resolving contradictions, or updating user preferences over time. Open-domain performance also lags behind some competing systems, notably Zep. These limitations place StructMem within an ongoing research agenda rather than as a complete solution to long-term agent memory (Xu et al., 23 Apr 2026).

Subsequent structured-memory work makes the surrounding design space more explicit. StructMemEval argues that many benchmarks reward retrieval more than structure and that memory agents often fail because they do not recognize the appropriate memory schema unless prompted (Shutova et al., 11 Feb 2026). Memora proposes a harmonic memory representation with primary abstractions, concrete memory values, and cue anchors, and formally shows that standard RAG and KG-based memory can emerge as special cases (Xia et al., 3 Feb 2026). MemIR argues for typed memory atoms that separate page atoms, span atoms, handle atoms, time atoms, pivot atoms, and claim atoms, with only claims treated as truth-bearing and all retrieval routed through claim-centered bundles (Jin et al., 25 May 2026). Together, these works suggest that StructMem occupies an influential position in a broader transition from flat conversational memory toward structured, hierarchical, and provenance-sensitive memory architectures.

In embodied and interactive domains, related ideas reappear under different forms. eMEM proposes a hybrid graph-based spatio-temporal memory with observation, episode, gist, and entity nodes, plus SQL, HNSW, and R-tree indexing for embodied agents (Rasheed et al., 2 Jun 2026). AndroTMem introduces Anchored State Memory, where long-horizon GUI interaction trajectories are converted into sparse typed anchors with causal links (Shi et al., 19 Mar 2026). MemCompiler reframes memory usage as state-conditioned memory compilation over a structured Brief State rather than static memory injection (Ding et al., 8 May 2026). These systems target different settings, but they reinforce a shared lesson that StructMem helped crystallize for conversational agents: long-horizon performance depends not only on storing more context, but on preserving and exploiting structure within memory.