Hierarchical Memory (H-MEM) Systems

• Hierarchical Memory (H-MEM) is a multi-level memory architecture that organizes raw data into atomic units and higher-level summaries for efficient long-term reasoning.
• H-MEM systems employ varied retrieval regimes—such as top-down routing, collapsed search, and LLM-guided navigation—to effectively access both detailed and abstracted information.
• Empirical evaluations show that H-MEM methods offer significant performance gains in dialogue and QA benchmarks while addressing challenges in dynamic updating and optimization trade-offs.

Hierarchical Memory (H-MEM) denotes a family of memory architectures in which stored information is organized across multiple levels of abstraction rather than as a flat collection of records. In contemporary language-agent and long-context systems, the central objective is to preserve fine-grained evidence while also constructing higher-level representatives that support efficient retrieval, compression, and long-horizon reasoning. A general formalization models raw data as atomic units produced by an extraction operator, recursively coarsened into higher layers by grouping and representative functions, and finally accessed by a traversal policy under a token budget (Talebirad et al., 23 Mar 2026). Within that broad concept, recent work spans tree-structured long-term memory for LLMs, segment-level recurrent transformers, hierarchical conversational memory, retrieval structures for web and embodied agents, and earlier neural memories that use tree navigation for sublinear access (Rezazadeh et al., 2024, He et al., 2024, Sun et al., 23 Jul 2025, Andrychowicz et al., 2016).

1. Formal definition and conceptual scope

A unifying theory describes hierarchical memory as a pipeline

$$D \xrightarrow{\alpha} G_0 \xrightarrow{C_1} G_1 \xrightarrow{C_2} \cdots \xrightarrow{C_L} G_L \xrightarrow{\tau} S,$$

where extraction $\alpha$ maps raw data to atomic information units, coarsening $C=(\pi,\rho)$ groups units and assigns representatives, and traversal $\tau$ retrieves a subset of atoms under a context budget (Talebirad et al., 23 Mar 2026). In this formulation, the hierarchy is not restricted to trees: strict tree structures arise when each unit belongs to exactly one parent, whereas overlapping group membership yields a DAG variant (Talebirad et al., 23 Mar 2026).

The same paper identifies a “self-sufficiency spectrum” for the representative function $\rho$,

$$\mathrm{SS}(\rho, G_j) = \frac{I(G_j;\rho(G_j))}{H(G_j)} = 1 - \frac{H(G_j \mid \rho(G_j))}{H(G_j)},$$

and argues that this quantity constrains viable retrieval strategies (Talebirad et al., 23 Mar 2026). Highly self-sufficient representatives, such as detailed summaries, can support retrieval directly at upper levels; low-self-sufficiency representatives, such as routing labels, mainly function as indices and therefore require top-down refinement (Talebirad et al., 23 Mar 2026). This coarsening-traversal coupling has become one of the principal organizing ideas in later H-MEM systems.

A recurrent misconception is that hierarchical memory is merely “summarization.” The theoretical treatment explicitly separates extraction, coarsening, and traversal, implying that summary quality alone does not determine system behavior; grouping quality and retrieval policy are equally central (Talebirad et al., 23 Mar 2026). A related misconception is that H-MEM is tied only to text archives. The cited systems instantiate the same abstraction over document corpora, dialogue histories, agent execution traces, navigation trajectories, and robotic task state (Talebirad et al., 23 Mar 2026).

2. Structural patterns and levels of organization

Recent H-MEM systems differ primarily in what constitutes a memory unit and how many levels are used. The architecture called H-MEM for long-term dialogue agents uses a four-layer hierarchy—Domain, Category, Memory Trace, and Episode—where the upper three layers act as a hierarchical index and the episode layer stores detailed interaction content together with timestamp and inferred user profile (Sun et al., 23 Jul 2025). This design emphasizes semantic abstraction plus positional index routing rather than globally searching the entire memory bank.

MemTree instead represents memory as a rooted tree $T=(V,E)$, with each node

$v = [c_v, e_v, p_v, \mathcal{C}_v, d_v],$

where $c_v$ is aggregated textual content, $e_v$ is an embedding, $\alpha$0 is the parent, $\alpha$1 the children, and $\alpha$2 the depth (Rezazadeh et al., 2024). Higher nodes store broader summaries; deeper nodes preserve more specific events or facts. This is a schema-like tree in which parent nodes are rewritten as descendants accumulate, so abstraction increases toward the root (Rezazadeh et al., 2024).

Other systems make the hierarchy explicitly asymmetric. HiMem for long-horizon dialogues separates Episode Memory, which stores temporally grounded dialogue segments, from Note Memory, which stores abstracted reusable knowledge such as facts, preferences, and profile information (Zhang et al., 10 Jan 2026). HiGMem adopts a two-level Event–Turn structure, using event summaries as semantic anchors for deciding which underlying turns are worth reading (Cao et al., 20 Apr 2026). HORMA organizes experience in a file-system-like workspace whose directories reflect entities, dates, topics, events, and relations, with summary notes linked back to raw trajectories by provenance markers (Hsu et al., 10 Jun 2026).

Not all H-MEM systems are text-first. HMT, the Hierarchical Memory Transformer, divides memory into sensory memory, short-term memory, and long-term memory at the segment level, preserving recent token embeddings, caching segment summaries, and recalling relevant segment memories during recurrent processing (He et al., 2024). HiMemVLN uses a two-level hierarchy in vision-language navigation: a short-term visual graph memory for localization and revisit suppression, and a long-term semantic schema memory for persistent task grounding (Lyu et al., 16 Mar 2026). HiMem-WAM similarly stores compact skill-level task tokens aligned with hierarchical latent actions and writes them at predicted skill boundaries rather than every timestep (sun et al., 9 Jun 2026).

Historically, Hierarchical Attentive Memory (HAM) is an earlier neural instantiation in which memory cells occupy the leaves of a full binary tree and internal nodes store routing summaries, giving $\alpha$3 access and update cost (Andrychowicz et al., 2016). This earlier usage establishes the computational idea of hierarchical navigation, although its setting is algorithmic learning rather than long-term agent memory.

3. Memory construction, consolidation, and online updating

A distinguishing feature of H-MEM is that stored items often evolve after insertion. MemTree performs online insertion by embedding new content, traversing downward by cosine similarity among child nodes, and deciding whether to descend or branch using a depth-dependent threshold

$\alpha$4

with $\alpha$5 and $\alpha$6 in the appendix (Rezazadeh et al., 2024). After insertion, all parent nodes on the traversal path are rewritten by an LLM-based aggregation step and re-embedded, so semantic information percolates upward and internal nodes become progressively more abstract (Rezazadeh et al., 2024). The paper explicitly connects this procedure to Online Top-Down clustering and gives an informal guarantee under a $\alpha$7-well-separated condition via the Moseley–Wang revenue objective (Rezazadeh et al., 2024).

The dialogue-oriented HiMem constructs Episode Memory through Topic-Aware Event–Surprise Dual-Channel Segmentation, introducing a boundary when there is either a topical shift or a salient discontinuity such as sudden intent or emotion change (Zhang et al., 10 Jan 2026). Note Memory is then built by a three-stage extraction pipeline: independently interpretable units, high-confidence implicit knowledge, and non-destructive normalization including deduplication, coreference resolution, and temporal normalization (Zhang et al., 10 Jan 2026). This suggests a construction regime in which the lower layer preserves contextual fidelity and the upper layer privileges durable reuse.

The H-MEM architecture of 2025 adds feedback-sensitive updating. After each user–LLM interaction, a memory extraction model parses the interaction into four semantic layers, encodes each layer into dense vectors, and supports a self-adaptation hierarchy adjustment interface that can reduce or increase the number of layers depending on conversation complexity (Sun et al., 23 Jul 2025). Update strength is not determined solely by forgetting curves: approval increases memory weight, absence of feedback follows natural shrinkage, and rebuttal decreases memory weight (Sun et al., 23 Jul 2025).

Hybrid structures extend consolidation beyond a pure hierarchy. H-Mem for agent QA builds a temporal-semantic tree in which leaf nodes store atomic events and higher nodes store summaries over progressively larger time windows such as day, week, month, and year, while simultaneously constructing a knowledge graph over entities and relations (Yu et al., 15 May 2026). Consolidation is permitted only when candidate nodes are both temporally close and semantically similar enough, using level-specific thresholds that decrease with abstraction level (Yu et al., 15 May 2026). A plausible implication is that temporal scope and semantic scope are treated as coupled dimensions of long-term memory rather than independent indices.

4. Retrieval regimes: top-down routing, collapsed search, and guided navigation

Retrieval in H-MEM systems is as important as storage structure. The four-layer H-MEM architecture retrieves top-down: it embeds the query, searches the highest abstraction layer with FAISS, selects top-$\alpha$8 entries, and then follows stored child indices recursively until the episode layer is reached (Sun et al., 23 Jul 2025). The recursive step is written as

$\alpha$9

so similarity is computed only within layer-local candidate sets rather than against the full memory bank (Sun et al., 23 Jul 2025). Under the simplified layout used in the paper, flat retrieval scales as $C=(\pi,\rho)$0, whereas hierarchical retrieval is approximated as $C=(\pi,\rho)$1 (Sun et al., 23 Jul 2025).

MemTree adopts a different regime: “collapsed tree” retrieval inspired by RAPTOR (Rezazadeh et al., 2024). All nodes are treated as a flat candidate set, the query embedding is compared to all node embeddings, nodes below a retrieval threshold are filtered out, and the top-$C=(\pi,\rho)$2 most similar nodes are returned (Rezazadeh et al., 2024). The paper reports that collapsed retrieval generally outperforms traversal retrieval unless traversal uses a large $C=(\pi,\rho)$3, because greedy path pruning can miss relevant deeper nodes (Rezazadeh et al., 2024). This empirical result is consistent with the theoretical claim that traversal policy should match representative self-sufficiency (Talebirad et al., 23 Mar 2026).

HiGMem inserts LLM judgment directly into retrieval. Given a query, the system retrieves event-level and turn-level candidates by similarity, then lets the LLM inspect event summaries and linked turns to predict which turns inside each event are likely to answer the query (Cao et al., 20 Apr 2026). The final evidence set is the union of directly retrieved turns and predicted turns, followed by filtering (Cao et al., 20 Apr 2026). The hierarchy is therefore not only a compression device; it is a coarse semantic search space that supports reasoning about utility before committing answer-stage context.

Several agent systems replace similarity-centric retrieval with navigation. HORMA uses a lightweight retrieval agent that traverses a file-system-like hierarchy via actions such as ls, cd, grep, cat, select, and done, and optimizes retrieval with Group Relative Policy Optimization plus an auxiliary Jaccard reward over evidence overlap (Hsu et al., 10 Jun 2026). The Hierarchical Memory Tree for web agents retrieves Intent nodes, then Stage nodes scored by semantic similarity plus a condition-matching term based on Jaccard similarity between stage conditions and current observation summaries, and finally Action exemplars that the Actor grounds to current UI elements (Tan et al., 7 Mar 2026). Mnemis combines a fast System-1 route over a base graph with a System-2 Global Selection route that performs top-down browsing through a hierarchical graph, using LLM judgments over category names and tags rather than an explicit vector score (Tang et al., 17 Feb 2026).

These variants clarify a second common misconception: hierarchical memory does not imply a single standard retrieval algorithm. The literature includes top-down FAISS routing, collapsed global search, LLM-guided event-to-turn expansion, RL-trained directory navigation, and dual-route graph browsing (Sun et al., 23 Jul 2025, Rezazadeh et al., 2024, Cao et al., 20 Apr 2026, Hsu et al., 10 Jun 2026, Tang et al., 17 Feb 2026).

5. Empirical performance across domains

In long-horizon dialogue, H-MEM and related systems are chiefly evaluated on LoCoMo and related benchmarks. The four-layer H-MEM reports average improvements of +14.98 F1 and +12.77 BLEU-1 over five baselines across model and task configurations, with especially large gains on Multi-Hop (+21.25 F1, +17.65 BLEU-1) and Adversarial (+16.71 F1, +12.03 BLEU-1) settings (Sun et al., 23 Jul 2025). It also reports inference time below 100 ms at maximum memory load, compared with MemoryBank exceeding 400 ms (Sun et al., 23 Jul 2025). HiGMem obtains the best F1 on four of five LoCoMo10 categories and improves adversarial F1 from 0.54 to 0.78 over A-Mem while retrieving 8.09 turns on average versus 99.84 for A-Mem (Cao et al., 20 Apr 2026). HiMem for long-horizon dialogues reports overall GPT-Score 80.71 and F1 34.95 in hybrid retrieval mode, outperforming A-MEM, SeCom, and Mem0 on LoCoMo, with particular strength on Multi-Hop and Temporal tasks (Zhang et al., 10 Jan 2026).

MemTree broadens the evidence base to both dialogue and document QA. On MSC, among query-plus-memory methods it reaches 84.8% accuracy and 79.9 ROUGE-L recall; on MSC-E it achieves 82.5% overall accuracy, outperforming MemoryStream’s 80.7%; on QuALITY it reaches 59.8% overall accuracy; and on MultiHop RAG it obtains 80.5% overall accuracy, surpassing MemoryStream and matching or exceeding offline methods on the hardest query types, especially temporal reasoning (Rezazadeh et al., 2024). The paper notes a mild recency bias due to sequential updating, but still reports dominance across evidence ranges in MSC-E (Rezazadeh et al., 2024).

For long-context sequence modeling, HMT reports consistent improvements over context-constrained backbones and prior memory-augmented methods. Reported gains include 25.5% better on Wikitext-103 and 11.4% better on PG-19 for OPT 2.7B + HMT, and 17.6% better on Wikitext-103 and 9.48% better on PG-19 for OpenLlamaV2 3B + HMT (He et al., 2024). It also reports comparable or superior generation quality to long-context LLMs with $C=(\pi,\rho)$4 fewer parameters and $C=(\pi,\rho)$5 less inference memory (He et al., 2024).

Agentic and embodied settings show similar patterns. HiMemVLN identifies “Navigation Amnesia” and reports simulated VLN-CE gains from SR 14 to 30 and real-world gains from SR 18 to 32 over open-source baselines (Lyu et al., 16 Mar 2026). HiMem-WAM reports 26.3% total average SR on RMBench and attributes its gains to memory-dependent long-horizon manipulation, with larger improvements on Medium and Hard real-world tasks than on Easy tasks (sun et al., 9 Jun 2026). The web-agent Hierarchical Memory Tree reports WebArena TaskSR 38.7 versus 35.5 for AWM and 32.1 for Flat Retrieval, together with strong cross-website improvements on Mind2Web (Tan et al., 7 Mar 2026). HORMA improves performance on ALFWorld, LoCoMo, and LongMemEval while using only about 3.07%–22.17% of baseline token usage on LoCoMo and 1.24%–16.19% on LongMemEval (Hsu et al., 10 Jun 2026).

6. Historical antecedents, limitations, and open questions

The computational intuition behind H-MEM predates contemporary LLM memory systems. The Hierarchical Memory Model of computation describes non-uniform memory levels with different access costs and shows that binary search tree structure and memory placement must be optimized jointly under such a hierarchy (0804.0940). In neural memory, HAM demonstrated that hierarchical tree navigation can reduce memory access from $C=(\pi,\rho)$6 to $C=(\pi,\rho)$7 while learning tasks such as merging, sorting, and binary search from input-output examples (Andrychowicz et al., 2016). Hierarchical Memory Networks subsequently explored a hybrid of hard and soft attention using Maximum Inner Product Search to restrict softmax to a candidate subset, improving scalability for large-scale QA (Chandar et al., 2016). These earlier works do not address modern long-term agent memory directly, but they establish two enduring H-MEM principles: structural routing and sublinear or reduced-cost access.

Despite broad empirical success, the literature is explicit about unresolved problems. Dynamic evolution remains difficult to formalize: the general theory notes that insertion, restructuring, split/merge, decay, and reinforcement complicate the simple fixed-hierarchy picture and can break the underlying Markov-chain arguments when coarsening becomes query-conditioned or stateful (Talebirad et al., 23 Mar 2026). Several systems also expose trade-offs between precision, recall, and cost. HiGMem shifts cost from answer time to memory construction and retrieval reasoning, so its practical advantage depends on a deployment in which memory operations are relatively cheap and the final answer model is expensive (Cao et al., 20 Apr 2026). Mnemis reports substantial token and runtime costs for graph construction and hierarchical selection, and currently rebuilds its hierarchy periodically rather than incrementally (Tang et al., 17 Feb 2026). HMT retains only the most recent $C=(\pi,\rho)$8 memory embeddings and therefore “cannot recall more than $C=(\pi,\rho)$9 segments” (He et al., 2024). HiMemVLN depends on hand-designed thresholds and external visual tools, while HiMem-WAM remains below MEM-0 on harder $\tau$0 tasks (Lyu et al., 16 Mar 2026, sun et al., 9 Jun 2026).

A broader controversy concerns what upper-layer nodes should contain. Some systems use substantive summaries and therefore support collapsed search or direct answering from non-leaf nodes; others use labels or compact schema elements that mainly support routing (Talebirad et al., 23 Mar 2026). This is not merely an implementation detail. Theoretical analysis argues that mismatching representative type and traversal strategy wastes budget: collapsed search over referential labels wastes tokens on weak summaries, while top-down traversal through already self-sufficient summaries expands unnecessarily (Talebirad et al., 23 Mar 2026). The present literature therefore suggests that H-MEM is best understood not as a single architecture, but as a design space defined by four coupled choices: the atomization of experience, the grouping map, the representative function, and the traversal policy (Talebirad et al., 23 Mar 2026).