Hierarchical Event Memory

• Hierarchical Event Memory is a structured memory architecture that organizes experience into semantically coherent events across multiple abstraction levels.
• It employs event segmentation, summarization, and dual-layer retrieval to preserve detailed context while compressing long-term information.
• Applications span streaming video, dialogue agents, and web systems, demonstrating improvements in retrieval precision and bounded-context reasoning.

Searching arXiv for papers on hierarchical event memory and related structured memory systems. Hierarchical event memory denotes a class of memory architectures that organize experience into event-structured units across multiple levels of abstraction rather than storing a flat sequence of frames, turns, or retrieved chunks. In recent agent and video-understanding systems, the term refers to mechanisms that preserve semantically coherent episodes, compress or consolidate them into higher-level representations, and support selective recovery of detailed evidence when needed. Across streaming video, long-horizon dialogue, conversational QA, and web agents, hierarchical event memory is motivated by a recurrent systems problem: incoming history is effectively unbounded, whereas model context windows, retrieval budgets, and inference-time attention remain bounded. The resulting architectures typically separate recent high-fidelity memory from long-term structured memory, use event segmentation or analogous boundary detection to define memory units, and couple retrieval with abstraction level so that coarse representations guide access to fine-grained evidence (Wen et al., 17 Feb 2026, Talebirad et al., 23 Mar 2026).

1. Conceptual foundations and scope

Hierarchical event memory has emerged as a response to the inadequacy of flat memory stores for long-horizon reasoning. In online video understanding, the core conflict is between an unbounded input stream and a finite multimodal LLM context window, which creates simultaneous pressure to preserve long-range history and retain fine-grained recent detail (Wen et al., 17 Feb 2026). In dialogue and language-agent settings, analogous failures appear as scattered retrieval, semantic fragmentation, workflow mismatch, and context dilution, all of which arise when systems retrieve isolated chunks or linear traces without preserving the event structure that gave those items meaning (Lu et al., 10 Jan 2026, Zou et al., 12 Jan 2026, Tan et al., 7 Mar 2026).

A common feature across the literature is that memory units are treated as events, episodes, stages, or other semantically coherent segments rather than arbitrary windows. In ES-Mem, a dialogue event is defined as “a temporal segment centered on a stable topic or intent, characterized by high internal semantic homogeneity and bounded by distinct shifts in topic, task phases, or interaction patterns” (Zou et al., 12 Jan 2026). In EventMemAgent, the stream is organized into a set of short-term events $\mathcal{E}_{st} = \{E_1, E_2, \dots, E_m\}$, where completed events and the current active event coexist within a fixed-capacity short-term memory (Wen et al., 17 Feb 2026). In PyraVid, events are embedded within a coarse-to-fine multimodal pyramid consisting of fact memory, clip memory, and global memory (Yan et al., 16 May 2026).

This suggests that hierarchical event memory is best understood not as a single implementation, but as a design principle: preserve event boundaries, maintain multiple abstraction levels, and ensure that summary-level reasoning remains linked to raw evidence. A plausible implication is that the event abstraction serves two distinct roles: compression for bounded inference, and indexing for targeted reconstruction.

2. Structural principles of hierarchical event memory

Recent systems converge on layered organizations that separate detailed episodic traces from more durable abstractions. The most common form is a dual-layer or multi-layer hierarchy in which lower layers preserve temporally local, high-fidelity evidence while upper layers store summaries, structured tuples, or semantic nodes.

In EventMemAgent, the hierarchy consists of short-term memory and long-term memory. The short-term layer is event-organized rather than frame-flat, with event capacity constrained by $\sum_{i=1}^{m} n_i \le K$, while the long-term layer archives past events as structured tuples $$E_i' = \{I_i^{first}, c_i, \mathbf{e}_i, \Delta_i\}$$ containing a first-frame anchor, a caption, a semantic embedding, and a change log (Wen et al., 17 Feb 2026). In HiGMem, the two levels are event summaries and dialogue turns; event nodes act as semantic anchors that let the model inspect higher-level event descriptions before deciding which turns are worth reading (Cao et al., 20 Apr 2026). In HiMem, the two linked layers are Episode Memory and Note Memory, where episodic records preserve temporally grounded interaction segments and notes capture stable knowledge such as facts, preferences, and profiles (Zhang et al., 10 Jan 2026).

Other architectures generalize the same principle. SEEM separates a Graph Memory Layer storing relational quadruples $$\mathcal{K}_t = \{ (s, r, o, \tau) \mid s, o \in \mathcal{E}, r \in \mathcal{R}, \tau \in \mathcal{T} \}$$ from an Episodic Memory Layer storing Episodic Event Frames, thereby distinguishing stable relational facts from narrative progression (Lu et al., 10 Jan 2026). StructMem preserves event-level bindings as timestamp-anchored factual and relational entries and periodically synthesizes higher-level consolidated memories across events (Xu et al., 23 Apr 2026). GAM separates an Event Progression Graph from a Topic Associative Network, with archived event graphs and cross-layer links mediating between the two (Wu et al., 14 Apr 2026). HMT for web agents uses a three-level hierarchy—Intent, Stage, and Action—to decouple logical planning from site-specific execution details (Tan et al., 7 Mar 2026).

Theoretical work abstracts these systems using three operators: extraction $\alpha$, coarsening $C=(\pi,\rho)$, and traversal $\tau$, where extraction creates atomic units, coarsening groups them and assigns representatives, and traversal selects atomic content under a token budget (Talebirad et al., 23 Mar 2026). Particularly important is the self-sufficiency spectrum of the representative function $\rho$: high-self-sufficiency summaries can sometimes answer queries directly, whereas low-self-sufficiency representatives function primarily as routing labels (Talebirad et al., 23 Mar 2026). This provides a formal explanation for why some hierarchical memories use coarse summaries for direct reasoning while others use them mainly to guide deeper expansion.

3. Event segmentation and memory construction

The defining operation in hierarchical event memory is the construction of event units. Systems differ in modality and detector design, but most implement some boundary-sensitive process to prevent semantically incoherent memory units.

In streaming video, EventMemAgent performs online event segmentation at 1 FPS. For each incoming frame $f_t$, it computes a normalized grayscale histogram $\mathbf{h}_t$ and compares it with the average histogram of the active event $\sum_{i=1}^{m} n_i \le K$0 using Pearson correlation,

$\sum_{i=1}^{m} n_i \le K$1

A boundary is triggered when $\sum_{i=1}^{m} n_i \le K$2; the appendix adds a minimum duration constraint requiring the current event length to exceed 8 frames in order to avoid spurious boundaries from visual jitter (Wen et al., 17 Feb 2026). HEM-LLM segments long videos by computing adjacent-frame cosine similarity over pooled image descriptors and choosing the $\sum_{i=1}^{m} n_i \le K$3 smallest similarity values as segmentation points, producing event segments $\sum_{i=1}^{m} n_i \le K$4 (Cheng et al., 2024). OASIS constructs event nodes from sequential, non-overlapping temporal windows and later merges them upward into an Event Forest when the root-set size exceeds a threshold (Liang et al., 18 Apr 2026).

In dialogue, ES-Mem uses a two-stage Dynamic Event Segmentation module. Topic Coherence Detection computes mutual information proxies between adjacent turn-topic embeddings under a Gaussian assumption,

$\sum_{i=1}^{m} n_i \le K$5

selects candidates via a quantile threshold, and then refines them with an LLM-based Intent-Aware Boundary Refinement stage that assigns boundary-positive or boundary-negative labels and accepts boundaries whose confidence exceeds $\sum_{i=1}^{m} n_i \le K$6 (Zou et al., 12 Jan 2026). HiMem uses a Topic-Aware Event–Surprise Dual-Channel Segmentation strategy in which a boundary is introduced when either a topical shift or a salient discontinuity fires under an OR rule (Zhang et al., 10 Jan 2026). GAM detects semantic shift with an LLM-based discriminator on sparse maintenance events such as session end markers, natural pauses, and buffer overflow, formalized by

$\sum_{i=1}^{m} n_i \le K$7

although the paper notes that $\sum_{i=1}^{m} n_i \le K$8 is not computed directly in practice (Wu et al., 14 Apr 2026).

Memory construction usually includes not only segmentation but also structured representation. SEEM converts passages into Episodic Event Frames containing provenance pointers, summaries, and event attributes including Participants, Action, Time, Location, Causality, and Manner (Lu et al., 10 Jan 2026). EventMemAgent converts evicted short-term events into tuples with anchors, captions, embeddings, and change logs (Wen et al., 17 Feb 2026). ES-Mem constructs three-level memory units $\sum_{i=1}^{m} n_i \le K$9 comprising refined boundaries, event summaries, raw context, and timestamps (Zou et al., 12 Jan 2026).

A recurrent design decision is that segmentation should preserve semantic integrity rather than enforce uniform chunk size. The explicit critique of fixed-length segmentation appears in EventMemAgent, ES-Mem, HEM-LLM, and HiMem, each of which associates rigid chunking with fragmentation of coherent events (Wen et al., 17 Feb 2026, Zou et al., 12 Jan 2026, Cheng et al., 2024, Zhang et al., 10 Jan 2026).

4. Retrieval, traversal, and evidence reconstruction

Hierarchical event memory changes not only what is stored but how retrieval is performed. Retrieval is typically coarse-to-fine: coarse structures identify a promising episode or stage, and only then does the system descend to summaries, turns, frames, or raw passages.

EventMemAgent supports temporal retrieval by interval overlap,

$$E_i' = \{I_i^{first}, c_i, \mathbf{e}_i, \Delta_i\}$$0

and semantic retrieval by cosine similarity between query embedding $$E_i' = \{I_i^{first}, c_i, \mathbf{e}_i, \Delta_i\}$$1 and stored event embedding $$E_i' = \{I_i^{first}, c_i, \mathbf{e}_i, \Delta_i\}$$2, with the appendix specifying that only the top-3 events with similarity score greater than 0.3 are returned (Wen et al., 17 Feb 2026). ES-Mem first scans refined boundary representations as anchors, expands the surrounding interval, and then reranks candidates using a mixture of summary similarity and contextual anchor support,

$$E_i' = \{I_i^{first}, c_i, \mathbf{e}_i, \Delta_i\}$$3

before returning raw context for generation (Zou et al., 12 Jan 2026). HiGMem retrieves candidate events and turns in parallel, then uses event summaries as semantic anchors so that the LLM predicts which turns inside each event are worth reading (Cao et al., 20 Apr 2026).

SEEM’s Reverse Provenance Expansion is a particularly explicit reconstruction mechanism. After graph-layer retrieval produces seed facts and passages, the system uses the event-frame provenance to expand from retrieved fragments into all passages that contributed to the consolidated event,

$$E_i' = \{I_i^{first}, c_i, \mathbf{e}_i, \Delta_i\}$$4

with the practical constraint that the final expanded evidence set is capped at at most twice the initial retrieval budget (Lu et al., 10 Jan 2026). StructMem similarly reconstructs complete event context by retrieving seed entries, recovering all entries with the same timestamp,

$$E_i' = \{I_i^{first}, c_i, \mathbf{e}_i, \Delta_i\}$$5

and then synthesizing cross-event context from buffered and reconstructed material (Xu et al., 23 Apr 2026).

Video systems introduce structured expansion over event hierarchies. OASIS answers first from a coarse context consisting of a short window, a medium buffer, root summaries from an Event Forest, and a QA summary. Only when the model emits <tool_call> does it derive an internal retrieval intent $$E_i' = \{I_i^{first}, c_i, \mathbf{e}_i, \Delta_i\}$$6, retrieve up to $$E_i' = \{I_i^{first}, c_i, \mathbf{e}_i, \Delta_i\}$$7 event nodes and $$E_i' = \{I_i^{first}, c_i, \mathbf{e}_i, \Delta_i\}$$8 QA pair, and produce a final answer from both coarse and fine context (Liang et al., 18 Apr 2026). PyraVid starts from semantically similar fact nodes,

$$E_i' = \{I_i^{first}, c_i, \mathbf{e}_i, \Delta_i\}$$9

and iteratively expands through structured links

$$\mathcal{K}_t = \{ (s, r, o, \tau) \mid s, o \in \mathcal{E}, r \in \mathcal{R}, \tau \in \mathcal{T} \}$$0

while pruning candidates with an agentic filter $$\mathcal{K}_t = \{ (s, r, o, \tau) \mid s, o \in \mathcal{E}, r \in \mathcal{R}, \tau \in \mathcal{T} \}$$1 (Yan et al., 16 May 2026).

A common misconception is that hierarchical memory is simply a larger retrieval system. The cited systems instead show that hierarchy alters retrieval semantics: summaries or boundaries function as anchors, routing labels, or stage validators, and the retrieval unit is often an event neighborhood rather than an isolated nearest neighbor. This is precisely the coarsening–traversal coupling emphasized in the formal theory: retrieval behavior must match the informational role of the representative (Talebirad et al., 23 Mar 2026).

5. Domain-specific realizations

The recent literature contains several distinct realizations of hierarchical event memory, each adapted to the modality and reasoning task.

Domain	Representative system	Hierarchical organization
Streaming video reasoning	EventMemAgent (Wen et al., 17 Feb 2026)	STM event buffer + LTM event tuples
Long video reasoning	PyraVid (Yan et al., 16 May 2026)	fact memory + clip memory + global memory
Streaming video QA	OASIS (Liang et al., 18 Apr 2026)	short window + medium buffer + Event Forest
Long dialogue agents	HiMem (Zhang et al., 10 Jan 2026)	Episode Memory + Note Memory
Dialogue episodic retrieval	ES-Mem (Zou et al., 12 Jan 2026)	refined boundary + summary + raw context
Conversational QA	SEEM (Lu et al., 10 Jan 2026)	Graph Memory Layer + Episodic Memory Layer
Web agents	HMT (Tan et al., 7 Mar 2026)	Intent + Stage + Action

In online video understanding, EventMemAgent combines event-aware short-term buffering, event-granular reservoir sampling, structured long-term archival, and a multi-granular perception toolkit comprising search_memory, [OCR](https://www.emergentmind.com/topics/optimal-common-resource-ocr), and detect_objects (Wen et al., 17 Feb 2026). In online video temporal grounding, HEM stores historical event proposals at multiple scales, allocates memory capacity by scale according to positive-sample frequency,

$$\mathcal{K}_t = \{ (s, r, o, \tau) \mid s, o \in \mathcal{E}, r \in \mathcal{R}, \tau \in \mathcal{T} \}$$2

and updates overfull memory by merging adjacent events whose cosine similarity exceeds $$\mathcal{K}_t = \{ (s, r, o, \tau) \mid s, o \in \mathcal{E}, r \in \mathcal{R}, \tau \in \mathcal{T} \}$$3, otherwise falling back to FIFO (Zheng et al., 6 Aug 2025).

In long dialogue, HiMem uses event-segmented episodes as grounding evidence and notes as stable knowledge, with best-effort retrieval querying Note Memory first and descending to Episode Memory only if a fixed deterministic self-evaluation prompt judges the note evidence insufficient (Zhang et al., 10 Jan 2026). ES-Mem uses refined boundaries as “high-level cognitive anchors,” first retrieving the right event boundary, then unpacking its associated episode (Zou et al., 12 Jan 2026). GAM isolates live interaction in an Event Progression Graph with a 2048-token buffer and consolidates only at semantic shifts into topic nodes $$\mathcal{K}_t = \{ (s, r, o, \tau) \mid s, o \in \mathcal{E}, r \in \mathcal{R}, \tau \in \mathcal{T} \}$$4 that retain both summary and raw evidence (Wu et al., 14 Apr 2026).

For web agents, HMT operationalizes hierarchical event memory as a workflow memory: intent nodes normalize user instructions, stage nodes encode reusable subgoals with pre-conditions and post-conditions, and action nodes store semantic action patterns rather than brittle site-specific identifiers (Tan et al., 7 Mar 2026). This suggests that the concept extends naturally beyond temporal events in video or dialogue to structured task progression.

A broader theoretical generalization also appears in sequence science. The study of bursty trains in empirical event sequences argues that events are hierarchically structured across timescales, with bursts formed from smaller bursts and characterized by heavy-tailed burst-size and merging-number distributions. The dynamic model maintains level-specific memory variables $$\mathcal{K}_t = \{ (s, r, o, \tau) \mid s, o \in \mathcal{E}, r \in \mathcal{R}, \tau \in \mathcal{T} \}$$5 and transition probabilities

$$\mathcal{K}_t = \{ (s, r, o, \tau) \mid s, o \in \mathcal{E}, r \in \mathcal{R}, \tau \in \mathcal{T} \}$$6

treating hierarchical burst organization as a multiscale memory mechanism embedded in the dynamics (Hiraoka et al., 14 Aug 2025). Although this work is not an agent-memory architecture, it provides a complementary notion of hierarchical event memory as nested temporal dependence.

6. Empirical performance, benefits, and limitations

Across tasks, hierarchical event memory is associated with gains in long-range reasoning, temporal coherence, retrieval precision, and bounded-context efficiency. The gains are usually attributed not to memory size alone, but to semantically structured segmentation and selective expansion.

EventMemAgent reports a direct ablation against fixed-length segmentation memory split every 30 seconds. On OVO-Bench, hierarchical memory scores 60.75 versus 60.16 for fixed-length memory, a drop of 0.59; on StreamingBench, hierarchical memory scores 77.00 versus 76.80, a drop of 0.20 (Wen et al., 17 Feb 2026). With at most 32 input frames, EventMemAgent achieves 60.75% overall on OVO-Bench (Wen et al., 17 Feb 2026). HiGMem reports that on LoCoMo10 it retrieves an average of 8.09 turns versus 99.84 for A-Mem while maintaining recall 0.7241 versus 0.7502 and improving precision 0.1909 versus 0.0101 (Cao et al., 20 Apr 2026). In the same paper, the “w/o Hierarchy” ablation drops to F1 = 0.39 and Recall@K = 0.55, compared with full HiGMem at F1 = 0.49 and Recall@K = 0.72 (Cao et al., 20 Apr 2026).

SEEM reports on LoCoMo 56.1 BLEU-1, 61.1 F1, and 78.0 $$\mathcal{K}_t = \{ (s, r, o, \tau) \mid s, o \in \mathcal{E}, r \in \mathcal{R}, \tau \in \mathcal{T} \}$$7, and on LongMemEval 65.0 accuracy, outperforming HippoRAG 2 by 2.8 F1 and 1.5 $$\mathcal{K}_t = \{ (s, r, o, \tau) \mid s, o \in \mathcal{E}, r \in \mathcal{R}, \tau \in \mathcal{T} \}$$8 on LoCoMo and by 4.4 absolute accuracy on LongMemEval (Lu et al., 10 Jan 2026). StructMem reports 76.82 overall on LoCoMo, with 81.62 on Temporal reasoning and 68.77 on Multi-hop, while using 1.937M build tokens, 1056 API calls, and 22,854 s runtime, compared with substantially larger costs for several graph-based baselines (Xu et al., 23 Apr 2026). HiMem reports 80.71 overall GPT-Score on LoCoMo, compared with 51.88 for A-MEM, 69.03 for SeCom, and 68.74 for Mem0, with especially strong Multi-Hop at 70.92 and Temporal at 74.77 (Zhang et al., 10 Jan 2026). ES-Mem reports overall LoCoMo performance of 45.56 F1 / 39.70 BLEU-1 with GPT-4o-mini, and 72.40 overall on LongMemEval-S, while using 2925 tokens and 1.423 s latency on LoCoMo (Zou et al., 12 Jan 2026).

Video hierarchies show similar patterns. PyraVid reports 69.1 on Video-MME (Long) and 58.5 on LVBench for PyraVid(32B), with ablations showing that “Hierarchical Memory w/o prune” falls to 63.5 on Video-MME and has mean latency 21.99 on LVBench, compared with full PyraVid latency 7.26 (Yan et al., 16 May 2026). OASIS improves Qwen3-VL-8B on OVO-Bench from 66.79 to 78.14 in Perception Avg and from 51.19 to 57.21 in Backward Avg, while keeping token cost around 10k tokens across datasets versus 29,517 tokens for a full-context baseline on StreamingBench and reducing peak GPU memory from 76.59 GB to 28.48 GB on OVO-Bench (Liang et al., 18 Apr 2026). HEM for online video temporal grounding reports on TACoS $$\mathcal{K}_t = \{ (s, r, o, \tau) \mid s, o \in \mathcal{E}, r \in \mathcal{R}, \tau \in \mathcal{T} \}$$9 without future prediction and 37.44 with future prediction, compared with a baseline at 29.74, and on ActivityNet Captions 45.29 without future prediction and 42.89 with future prediction versus 25.48 baseline (Zheng et al., 6 Aug 2025).

The limitations are also recurrent. EventMemAgent explicitly implies that grayscale-histogram boundary detection is heuristic, that memory retrieval is capped to top-3 events with similarity $\alpha$0, that compression loses detail, and that the sparse binary reward in Agentic RL may make learning difficult (Wen et al., 17 Feb 2026). HiMem notes the absence of explicit conflict resolution beyond reconsolidation triggers and identifies memory decay, revision, and automated prompt optimization as future work (Zhang et al., 10 Jan 2026). StructMem notes dependence on prompt quality and the lack of explicit conflict resolution or memory updating (Xu et al., 23 Apr 2026). The theory paper emphasizes that most current analysis assumes static hierarchies, whereas real systems insert, split, merge, reinforce, and decay memory over time (Talebirad et al., 23 Mar 2026).

A plausible synthesis is that hierarchical event memory consistently improves reasoning when the task depends on temporal continuity, episodic localization, or stage-sensitive context selection, but its effectiveness depends strongly on three fragile components: the fidelity of event segmentation, the adequacy of summary-level representations, and the correctness of the traversal policy that decides when to expand.

7. Relation to cognitive theory and open research directions

Several systems explicitly ground their designs in cognitive theory. PyraVid is inspired by Event Segmentation Theory, which proposes that humans parse continuous experience into meaningful events across multiple temporal scales (Yan et al., 16 May 2026). ES-Mem also draws directly on Event Segmentation Theory and treats event boundaries as cognitive anchors for later episodic recall (Zou et al., 12 Jan 2026). HiMem describes its episode and note layers as cognitively consistent with the distinction between grounded events and stable knowledge (Zhang et al., 10 Jan 2026). SEEM cites frame semantics and cognitive frame theory in defining Episodic Event Frames (Lu et al., 10 Jan 2026).

Theoretical work suggests a unifying research agenda. The operator view of extraction, coarsening, and traversal frames hierarchical event memory as a general control architecture for bounded reasoning rather than a domain-specific trick (Talebirad et al., 23 Mar 2026). This perspective highlights unresolved questions about representative self-sufficiency, grouping quality, and dynamic hierarchy maintenance. The same paper identifies adaptive, dynamic hierarchy construction as the most important open problem (Talebirad et al., 23 Mar 2026).

Several practical directions recur across the literature. One is dynamic updating and reconsolidation: HiMem introduces conflict-aware Memory Reconsolidation with ADD, UPDATE, and DELETE operations, but more explicit lifelong revision remains open (Zhang et al., 10 Jan 2026). Another is richer multimodal grounding: PyraVid extends event memory with key frames, ASR, face IDs, voice IDs, and person profiles, indicating a route toward identity-consistent memory in long videos (Yan et al., 16 May 2026). A third is agentic retrieval control: EventMemAgent uses ReAct-style trajectories and Group Relative Policy Optimization with group size $\alpha$1, KL coefficient = 0, optimizer = AdamW, learning rate $\alpha$2, and one epoch to internalize when to search memory or use OCR and detection (Wen et al., 17 Feb 2026). OASIS instead offers a training-free form of controlled refinement in which retrieval is triggered only when coarse reasoning emits <tool_call> (Liang et al., 18 Apr 2026).

A persistent controversy concerns whether hierarchical memory should prioritize summary quality or retrieval routing. The formal literature argues that these are distinct functions governed by the self-sufficiency of representatives (Talebirad et al., 23 Mar 2026). Empirical systems illustrate both ends: HiGMem’s event summaries function as semantic anchors for selecting turns (Cao et al., 20 Apr 2026), whereas SEEM’s Episodic Event Frames can serve as substantive narrative scaffolds during evidence reconstruction (Lu et al., 10 Jan 2026). This suggests that “hierarchical event memory” is not a single point on the design space but a family of architectures whose success depends on matching representative type to traversal strategy.

In current usage, hierarchical event memory therefore designates a structured-memory paradigm in which events are the primary organizational unit, abstraction is layered rather than flat, and retrieval is guided by event-level structure before detailed evidence is exposed. Across video streams, long dialogues, web trajectories, and multimodal reasoning, the central claim is consistent: memory is more useful when stored in the shape of experience rather than as an undifferentiated collection of retrievable fragments (Wen et al., 17 Feb 2026, Lu et al., 10 Jan 2026, Zou et al., 12 Jan 2026, Tan et al., 7 Mar 2026).