StreamMemBench: Streaming Evaluation of Agent Memory for Future-Oriented Assistance

Abstract: A central role of personal-agent memory is to turn stored information and prior interactions into future-oriented assistance. In daily use, useful cues come from what the agent observes and how the user interacts with the agent, and the agent must carry them forward from the current request to similar future tasks. Existing memory benchmarks usually test dialogue recall or task improvement in isolation, leaving the trajectory from streaming observations to later assistance largely untested. We introduce StreamMemBench, a streaming benchmark that constructs a two-step task sequence around each evidence anchor from EgoLife egocentric streams. The initial task tests evidence use, while the follow-up task tests whether feedback and interaction experience are reused. Four metrics diagnose evidence recall, initial evidence use, feedback incorporation, and follow-up reuse. Experiments with eight memory systems across two backbones show that current systems often fail to use observed evidence or turn feedback into reliable follow-up behavior, even when evidence is stored or feedback is incorporated locally. StreamMemBench is publicly available at https://github.com/landian60/StreamMemBench.

• The paper introduces a streaming evaluation paradigm that measures both immediate evidence use (IEU) and follow-up reuse (FUR) of agent memory.
• It uses the EgoLife dataset to transform lifelog segments into evidence anchors, enabling traceable memory extraction through multi-stage tasks.
• Experimental results reveal a significant gap between memory fidelity and effective evidence utilization, highlighting limitations in current agent memory systems.

StreamMemBench: Streaming Evaluation of Agent Memory for Future-Oriented Assistance

Motivation and Problem Formulation

Personal agents leveraging LLMs require robust memory mechanisms to provide context-sensitive, future-oriented assistance. Traditional memory benchmarks disproportionately focus on recall or one-off performance improvements without capturing the end-to-end trajectory spanning streaming observation, context extraction, user interaction, and subsequent assistance. The disconnect between memory storage and its operationalization in follow-up behavior motivates the need for benchmarks that target not only memory fidelity but also the agent’s ability to apply and consolidate knowledge across temporally extended scenarios.

StreamMemBench addresses these gaps by formalizing a streaming evaluation paradigm that integrates egocentric observations and interaction feedback into the evaluation of capability transfer across chronologically ordered personal-agent tasks. The framework probes both the immediate utilization of observed evidence (Initial Evidence Use, IEU) and the agent’s ability to reuse consolidated experience and feedback for future-oriented assistance (Follow-up Reuse, FUR).

Figure 1: A streaming view of personal-agent memory, illustrating memory extraction from an egocentric stream to support initial and subsequent tasks.

Benchmark Construction and Evaluation Protocol

StreamMemBench is built upon the EgoLife dataset, which comprises continuous egocentric lifelog segments with rich multimodal observations. The benchmark construction process involves transforming lifelog segments into evidence anchors—granular, traceable memory units tied to specific, verifiable observations. Each anchor substantiates two sequential tasks: an initial request grounded in the anchor and a distinct follow-up request leveraging the same evidence—compelling agents to demonstrate both contextualization and eventual experience consolidation.

Figure 2: Overview of the StreamMemBench pipeline from lifelog to evidence anchor, task query generation, feedback collection, and downstream reuse evaluation.

The evaluation trajectory for each anchor involves the following stages:

• Fidelity: Verifies whether the agent has successfully encoded the evidence anchor into memory.
• IEU: Assesses if the agent’s response to the initial task is grounded in the relevant evidence.
• Feedback Incorporation (FI): Measures the agent's ability to integrate corrective feedback into the immediate post-feedback response.
• FUR: Evaluates whether the prior evidence and feedback inform the agent’s response to the follow-up task without explicit prompting.

Critically, the protocol does not permit explicit disclosure of evidence in queries and utilizes simulated user feedback both to diagnose gaps in memory utilization and to disambiguate failures arising from evidence retention, feedback assimilation, or reuse.

Comparative Analysis to Existing Benchmarks

Unlike prior works such as LoCoMo [maharana2024locomo], PersonaMem [jiang2025personamem], LongMemEval [wu2025longmemeval], and MemoryAgentBench [hu2026memoryagentbench], StreamMemBench uniquely:

• Anchors all tasks in real egocentric, time-ordered observation streams rather than synthesized or dialogue-generated data.
• Diagnoses memory at several lifecycle stages, not just aggregate task improvement.
• Emphasizes traceability from stream observation to downstream task usage, revealing bottlenecks in application and consolidation rather than mere storage.

Experimental Results

Evaluation was conducted on eight memory architectures spanning retrieval-based (RAG_raw, RAG_ext), fact-based (Mem0), structured (EverMemOS), event-linking (A-Mem), multi-level consolidation (MemOS, MemoryOS), and skill-based (MemSkill) systems. Each system was benchmarked on DeepSeek-V4-Flash and Gemini-3-Flash models.

Quantitatively, the pass rates for pivotal metrics demonstrated a pronounced gap between evidence retention (Fidelity) and successful evidence use (IEU and FUR):

• Systems often exhibited perfect Fidelity but considerably lower IEU and FUR, indicating that successfully storing memory does not guarantee its operational use in future assistance.
• For example, MemoryOS achieved 100% Fidelity, but only 23.93% IEU and 61.95% FUR on DeepSeek-V4-Flash.
• A-Mem and MemoryOS outperformed others on FUR, while MemOS lagged dramatically, revealing method-specific limitations in experience consolidation and retrieval.
• FI scores generally exceeded FUR, exposing that many agents can leverage feedback in-the-moment but fail to consolidate or recall this correction for future tasks.

  Figure 3: Capability curves for DeepSeek-V4-Flash across stream positions, demonstrating declining IEU (task use) over time despite stable evidence availability, highlighting decoupling between storage and use.

Further analysis indicated:

• Temporal drift: IEU degrades as the chronological distance from evidence acquisition increases, while Fidelity remains stationary, especially in retrieval-based systems.
• Failure mode taxonomy: Failure points included initial evidence neglect (IEU failure), feedback incorporation lapses (FI failure), non-consolidation of corrections (correction consolidation failure), and failure to persist evidence use (evidence-use persistence failure). Notably, many failures do not originate from storage/formation issues but from inability to surface or re-contextualize evidence at inference.

  Figure 4: Failure mode distribution showing where in the lifecycle evidence is lost, not applied, or not reused. Memory designs differ in the location and nature of their breakdowns.

Dataset Scope and Task Diversity

StreamMemBench encompasses 8,107 evidence anchors and 16,214 task queries across diverse assistance scenarios, with queries equally distributed between state-oriented (applying known user information) and inference-oriented (reasoning-based) tasks. Approximately half of the queries pertain to user self-attributes, with the remainder spanning social contexts.

Figure 5: Representative evidence/query pairs clarify the spectrum of contexts and reasoning burdens evaluated in the benchmark.

Figure 6: Breakdown of task scenarios, evidencing broad coverage of real-world personal assistance types.

Implications and Future Directions

Empirical evidence from StreamMemBench decisively shows that current memory system designs for LLM agents are inadequate for end-to-end future-oriented assistance over streaming data. Even advanced multi-level memory architectures fail to secure high rates of evidence-governed behavior transfer. The pronounced drop between retention and reuse posits that future memory systems must integrate adaptive consolidation mechanisms, selective retrieval policies, and more sophisticated methods for temporal abstraction.

Practically, the implications extend to any setting where personal agents must draw longitudinally from user experience—household robotics, assistive technologies, and lifelong personal productivity tools. Theoretically, these findings argue for a unification of episodic and semantic memory modeling, integrated user feedback assimilation, and lifecycle-oriented evaluation.

Further research could include:

• Expanding the benchmark to cover longer time horizons and more complex scenario compositions.
• Cross-evaluating with commercial/research memory agent variants not covered in the initial benchmark.
• Investigation of privacy-preserving deployment and user-modulated memory controls, given the sensitive nature of lifelog data.

Conclusion

StreamMemBench establishes a rigorous, streaming-first standard for evaluating agent memory in the context of user-assistance tasks. The evidence demonstrates that high memory fidelity is insufficient: diagnostic analysis must extend across retention, context-aware use, feedback incorporation, and cross-turn reuse. Existing systems reveal substantial limitations on these axes, highlighting ample opportunity for advancements in memory system architecture, consolidation, and evaluation methodology for agentic LLMs (2606.14571).