MEMENTO: Memory for Personalized Assistance

• The paper introduces the MEMENTO framework to evaluate how LLM-powered agents leverage episodic memory for personalized object rearrangement tasks.
• It employs a two-stage evaluation that separates memory acquisition from utilization to measure performance differences in context-based instructions.
• Results show significant drops in success rates when agents recall episodic details, highlighting the need for improved memory retrieval strategies.

MEMENTO: Memory Utilization for Personalized Embodied Assistance

MEMENTO is a framework designed to systematically evaluate and quantify the capacity of LLM-powered embodied agents to leverage episodic memory for personalized assistance in object rearrangement tasks. Unlike classical benchmarks that emphasize “one-shot” instruction following, MEMENTO probes an agent’s ability to recall and utilize user-specific interaction history to interpret ambiguous, dynamic commands reminiscent of real-world domestic environments (Kwon et al., 22 May 2025).

1. Motivation and Objectives

Traditional embodied AI benchmarks, such as object rearrangement in simulated households, focus on single-turn, fully specified instructions that require no inference about user-specific semantics. In practical settings, users issue instructions that depend on prior personalized context, e.g., “Bring me my coffee mug” (requiring mapping to the user’s preferred cup), or “Set up my remote-work station” (requiring identification of object–location configurations based on the user’s habitual routines). MEMENTO’s principal goal is to quantify how effectively LLM-driven agents access and utilize episodic memory to perform such personalized reasoning, moving beyond mere instruction following to genuine context-aware assistance.

The framework targets two core types of personalized inference:

1. Object semantics: Mapping underspecified user references (“my mug,” “our charger”) to specific objects, based on history (ownership, user preferences, usage).
2. User patterns: Reconstructing multi-object arrangements (e.g., “my breakfast setup”) by inferring schema from repeated routines.

2. Two-Stage Memory Evaluation Protocol

To dissect memory utilization independently from policy planning or control, MEMENTO divides each scenario into two linked evaluation stages, both set in the same synthetic household scene $S$ and with the same ground-truth goal $g$.

Stage 1: Memory Acquisition

• The agent receives a fully specified instruction $I_{\textrm{acq}}$, containing explicit personalized knowledge (e.g., “Bring the cup labeled 'Dad’s Coffee' from the kitchen table to the living room. This is my favorite mug.”).
• The agent executes the rearrangement; the interaction trajectory $h_{\textrm{acq}} = (I_{\textrm{acq}}, \tau_k)$ (with $\tau_k = (w_1, a_1, \dots, w_k)$) is recorded as episodic memory.
• No memory retrieval is required in this stage; $\phi(I_{\textrm{acq}}) \rightarrow g$ by construction.

Stage 2: Memory Utilization

• The same scene and goal $g$ are reused, but the agent receives an underspecified instruction $I_{\textrm{util}}$, such as “Could you fetch my morning cuppa?”.
• Success on this stage requires recalling and grounding the appropriate segment(s) of $h_{\textrm{acq}}$.
• Task variants:
  • Single-memory: Only one past episode needs to be recalled.
  • Joint-memory: The instruction references two (or more) separate past episodes that must be retrieved and integrated.

This design isolates the “memory effect” by holding environment and goals fixed, varying only the demand for memory retrieval and reasoning.

3. Formalization and Metrics

Let $\epsilon = (S, I, g)$ denote a base episode. The split stages are:

$g$0

The LLM-led planner $g$1 receives per-timestep tokens $g$2 and returns the next action. Key evaluation metrics:

• Percent Complete (PC): Fraction of object–location subtasks satisfied per goal.
• Success Rate (SR): Fraction of episodes with $g$3.
• Planning cycles: LLM calls required.
• Simulator steps: Timesteps until termination.

To quantify memory impact:

$g$4

These deltas directly attribute any performance drop to episodic memory failure or difficulty.

4. Task Design and Instruction Generation

Tasks originate from the PartNR benchmark, inherently requiring multiple object–location rearrangements per instruction. To enforce personalization reliance, distractor objects of the same type are introduced, so generic policies cannot “brute-force” the task.

Instructions are constructed in pairs:

• Memory Acquisition ($g$5): Explicitly names both the object(s) and the history, e.g., “That mug is my favorite coffee mug.”
• Memory Utilization ($g$6): GPT-4o automatically reformulates each $g$7 to be underspecified, using naturalistic references (e.g., “Bring me my usual morning mug.”), which require exploitation of episodic memory.

For joint-memory tasks, two such underspecified requests are concatenated, necessitating multi-episode retrieval and integration.

5. Experimental Setup and Memory Retrieval

Benchmarked models include GPT-4o, Claude-3.5-Sonnet, Qwen-2.5 (72b/7b), and Llama-3.1 (70b/8b). For episodic memory, top-$g$8 nearest neighbor retrieval in embedding space is employed, always ensuring the correct $g$9 is available in the corpus.

Scenario statistics:

• 201 single-memory episodes (split: 89 object semantics, 112 user patterns)
• 36 joint-memory episodes (requiring multi-memory reasoning)

6. Key Findings: Quantitative and Qualitative

Significant limitations in memory utilization are observed:

• Even GPT-4o drops from 95.0% SR (acquisition) to:
  • 85.1% (–9.9 pts) on single-memory tasks
  • 63.9% (–30.5 pts) on joint-memory tasks
• Knowledge-type breakdown:
  • Object semantics tasks: 8–10 pts SR drop
  • User patterns: 20–30 pts SR drop
• Sensitivity to distractor memories: Increasing $I_{\textrm{acq}}$0—and thus retrieval noise—disproportionately hurts performance in user pattern tasks.
• Case studies:
  • Agents misinterpret personalized user references (e.g., treating “my mug” as generic)
  • Overuse of commonsense, underuse of episodic facts (e.g., “movie-night setup” assembles popcorn and remote generically)
  • Hallucination of prior details never observed in memory (false personalization)

This establishes a robust gap between instruction-following and true personalized reasoning.

7. Implications and Future Research Trajectories

MEMENTO demonstrates that state-of-the-art LLM-powered embodied agents—despite exhibiting high performance in one-shot, fully observable settings—showmarked performance degradation when challenged to recall and combine episodic memories for personalized tasks. The findings surface a substantial research agenda:

• Richer memory indexing: Temporal graph-based memory structures may provide hierarchical, time-sensitive access.
• Learned retrieval: Fine-tuned retrieval heads or supervised indexers to better filter and prioritize memories.
• Multi-modal memory: Integration of both visual and linguistic episode representations for richer personalization.
• Privacy and robustness: Dynamic, privacy-preserving memory management protocols suitable for real-world deployment.
• Ambiguity management: Protocols for resolving ambiguous or indirect user language.

Overall, MEMENTO (P-Habitat) delivers a standardized, rigorous methodology for evaluating memory-centric personalization in embodied agents, charting a path toward assistants capable of truly user-specific, context-aware behaviors (Kwon et al., 22 May 2025).