Memory-Augmented AR Agents

• Memory-Augmented AR Agents are AR assistants that integrate real-time perception with persistent memory to personalize interactions and maintain long-term dialogue.
• These agents utilize varied memory substrates—including episodic, graph-based, and compressed representations—to support selective retrieval and evolving contextual understanding.
• Evaluations reveal improved recall and contextual accuracy while highlighting challenges in balancing memory quantity, latency, and social appropriateness in AR applications.

Searching arXiv for the cited works to ground the article in the latest paper records. Memory-augmented AR agents are augmented-reality assistants that couple real-time perception and interaction with persistent external memory, so that future behavior depends not only on the current observation or utterance but also on stored semantic, episodic, procedural, or compressed representations of prior interactions. In the cited literature, direct AR instantiations include an AR glasses–based “AR Secretary Agent” that identifies who the wearer is conversing with and summarizes previous discussions, and “Livia,” an emotion-aware AR companion with progressive memory compression (Haddad et al., 17 May 2025, Xi et al., 12 Aug 2025). Closely related work on lifelong dialogue, personalized assistants, embodied reinforcement learning, and long-horizon memory architectures supplies the underlying design vocabulary: timeline-linked memory, editable memory graphs, multi-graph memory, continuous memory, active reconstruction, and critic- or reward-driven memory evolution (Ong et al., 2024, Wang et al., 2024, Jiang et al., 6 Jan 2026, Ji et al., 4 Jun 2026).

1. Problem formulation and agent setting

The common problem is long-term, multi-session interaction under bounded context, partial observability, and evolving user or environment state. In lifelong dialogue, the failure modes are explicit: systems may “over-attend to the most recent user utterance,” “fail to recall relevant earlier events,” “retrieve only shallowly similar memories,” or lose critical information when old memories are overwritten or compressed (Ong et al., 2024). In embodied and AR-adjacent settings, the same issue appears as a sequential decision problem in which the agent must remain contextualized over egocentric streams, interrupted tasks, and persistent world state rather than repeatedly reconstructing state from raw traces (Gupta et al., 22 Oct 2025, Zhang et al., 13 Feb 2026).

A useful formal lens is the multi-session Memory-Agent-Environment loop. MemoryArena models a task as a sequence of subtasks $S=\{s_i\}_{i=1}^n$, with retrieval and update as first-class operations:

$$m_{i,t} = \mathrm{RETRIEVE}(M, s_i, a_{i,1:t-1}, o_{i,1:t-1})$$

$$a_{i,t} \sim \pi_A(\cdot \mid s_i, o_{i,1:t-1}, a_{i,1:t-1}, m_{i,t})$$

$$M \leftarrow \mathrm{UPDATE}(M, (o_{i,1:T_i}, a_{i,1:T_i}))$$

This framing is directly compatible with AR assistance, where later action is intentionally underspecified unless the system preserved the right information from earlier sessions (He et al., 18 Feb 2026).

Direct AR systems instantiate this problem concretely. AR Secretary Agent uses AR glasses, visual and auditory capture, face recognition, speech transcription, database-backed memory, and GPT-4 summarization to provide real-time information about conversation partners and previous discussions; in a user study with 13 participants, it reported “up to 20\% memory enhancement on our study” (Haddad et al., 17 May 2025). Livia addresses longitudinal companionship by combining multimodal affective computing, long-term memory with progressive compression, and AR-driven embodied interaction (Xi et al., 12 Aug 2025). Taken together, these systems define the AR-specific version of the memory problem: persistent personalization, task continuity, and real-time response under storage, latency, and social appropriateness constraints.

2. Memory substrates and representation schemes

The literature does not converge on a single memory substrate. Instead, it presents a spectrum from verbatim episodic storage to graph-structured, compressed, and continuously learned memories.

One end of the spectrum stores raw conversational experience. “Knowledge Access Beats Model Size” stores a memory item as a raw conversational turn-pair with a session timestamp prefix, explicitly favoring verbatim storage because generated summaries or synthetic memory entries can introduce hallucinated content and “RAG poisoning” (Liu et al., 24 Mar 2026). This is episodic memory in high-fidelity form. AR Secretary Agent is close in spirit: it uses person-centered memory tied to future encounters, though its contribution is system-level rather than a formal memory algorithm (Haddad et al., 17 May 2025).

A second family organizes memory structurally rather than as a flat store. THEANINE “discards memory removal and manages large-scale memories by linking them based on their temporal and cause-effect relation,” then augments response generation with “memory timelines - series of memories representing the evolution or causality of relevant past events” (Ong et al., 2024). EMG-RAG treats smartphone-derived memory as an Editable Memory Graph and combines it with Retrieval-Augmented Generation, with reinforcement learning used to optimize selectability (Wang et al., 2024). MAGMA generalizes this idea into a time-varying directed multigraph with orthogonal semantic, temporal, causal, and entity graphs, arguing that monolithic vector stores entangle relations that should remain distinct (Jiang et al., 6 Jan 2026).

A third family makes memory explicitly typed and heterogeneous. AdMem decomposes long-term memory as

$M = M_S \oplus M_E \oplus M_P$

for semantic, episodic, and procedural memory, within a bi-level short-term/long-term architecture (Wang et al., 5 Jun 2026). Memory-R2 uses flat memory entries with fields such as memory_id, speaker, content, session_time, and dia_ids, while constraining each stored fact to be atomic, self-contained, and typically no more than 20 words in the content field (Yan et al., 20 May 2026). This suggests a recurring design principle: memory utility often increases when descriptive facts, event traces, and action guidance are represented separately rather than conflated.

A fourth family emphasizes compression. Memo partitions long histories into segments of length $l_{seg}$ and generates $l_{sum}$ summary tokens after each segment; future segments attend to accumulated summaries rather than raw observations from earlier segments (Gupta et al., 22 Oct 2025). Stable Hadamard Memory maintains a matrix memory with adaptive calibration,

$M_t = M_{t-1}\odot C_t + U_t,$

where calibration suppresses stale traces and reinforces useful ones (Le et al., 2024). Livia compresses long-term interaction data in SQLite with Temporal Binary Compression and Dynamic Importance Memory Filter, storing timestamps, utterance types, emotional contexts, and importance ratings (Xi et al., 12 Aug 2025).

Finally, some recent work argues that visually grounded agents should not collapse memory into text. “Auto-scaling Continuous Memory for GUI Agent” compresses each trajectory into “8 embeddings” and injects them directly into the backbone VLM, motivated by the observation that text summaries lose decisive visual cues such as exact widget size, position, layout, and appearance (Wu et al., 10 Oct 2025). This does not directly study AR, but it suggests that memory-augmented AR agents may also need latent multimodal memory rather than text-only recap when downstream policy depends on geometry and appearance.

3. Retrieval, reconstruction, and routing

Memory retrieval is no longer treated as a single top-$k$ similarity lookup. The cited work separates several distinct retrieval problems: selecting which memory store to query, selecting which items to surface, and deciding whether memory should be used at all.

The strongest critique targets the static retrieve-then-reason pipeline. MRAgent formalizes passive retrieval as

$\{v^{(1)}, \ldots, v^{(T)}\} = \pi_{\mathrm{p}}(x)$

and active reconstruction as

$$m_{i,t} = \mathrm{RETRIEVE}(M, s_i, a_{i,1:t-1}, o_{i,1:t-1})$$0

Its Cue-Tag-Content graph uses associative tags as semantic bridges, allowing the agent to iteratively explore and prune retrieval paths based on accumulated evidence rather than fixing retrieval from the initial query alone (Ji et al., 4 Jun 2026). THEANINE reaches a related conclusion from a dialogue perspective: relevant memory is often not a single snippet but a timeline of temporally or causally linked memories (Ong et al., 2024).

Routing appears at two levels. “Did You Check the Right Pocket?” treats retrieval as a store-routing problem over specialized stores such as Short-Term Memory, Summary, Long-Term Memory, and Episodic memory, and formalizes the decision as

$$m_{i,t} = \mathrm{RETRIEVE}(M, s_i, a_{i,1:t-1}, o_{i,1:t-1})$$1

Its empirical finding is that selective retrieval can improve both accuracy and token efficiency relative to uniform retrieval (Gaikwad, 8 Mar 2026). “Knowledge Access Beats Model Size” studies routing between a small and large model. A user query goes to a memory store, retrieved memories are injected into the small model’s prompt, confidence is computed from token log-probabilities, and the answer is either accepted or escalated. In that setting, memory improves correctness rather than routing behavior: routing alone already sends most traffic to the small model, but without memory the system remains confidently wrong (Liu et al., 24 Mar 2026).

Retrieval quality also depends on the scoring substrate. The same paper shows that hybrid retrieval combining BM25 and cosine similarity improves downstream QA, with LongMemEval F1 rising from 36.5 to 44.2 and BLEU-1 from 34.9 to 42.0 through the full 8B routing pipeline (Liu et al., 24 Mar 2026). EMG-RAG likewise frames selectability as a learned problem over structured memory (Wang et al., 2024).

A separate issue is whether memory should be applied at all. OP-Bench shows that memory-augmented personalized agents often retrieve and over-attend to user memories even when unnecessary, creating irrelevance, repetition, and sycophancy (Hu et al., 20 Jan 2026). Self-ReCheck addresses this by filtering candidate memories:

$$m_{i,t} = \mathrm{RETRIEVE}(M, s_i, a_{i,1:t-1}, o_{i,1:t-1})$$2

where the filter may return “NO RELEVANT CONTEXT” (Hu et al., 20 Jan 2026). For AR agents, this is not a peripheral issue. A plausible implication is that socially visible AR interventions require an even stricter abstention threshold than text chat, because semantically similar memory can still be pragmatically inappropriate in the current embodied context.

4. Memory formation, update, and learning

The update problem is as important as representation and retrieval. Several papers argue that long-context prompting and standard RAG are fundamentally passive because they defer state tracking, contradiction resolution, and evidence aggregation until query time (Zhang et al., 13 Feb 2026). In response, recent systems move memory management into the agent’s learning loop.

UMA maintains a “dual memory representation: a compact core summary for global context and a structured Memory Bank that supports explicit CRUD (create, update, delete, reorganize) over key value entries,” and learns memory operations jointly with question answering through end-to-end reinforcement learning (Zhang et al., 13 Feb 2026). Memo learns periodic summarization tokens inside a transformer policy; at timestep $$m_{i,t} = \mathrm{RETRIEVE}(M, s_i, a_{i,1:t-1}, o_{i,1:t-1})$$3, the model conditions on accumulated summary memory $$m_{i,t} = \mathrm{RETRIEVE}(M, s_i, a_{i,1:t-1}, o_{i,1:t-1})$$4 and the current unsummarized segment, rather than the entire raw prefix (Gupta et al., 22 Oct 2025). Stable Hadamard Memory proposes bounded, context-dependent memory calibration,

$$m_{i,t} = \mathrm{RETRIEVE}(M, s_i, a_{i,1:t-1}, o_{i,1:t-1})$$5

so each memory cell can be attenuated toward $$m_{i,t} = \mathrm{RETRIEVE}(M, s_i, a_{i,1:t-1}, o_{i,1:t-1})$$6 or amplified up to $$m_{i,t} = \mathrm{RETRIEVE}(M, s_i, a_{i,1:t-1}, o_{i,1:t-1})$$7, with the goal of enhancing memory capacity while mitigating numerical and learning challenges in long-horizon RL (Le et al., 2024).

Other work emphasizes delayed credit assignment. Memory-R2 argues that once different rollouts write, update, or delete different memories, they no longer share the same intermediate memory state, making ordinary group-relative comparisons unfair. Its LoGo-GRPO algorithm combines global trajectory-level optimization with local rerollouts from the same cached intermediate memory state, so that memory operations are compared under a common pre-session memory bank (Yan et al., 20 May 2026). AdMem addresses a related issue with an actor-memory-critic architecture. The actor states an expected outcome $$m_{i,t} = \mathrm{RETRIEVE}(M, s_i, a_{i,1:t-1}, o_{i,1:t-1})$$8 before acting; the critic later compares expected and actual outcome, emits a reward and reflection, and writes procedural memory that stores decision-making guidance rather than only successful traces (Wang et al., 5 Jun 2026).

A lighter-weight alternative appears in MARS. Reflection is stored in long-term memory as

$$m_{i,t} = \mathrm{RETRIEVE}(M, s_i, a_{i,1:t-1}, o_{i,1:t-1})$$9

and memory triage follows an Ebbinghaus-style forgetting rule

$$a_{i,t} \sim \pi_A(\cdot \mid s_i, o_{i,1:t-1}, a_{i,1:t-1}, m_{i,t})$$0

Thresholds determine whether optimized information remains in short-term memory, transfers to long-term memory, or is discarded (Liang et al., 25 Mar 2025). Livia reaches a similar operational goal through Temporal Binary Compression and Dynamic Importance Memory Filter: preserve what matters, summarize what ages, and drop what is both old and low-importance (Xi et al., 12 Aug 2025).

Across these systems, the common conclusion is that persistent memory should be edited, consolidated, and evaluated over time rather than treated as append-only context. This suggests that AR agents need memory formation and memory evolution as separate but coupled processes: one extracts durable state from multimodal experience, and the other decides what to preserve, revise, compress, or forget.

5. AR architectures and instantiated systems

Two papers supply direct AR implementations rather than AR analogies.

System	Memory design	Reported findings
AR Secretary Agent	AR glasses, speech transcription, face recognition, database-backed memory, GPT-4 summarization	13 participants; “up to 20\% memory enhancement”
Livia	SQLite long-term memory, Temporal Binary Compression, Dynamic Importance Memory Filter, modular agents, AR avatar	88% emotion accuracy; 31% higher engagement; 50KB to 15KB average per-user storage

AR Secretary Agent is a prototype memory-augmented augmented-reality assistant that “could discreetly provide real-time information to the wearer, identifying who they are conversing with and summarizing previous discussions” (Haddad et al., 17 May 2025). Its contribution is less a new formal memory algorithm than an end-to-end wearable pipeline tying AR glasses, speech transcription, face recognition, database-backed memory, and summarization into a real-time memory prosthesis. In the language of the broader literature, it is a person-centered episodic memory system surfaced opportunistically during future encounters.

Livia is architecturally richer. It uses four specialized agents: an Emotion Analyzer Agent, a Frontend Voice Interaction Agent, a Memory Compression Agent, and a Behavior Orchestration Agent, coupled to a Unity3D + Apple ARKit frontend (Xi et al., 12 Aug 2025). Text emotion is processed by a RoBERTa-based classifier, voice by a CNN-LSTM-based model that maps audio to valence and arousal, and dialogue generation is handled by a GPT-4-based LLM whose prompt includes recent turns, retrieved memory snippets, current emotional state, and a user-selected personality archetype (Xi et al., 12 Aug 2025). The memory subsystem stores long-term interaction data in SQLite and compresses it progressively through Temporal Binary Compression and Dynamic Importance Memory Filter, so that “important events” remain detailed more often than low-salience material. Over the evaluation period, average per-user memory storage fell from 50KB to 15KB; important events were recalled 92% of the time, whereas less significant details were recalled 65% of the time (Xi et al., 12 Aug 2025).

These AR systems expose two distinct application profiles. AR Secretary Agent is a real-time recall aid centered on identity continuity and previous conversations (Haddad et al., 17 May 2025). Livia is a longitudinal companion in which memory interacts with affect modeling, proactive behavior, and AR embodiment (Xi et al., 12 Aug 2025). A plausible implication is that memory-augmented AR agents are better understood as a family of systems rather than a single architecture class: social recall assistants, emotionally supportive companions, embodied task guides, and personalized spatial assistants can all be memory-augmented, but they emphasize different memory types and update policies.

Adjacent work broadens this picture. EMG-RAG transfers personalized agents into a real smartphone AI assistant, using user memories collected from conversations, screenshots, emails, calendars, location histories, and related artifacts (Wang et al., 2024). Continuous memory for GUI agents shows that trajectory memory can be compressed into 8 continuous vectors and prepended to a backbone VLM, with positive scaling as memory size and retrieval depth increase (Wu et al., 10 Oct 2025). Although neither paper is about AR directly, both suggest concrete design directions for AR systems that must preserve user- and environment-specific experience without replaying full histories in text.

6. Evaluation, limitations, and open questions

Evaluation remains unsettled because memory can be judged as recall, as action support, or as social appropriateness, and these criteria do not always align. THEANINE introduces TeaFarm, a “counterfactual-driven evaluation scheme” motivated by the limitations of G-Eval and human effort in assessing whether past memories are integrated into response generation (Ong et al., 2024). MemoryArena argues that existing evaluations split memorization and action incorrectly, and instead benchmarks interdependent multi-session tasks where later subtasks are intentionally unsolvable without preserved experience from earlier ones (He et al., 18 Feb 2026). This distinction matters for AR, where the key question is often not whether the system can recall a fact, but whether it can use prior grounded experience to guide later situated action.

Another major limitation is that more memory is not always better. OP-Bench formalizes over-personalization into Irrelevance, Repetition, and Sycophancy, and reports that introducing memory can reduce OP-Bench performance by 26.2% to 61.1% relative to BASE across evaluated systems (Hu et al., 20 Jan 2026). The same paper finds that models often assign more attention to memory tokens than query tokens, with average memory-to-query attention ratio exceeding $$a_{i,t} \sim \pi_A(\cdot \mid s_i, o_{i,1:t-1}, a_{i,1:t-1}, m_{i,t})$$1 (Hu et al., 20 Jan 2026). “Did You Check the Right Pocket?” reaches a systems-level version of the same point: uniform retrieval guarantees coverage, but increases waste, token cost, and irrelevant-context interference; on GPT-4o-mini, oracle routing achieved 86.7% accuracy with 299 context tokens, whereas uniform retrieval achieved 81.3% with 787 tokens (Gaikwad, 8 Mar 2026). These results jointly undermine the misconception that memory quality can be approximated by memory quantity.

Scalability is also an unresolved issue. MRAgent’s graph grows monotonically and defers complex reasoning to query time, which improves selectivity but raises latency for deep multi-step reconstruction (Ji et al., 4 Jun 2026). Livia explicitly raises battery drain, overheating, and continuous AR rendering cost, and recommends efficient rendering, adaptive frame rates, edge computing, and on-device processing (Xi et al., 12 Aug 2025). Continuous GUI memory reports that larger banks may stress latency and GPU memory, and flags freshness, deduplication, and provenance as open problems (Wu et al., 10 Oct 2025). This suggests that long-lived AR memory needs governance mechanisms in addition to retrieval algorithms: consolidation, eviction, freshness control, and store-specific cost models.

Finally, there are domain-specific constraints that the current literature addresses only partially. Privacy and consent are explicit concerns in Livia, which points to encryption, secure storage, explicit consent, and privacy-preserving methods as necessary for multimodal emotional data (Xi et al., 12 Aug 2025). Social appropriateness is explicit in OP-Bench, but only for conversational settings (Hu et al., 20 Jan 2026). Spatial grounding, object permanence across viewpoints, and multimodal uncertainty appear mainly by analogy in embodied RL, GUI memory, and multi-graph memory work rather than in direct AR evaluations (Gupta et al., 22 Oct 2025, Wu et al., 10 Oct 2025, Jiang et al., 6 Jan 2026). This suggests that the central open problem is not whether AR agents should have memory, but how to make memory selective, updateable, latency-bounded, multimodally grounded, and socially appropriate at the same time.