Memory Horizon and Augmentation

• Memory horizon and augmentation are fundamental concepts that define the effective temporal span and enhancement of memory in both cognitive and artificial systems.
• They employ methods such as global tokens, hierarchical memories, and structured layouts to extend context and improve retrieval accuracy.
• Empirical findings show significant performance gains in reinforcement learning, language models, and multimodal agents through optimized memory encoding, compression, and recall.

Memory horizon and memory augmentation constitute foundational axes in the study of cognitive and artificial agents capable of robust reasoning, planning, and action over extended temporal ranges. The memory horizon quantifies the effective temporal scope from which information can be stored, retrieved, and leveraged for downstream decision-making, while memory augmentation encompasses a diverse array of mechanisms that extend, compress, or otherwise structure this ability. As modern systems—spanning reinforcement learning agents, LLMs, world models, and multimodal agents—confront ever-longer contexts and more complex information streams, advances in memory horizon and augmentation are pivotal to enabling long-horizon intelligence.

1. Formal Definition and Taxonomy

The memory horizon, often conceptualized as the effective memory span $H^*$, is mathematically defined as the maximum temporal distance $H$ over which a system can maintain high-fidelity recall or leverage past information for correct prediction or action: $H^* = \max\{ H : R(H) \leq \epsilon \}$ for suitable recall/error metric $R(H)$ and error tolerance $\epsilon$ (Laird et al., 7 Dec 2025). This concept manifests in multiple domains:

• Reinforcement Learning (RL): $H^*$ reflects the time span over which transitions in the replay buffer influence value updates and policy adaptation, subject to buffer size, sampling, and memory updating schemes (Ramicic et al., 2019).
• Transformer-based Models: $H^*$ is fundamentally limited by context window size and attention span, mitigated by external memories, summary modules, and memory augmentation architectures (Gupta et al., 2020, Laird et al., 7 Dec 2025).
• World Models/Simulation: Memory horizon governs the maximum predictive or planning rollouts before perceptual drift or error accumulation degrades performance (Laird et al., 7 Dec 2025, Hong et al., 3 Dec 2025).

Memory augmentation refers to any explicit architectural, algorithmic, or training intervention designed to increase $H^*$, selectively preserve salient information, or optimize which memories are retained and accessed. Key axes of this taxonomy are memory encoding (how is history compressed/structured) and memory injection (how does the stored memory interface with a model's main computation) (Laird et al., 7 Dec 2025). Modalities include discrete key-value memories, hierarchical stores, attention-enabled caches, summary graphs, and even structured visual layouts (Shi et al., 29 Jan 2026).

2. Core Mechanisms for Extending Memory Horizon

2.1 Transformer and LM Variants

Global Memory Augmentation (GMAT):

GMAT prepends a small number $M$ of global memory tokens to the input, which are densely attended to in each layer, while the main token sequence uses sparse attention. Every token thus receives global, compressed context at each layer, extending memory horizon with $O(M(L+M))$ complexity—well below the $H$0 baseline for $H$1 (Gupta et al., 2020).

External and Hierarchical Memories:

Mem-T introduces a hierarchical memory database with distinct working, factual, experiential, and raw memory stores, enabling long-horizon agents to manage thousands of entries with dense, operation-attributed RL rewards and dynamic compression (Yue et al., 30 Jan 2026). Similarly, Memory-R2 employs co-optimized extractor and manager roles to segment, update, and compress memory streams across dozens of sessions, with credit assignment spanning both local and global scales (Yan et al., 20 May 2026).

Structure-Preserving Schemas:

AMA-Agent on AMA-Bench utilizes explicit causality graphs and tool-augmented retrieval to preserve logical dependencies that are frequently lost in lossy summarization or similarity-based retrieval, enabling robust reasoning across tens of thousands of tokens per trajectory (Zhao et al., 26 Feb 2026). GRAVITY injects structured entity, event, and topic representations directly into prompts for conversational agents, boosting LLM-judge performance by up to 10.1% on long-horizon benchmarks (Sun et al., 3 May 2026).

Memory-Embedding and Retrieval:

LMEB demonstrates that memory retrieval over long, temporally structured histories imposes fundamentally different requirements than standard passage retrieval, as evidenced by the weak correlation between MTEB and LMEB scores and the necessity for specialized, task-conditioned embeddings (Zhao et al., 13 Mar 2026).

2.2 RL and Control

Augmented Memory Replay (AMR):

AMR modulates the probability and reward weighting of transitions in the replay buffer, focusing learning on transitions with high TD error, reward, or entropy. This selective replay preserves informative experiences and maintains learning stability over longer horizons (up to +35.4% improvement in tasks like Reacher-v2) (Ramicic et al., 2019).

Meta-Reinforcement Learning with Memory Augmentation:

Meta-RL with memory-augmented LSTM architectures implicitly extends the memory horizon through stateful hidden memory, carried over PPO rollouts and training epochs. This enables context-aware task inference and superior generalization to out-of-distribution tasks without explicit windowing or hand-tuned truncation (Bao et al., 3 Feb 2025).

VQ-Memory for Robotics:

VQ-Memory applies vector-quantized VAEs to encode proprioceptive state windows into discrete token streams, achieving extreme compression and noise robustness for long-horizon manipulation tasks in non-Markovian domains. Gains are notable (>+30% success on multi-task benchmarks), with minimal overhead (L=40 integer tokens) (Honghui et al., 10 Mar 2026).

3. Advanced Architectures in Multimodal Contexts

Layout-Aware Visual Memory:

MemOCR departs from 1D text memory by rendering structured rich-text memories (with explicit salience cues via font, headings, and layout) into 2D images. Visual tokenization, coupled with budget-aware RL (GRPO), allows the agent to aggressively compress unimportant details while ensuring crucial facts survive extreme downsampling. This yields up to 8× greater effective context utilization and robustness compared to text baselines, especially under tight context budgets (Shi et al., 29 Jan 2026).

Continuous Multimodal Memory:

Auto-scaling GUI agents (CoMEM) encode entire interface trajectories as fixed-length sequences of learned visual–action embeddings via Q-Former networks, sidestepping the O($H$2) context bloat of text memories and achieving monotonic performance gains up to 100k stored segments, with empirical improvements sustained under both scaling and distribution shift (Wu et al., 10 Oct 2025).

MementoGUI:

This system modularizes memory into working and episodic components managed by a suite of learned controllers for selection, compression, and retrieval, effectively separating transient state from reusable cross-episode experience. Plug-and-play adaptation to frozen or closed-source MLLM GUI models is enabled, and multimodal fusion (text, ROI-level crops) increases step and trajectory success rates as episode length and memory bank size grow (Zeng et al., 18 May 2026).

Streaming World Models with Space-Time Compression:

Matrix-Game 3.0 and RELIC introduce spatially-compressed, camera-aware memory caches, unified memory-history-latent fusion with direct self-attention, and memory-aligned distillation. These allow real-time (>16 FPS), minute-scale (1,500+ frames) memory-consistent video generation with robust scene revisiting and long-horizon consistency in camera-based navigation (Hong et al., 3 Dec 2025, Wang et al., 10 Apr 2026).

4. Training Objectives and Optimization Paradigms

Uncertainty-Driven Memory Optimization:

Meta-Cognitive Memory Policy Optimization (MMPO) substitutes sparse end-task reward with dense, entropy-based penalties/rewards determined by epistemic uncertainty (Belief Entropy) at each memory update. This reduces semantic noise and “lost-in-the-middle” collapse in multi-million-token reasoning tasks, with improvements up to +5 points at 3.5M tokens (Liu et al., 28 May 2026).

Credit Assignment under Non-Stationarity:

Memory-R2 (LoGo-GRPO) resolves the fundamental problem of non-stationarity caused by memory edits across rollouts by incorporating local rerollouts from the same intermediate memory state and global (end-to-end) rollouts, ensuring both end-reward propagation and fair stepwise credit (Yan et al., 20 May 2026). MoT-GRPO (Mem-T) extends this by constructing retrieval-operation trees and propagating dense nodewise advantages, with ablated results showing major F1 degradation if reward densification or hierarchical structure is removed (Yue et al., 30 Jan 2026).

Zero-Intrusion, Plug-and-Play Augmentation:

GRAVITY and plug-in memory modules achieve non-intrusive augmentation of any base LLM, feeding structured anchors at prompt time and yielding consistent accuracy gains with no access to model internals, thus supporting rapid deployment across diverse memory systems (Sun et al., 3 May 2026).

5. Evaluation Protocols and Empirical Findings

Key benchmarks and metrics are tailored to measure memory horizon and the impact of augmentation:

• AMA-Bench: Multi-domain, long-horizon agentic trajectories paired with rigorous expert and synthetic QA probe recall, causality, state updating, and abstraction. Major findings include the necessity of causality graphs and tool-augmented retrieval; similarity-only or compressive methods degrade sharply as sequence lengths reach 128K tokens (Zhao et al., 26 Feb 2026).
• LMEB: 193 zero-shot retrieval tasks testing episodic, dialogue, semantic, and procedural memory over 22 datasets. Standard passage retrieval models perform poorly; performance does not simply monotonically scale with model size, and orthogonality with traditional retrieval is observed (Zhao et al., 13 Mar 2026).
• State Recall Tasks in World Models: Explicit evaluation of reconstructive fidelity after H-step open-loop rollouts reveals attention-based memory injection and SSM encoding provide the highest memory horizon and best performance on loop closure and perceptual drift prevention (Laird et al., 7 Dec 2025).
• Ablations: Removal of structured layout in MemOCR or of episodic/working memory in GUI agents produces severe robustness and accuracy drops, confirming the operational importance of these modules (Shi et al., 29 Jan 2026, Zeng et al., 18 May 2026).

6. Design Limitations, Trade-offs, and Future Directions

• Trade-offs: Overly aggressive compression (in layout-based memories or VQ-coding) risks obliterating fine-grained details necessary for stepwise or comparative reasoning. Memory-augmented models can incur increased latency or memory requirements, mitigated by spatial, temporal, or semantic compression schemes (Hong et al., 3 Dec 2025, Shi et al., 29 Jan 2026).
• Modality and Multilingual Coverage: Most empirical work remains limited to English textual or 2D visual domains; extending memory augmentation to multimodal, multilingual, and embodied agent settings remains a key challenge (Zhao et al., 13 Mar 2026).
• Dynamic Memory Adaptation: Dynamic switching between levels of detail, condensation, and resurfacing of old memories is an active area of research, with initial efforts leveraging gates, attention, and multi-tier stores (Bao et al., 3 Feb 2025, Zeng et al., 18 May 2026).
• Lifelong and Cross-Episode Memory: Progress in cross-episode retrieval, transfer, and lifelong memory structures is ongoing, with benchmarking (AMA-Bench, LMEB) and structured graph augmentation as research drivers (Zhao et al., 26 Feb 2026).
• Evaluation/Benchmarks: New metrics and tasks are required to separately probe memory formation, compression, updating, and retrieval—rather than end-to-end QA or success alone.

7. Summary Table: Representative Memory Augmentation Strategies

Category	Example System	Core Augmentation/Objective	Reported Gain
Global Memory Tokens	GMAT (Gupta et al., 2020)	M global tokens, dense access per layer	Up to +5-10% on global tasks
Structured Layouts	MemOCR (Shi et al., 29 Jan 2026)	2D visual memory, RL-salience budgets	+8×–10× token efficiency
RL Replay Augmentation	AMR (Ramicic et al., 2019)	TD error/entropy modulated replay	+12–35% cumulative reward
Memory-embedding	CoMEM (Wu et al., 10 Oct 2025)	Q-Former segment embeddings, monotonic scaling	+16–24% success vs text-only
Hierarchical Multi-Tier	Mem-T (Yue et al., 30 Jan 2026)	Factual/episodic/raw stores, MoT-GRPO	+14.9% F1, –24.5% tokens
Causal/Graph Augmented	AMA-Agent (Zhao et al., 26 Feb 2026)	Causality-graph + tool retrieval in agentic logs	+11.16% over best baseline
Metacognitive RL	MMPO (Liu et al., 28 May 2026)	Belief Entropy minimization at every summary step	+3–5% at scales >1M tokens
Plug-in Structure	GRAVITY (Sun et al., 3 May 2026)	Entity/event/topic injection as structured anchors	+7.5–10.1% LLM-judge accuracy

Memory horizon/augmentation thus constitutes a paradigmatic scaffold for the design of high-capacity, long-horizon intelligent systems. Advances along this dimension enable effective reasoning, planning, and adaptation in settings ranging from RL, language modeling, to multimodal simulation and embodied control. Central open challenges remain in dynamic memory adaptation, robust multimodal fusion, and the co-optimization of memory construction, compression, and retrieval, with empirical progress documented across diverse agentic and modeling domains.