Memory-as-Action in Cognitive Architectures

• Memory-as-Action is a paradigm that treats memory as an active computational and contextual module integrated into cognitive and robotic systems.
• It employs modular architectures combining working, long-term, and episodic memories with dynamic retrieval and editing operations for real-time decision-making.
• Empirical studies show marked improvements in task success, prediction accuracy, and efficiency across autonomous agents, robotics, and motion prediction.

Memory-as-Action is a paradigm in cognitive architectures and agentic learning systems that treats memory as an active module with operational significance for perception, reasoning, and control rather than a passive repository. In this perspective, memory continuously engages in computation, decision-making, management, and context curation serving the agent's objectives and constraints in complex, temporally extended tasks. The concept underlies recent advances in robotics, vision-language-action (VLA) models, reinforcement learning for autonomous agents, and human motion prediction, where memory structures, curates, and transforms historical context directly as an actionable part of the policy.

1. Conceptual Definition and Foundations

Memory-as-Action positions memory as an operational system that not only enables recall but actively mediates, transforms, and manages internal representations for ongoing reasoning and execution. Cognitive robot architectures (e.g., ArmarX) formalize memory as a “central active component . . . that mediates between semantic and sensorimotor representations, orchestrates the flow of data streams and events, and provides components . . . with data-driven services for abstraction . . . parametrization . . . [and] prediction of action effects” (Peller-Konrad et al., 2022). This active role encompasses multi-modal abstraction, associativity, introspection, episodic structuring, and adaptive resource management.

The notion extends to VLA models and autonomous agents, where memory is explicit in policy design and learning objectives rather than a fixed background process. In such systems, memory acts via retrieval, fusion, consolidation, management, and direct influence on decision/planning modules (Shi et al., 26 Aug 2025, Zhang et al., 14 Oct 2025, Li et al., 12 Nov 2025).

2. Architectural Realizations: Mechanisms of Actionable Memory

Architectural instantiations of Memory-as-Action typically include modular separation of working, long-term, and episodic memories, endowed with interfaces for manipulation and computational services:

• Working Memory: Immediate representations, fast access, low latency, integrated with current sensory inputs (e.g., perceptual and cognitive tokens in MemoryVLA (Shi et al., 26 Aug 2025)).
• Long-term/Episodic Memory: Banks, archives, or libraries indexed temporally and semantically, storing consolidated multi-modal entries (e.g., Perceptual–Cognitive Memory Bank (PCMB), soft-prompt libraries (Shi et al., 26 Aug 2025, Li et al., 12 Nov 2025)).
• Active Services: Retrieval via similarity search, dot-product attention, gate-fusion, consolidation via redundancy merging, prediction of future states (autoencoders, latent transition models), context curation operations (insert, delete, reorder, compress) as policy-level actions (Shi et al., 26 Aug 2025, Zhang et al., 14 Oct 2025, Peller-Konrad et al., 2022).
• Distributed Implementation: Memory servers per modality with introspectable formats, event-driven orchestration, and adaptive migration between working and long-term storage (Peller-Konrad et al., 2022).

A concise comparison is presented below:

Component	Role in Action	Example Mechanism
Working Memory	Buffer for control/decision	Token encoding, fast access, immediate embeddings
Episodic/Long-Term Memory	Historical context, prediction	Memory bank, soft prompts, AE archives
Memory Actions	Direct manipulation	Insert/delete/merge/summarize; explicit policy actions
Compression/Predictive	Structure and future queries	Autoencoding, latent transition, future-time queries

3. End-to-End Memory-Conditioned Action Generation

Recent frameworks tightly couple memory retrieval and manipulation procedures as integral steps in agent policy and control pipelines:

• VLA-Based Robotic Manipulation: MemoryVLA constructs working memory from perceptual and cognitive tokens; PCMB stores episodic entries with temporal encoding. Control proceeds via retrieval (Transformer-based attention), gate-fusion, and dynamic consolidation to maintain compactness (Shi et al., 26 Aug 2025). Action sequences are generated by memory-conditioned diffusion models, with temporally-aware trajectories.
• Demonstration-Episodic Prompting: MAP-VLA builds stage-segmented memory libraries from historical demonstrations, injects learned soft prompts into frozen VLA models, and ensembles base/memory-driven predictions via dynamic weights (Li et al., 12 Nov 2025).
• Stochastic Human Motion Prediction: Dual memory banks (STAB/ACB) for transition and characteristic priors, with adaptive fusion at each step (AAA), support temporally consistent and semantically correct prediction in CVAE-based models (Tang et al., 5 Jul 2025).
• Agentic RL Context Management: Autonomous context curation treats memory editing as first-class policy actions, managing the trajectory structure under resource constraints via Dynamic Context Policy Optimization (DCPO) (Zhang et al., 14 Oct 2025).

The operational pipeline in these systems typically alternates between action selection and explicit memory manipulation, ensuring that temporally relevant or decision-critical information is retained, fused, or updated at each step.

4. Explicit Memory Actions and Policy Learning

A distinguishing principle in the Memory-as-Action paradigm is the formal inclusion of memory-editing actions within the action set available to agents. Concretely, memory actions can include insertions, deletions, reorders, and compressions of the working memory or historical context. These are modeled as higher-order functions on the agent's history:

$a: H \longmapsto H'$

where $a$ chooses which subsequences to affect, enabling the policy $\pi_\theta(a\mid H)$ to jointly optimize task performance and resource constraints (Zhang et al., 14 Oct 2025). Such formulation is requisite for dynamic context management in long-horizon tasks, where context limits and irrelevant history can degrade both performance and survivable memory budgets.

DCPO further addresses trajectory fractures arising when non-prefix memory edits disrupt standard policy gradient assumptions; by segmenting the trajectory at memory action points and assigning advantages at the segment level, unbiased learning and accurate credit assignment are restored (Zhang et al., 14 Oct 2025).

5. Memory Banks, Attention, and Retrieval Mechanisms

The actionable characteristic of memory is often realized through learnable external memory banks, attention mechanisms, and retrieval policies:

• Soft-Transition Action Bank (STAB): Stores key-value representations for transitions indexed by past and future action pairs, retrieved via dot-product similarity and weighted aggregation (Tang et al., 5 Jul 2025).
• Action Characteristic Bank (ACB): Encodes priors for future actions, enabling semantic consistency in predicted sequences.
• Adaptive Attention Adjustment (AAA): Dynamically interpolates between transition and characteristic features at each timestep, parameterized by $\alpha_t$, which evolves based on cross-entropy losses during sequence generation.
• Perceptual-Cognitive Memory Bank (PCMB): Episodic structures storing perceptual and cognitive tokens, with retrieval via Transformer attention and gate-fusion to combine current and historical context, followed by redundancy-aware consolidation (Shi et al., 26 Aug 2025).
• Prompt-Tuned Memory Libraries: In MAP-VLA, demonstration-derived memory units (soft prompts) are learned per stage, aligned via trajectory segmentation and dynamic time-warping. Retrieval is performed via trajectory similarity matching, and integration employs dual forward passes with dynamic weighting (Li et al., 12 Nov 2025).

This actionable design ensures that memory is queried adaptively for each prediction or control command and that memory content is updated, merged, or filtered on a continual basis in accordance with ongoing experience and resource boundaries.

6. Empirical Evaluation and Impact

Memory-as-Action systems demonstrate marked improvements over baselines in long-horizon, temporally dependent tasks, as measured by success rates, efficiency, and prediction accuracy:

• MemoryVLA (Shi et al., 26 Aug 2025): Success rates surpass state-of-the-art baselines by 3–26 percentage points on multiple simulation and real-world robotic manipulation suites; critical gains are observed in tasks where immediate observation is ambiguous or insufficient.
• MAP-VLA (Li et al., 12 Nov 2025): Yields 7.0% absolute and 9.2% relative gain in simulation, up to 25% absolute gain in real-robot evaluations for long-horizon manipulation, with strong robustness under visual perturbations.
• Stochastic Human Motion (Tang et al., 5 Jul 2025): Action accuracy and diversity metrics consistently exceed earlier CVAE approaches; ablations confirm the necessity of both STAB and ACB for smooth transitions and per-action semantic fidelity.
• Agentic RL (Zhang et al., 14 Oct 2025): RL-trained MemAct policies achieve higher QA accuracy with substantially lower token consumption than both heuristic and larger-model baselines; joint optimization enables 40% higher rollout speed and improved context efficiency.
• ArmarX (Peller-Konrad et al., 2022): Active memory services yield efficient transfer (<7 ms per commit), extreme compression (>99% reduction rates), accurate reproduction (47 dB PSNR), and future-predictive capability (30–34 dB PSNR).

A plausible implication is that treating memory as a first-class, actionable policy component is essential for complex, non-Markovian domains, particularly as system scale and horizon length increase.

7. Outlook and Directions

While actionable memory mechanisms have been validated across robot control, autonomous RL, motion generation, and multi-modal agent systems, several areas remain for further development:

• More sophisticated memory-editing operations, such as hierarchical summarization or causal graph construction, may enable deeper abstraction and planning in lifelong agentic settings.
• Scaling distributed architectures (e.g., ArmarX’s memory servers) with robust introspection and migration could facilitate memory-as-action in multi-agent and embedded environments (Peller-Konrad et al., 2022).
• Integration with meta-learning and continual learning frameworks holds potential for adaptive resource allocation and lifelong context curation.
• Policy learning techniques like DCPO establish a blueprint for unbiased RL with trajectory fractures, but further algorithmic refinements could improve stability and generalization.

The convergence of memory, attention, and action in contemporary architectures evidences an emergent shift away from static, external memory and toward holistic, policy-coupled memory-as-action systems optimized via direct credit assignment and resource-aware learning strategies.