MemFactory: Unified Inference & Training Framework for Agent Memory

Abstract: Memory-augmented LLMs are essential for developing capable, long-term AI agents. Recently, applying Reinforcement Learning (RL) to optimize memory operations, such as extraction, updating, and retrieval, has emerged as a highly promising research direction. However, existing implementations remain highly fragmented and task-specific, lacking a unified infrastructure to streamline the integration, training, and evaluation of these complex pipelines. To address this gap, we present MemFactory, the first unified, highly modular training and inference framework specifically designed for memory-augmented agents. Inspired by the success of unified fine-tuning frameworks like LLaMA-Factory, MemFactory abstracts the memory lifecycle into atomic, plug-and-play components, enabling researchers to seamlessly construct custom memory agents via a "Lego-like" architecture. Furthermore, the framework natively integrates Group Relative Policy Optimization (GRPO) to fine-tune internal memory management policies driven by multi-dimensional environmental rewards. MemFactory provides out-of-the-box support for recent cutting-edge paradigms, including Memory-R1, RMM, and MemAgent. We empirically validate MemFactory on the open-source MemAgent architecture using its publicly available training and evaluation data. Across both in-domain and out-of-distribution evaluation sets, MemFactory consistently improves performance over the corresponding base models, with relative gains of up to 14.8%. By providing a standardized, extensible, and easy-to-use infrastructure, MemFactory significantly lowers the barrier to entry, paving the way for future innovations in memory-driven AI agents.

• The paper presents a unified RL framework that standardizes the agent memory lifecycle into composable modules for efficient long-context optimization.
• It leverages the novel GRPO algorithm to minimize VRAM usage and enhance training efficiency, yielding up to 14.8% performance improvement across tasks.
• The framework supports rapid prototyping and reproducible baselines for various memory-augmented architectures, reducing integration burdens on commodity hardware.

MemFactory: A Unified Framework for Memory-Augmented RL Agents

Motivation and Context

Memory-augmented LLMs are indispensable for the transition toward persistent, context-rich AI agents capable of long-range reasoning, continual adaptation, and personalized dialogue across protracted interactions. Fragmentation in the implementation ecosystem has hindered research pace—existing approaches to RL-based memory optimization are typically dataset- or methodology-specific, tightly coupled to ad hoc pipelines, and pose nontrivial integration burdens. While unified training frameworks exist for LLM fine-tuning, they are predominantly stateless and do not support the stateful operations intrinsic to memory-centric agents. MemFactory directly addresses this gap, offering a standardized and modular RL-centric infrastructure designed to facilitate innovation in memory augmentation for LLM-based agents.

Framework Architecture

MemFactory abstracts the entire agent memory lifecycle into composable atomic modules, orchestrated across four primary layers: Module, Agent, Environment, and Trainer.

Figure 1: The overall architecture of MemFactory, illustrating the interdependencies among the Modules, Agent, Environment, and Trainer layers.

• Module Layer encapsulates the engineering of memory operations—structured as Extraction, Update, Retrieval, and end-to-end Agent modules. Each module adheres to standard interfaces (generate, rollout, inference) to support seamless transition between training and deployment, as well as across RL and static paradigms.
• Agent Layer implements policy composition and execution. By leveraging plug-and-play modular integration, agents can be rapidly instantiated for classical architectures (e.g., Memory-R1, MemAgent, RMM) or novel configurations, with direct support for long-context optimization via efficient pre-trained model integration (Transformers, FlashAttention-2).
• Environment Layer mediates data loading and multi-dimensional reward computation. It standardizes state representation and supports both explicit (memory banks) and implicit (long-context) agent-environment interfaces, enabling adaptable reward schedules including LLM-as-judge scoring and format-based correctness.
• Trainer Layer employs Group Relative Policy Optimization (GRPO) for memory- and compute-efficient RL. By normalizing rewards among sampled policy outputs on a per-batch basis, GRPO entirely removes the requirement for a separately parameterized critic, significantly reducing VRAM demands in prolonged-context scenarios.

This architecture enables flexible iteration on memory policy research, supports out-of-the-box baselines for recent SOTA paradigms, and establishes a paradigm for reproducible, unified Memory-RL experimentation.

Technical Foundations and Algorithmic Contributions

Atomicization of Memory Operations

Standardizing the memory pipeline into extraction, update, and retrieval modules mirrors classical CRUD processes but endows the system with reinforcement-driven adaptability. For instance, the NaiveExtractor and NaiveUpdater replicate the paradigm in Memory-R1 but are interchangeable with more advanced selectors or consolidation engines. The RerankRetriever exemplifies modular enhancement by incorporating LRM-based reranking atop base semantic retrieval, reflecting best practices in post-retrieval factuality correction.

For end-to-end memory models (e.g., MemAgent), the Agent Module directly transforms input contexts and latent memory into a new recurrent state, bypassing discrete CRUD steps and mirroring recent trends in neural memory compression through scheduled overwriting rather than incremental modification.

RL Policy Optimization: GRPO

Conventional PPO-based RL for LLMs entails high computational overhead due to the critic. GRPO, as natively integrated in MemFactory, optimizes for both efficiency and stability by computing advantage for each policy instance as follows:

$$\hat{A}_i = \frac{r_i - \text{mean}(\{r_1, \dots, r_G\})}{\text{std}(\{r_1, \dots, r_G\})}$$

where $r_i$ is the reward for the $i^{th}$ response among $G$ candidates. This minimizes VRAM, enables RL training on resource-constrained hardware, and naturally supports outcome-driven or LLM-as-judge reward mechanisms—a critical requirement for long-context or retrieval-augmented tasks.

Empirical Evaluation

MemFactory’s empirical study utilizes the MemAgent architecture as the reference agent, employing Qwen3-1.7B and Qwen3-4B-Instruct as base LLMs. Evaluation encompasses in-domain (eval_50, eval_100) and OOD (eval_fwe_16384) tasks from the MemAgent benchmark suite, with all pipelines executable on a single A800 80GB GPU. Experimental pipeline modifications preserve long-context characteristics while enhancing training efficiency (downsampled context length).

The results substantiate several strong claims:

• Consistent Performance Gain: MemFactory-trained RL agents exhibit a 14.8% average improvement on the smaller base model. The 4B foundation model realizes a 7.3% mean improvement. Gains are maintained under out-of-distribution evaluation—demonstrating the policy generalization capacity of the GRPO-optimized agents.
• Practical Training Efficiency: The entire process is reproducible and tractable on commodity GPU infrastructure, lowering the entry barrier for RL-based memory research.

Implications and Prospects for Memory-RL Research

MemFactory’s unified abstraction is positioned to significantly accelerate empirical research and combinatorial baselining in memory-augmented RL for LLMs. It creates infrastructure for:

• Rapid Prototyping: Modular assembly and direct comparison of memory lifecycles, retrieval, and update policies.
• Benchmarking and SOTA Integration: Out-of-the-box implementations for Memory-R1, MemAgent, and RMM enable systematic SOTA benchmarking.
• RL-driven Agent Innovation: Researchers can explore nontrivial reward schedules and train recurrent or compositional agents with minimal engineering overhead.

The architecture is extensible—future directions could integrate hybrid offline/online RL, multi-agent memory synchronization, or dynamic data curriculum. Additionally, the modular interface could facilitate direct adoption of more sophisticated reasoning and reranking modules, or multi-modal memory engines. The native use of sample-efficient RL algorithms (GRPO) suggests practical viability for scaling toward billion-parameter class LLMs and even multi-task/autonomous agent platforms.

Conclusion

MemFactory fills a critical infrastructure void in the LLM ecosystem, presenting a modular, extensible, and RL-native framework for memory-augmented agents (2603.29493). It standardizes interfaces and decouples engineering, enabling rapid adoption and innovation in policy-driven memory management. Empirical evidence demonstrates its effectiveness in both in-domain and OOD tasks, justifying its adoption as a baseline for RL-based memory research. Its practical and theoretical flexibility positions it as a foundational infrastructure for the next wave of persistent, adaptive, long-horizon language agents.