MemFactory: Unified Memory Management in AI

• MemFactory is a suite of frameworks that combines factorized neural architectures, modular reinforcement learning pipelines, and graph-based inference to manage explicit memory in AI systems.
• Factorization Memory replaces traditional self-attention with slot-based recurrent updates, matching Transformer performance on short contexts while excelling in long-context extrapolation.
• MemFactory frameworks leverage reinforcement learning and proactive message passing to optimize memory retrieval and inference, enabling robust, scalable deployment in diverse applications.

MemFactory encompasses a set of distinct but foundational frameworks in the machine learning literature that address memory-augmented architectures, unified agent-training pipelines for memory management, and probabilistic graphical models over memory factors. Three principal threads characterize the domain: (1) Factorization Memory as a recurrent neural architecture for language modeling (Xiong et al., 31 Oct 2025), (2) the MemFactory framework for modular training and inference of memory-augmented agents using reinforcement learning (Guo et al., 31 Mar 2026), and (3) Memory Factor Networks and Proactive Message Passing as a convergent message-passing scheme for data-efficient inference (Eschenfeldt et al., 2016). These approaches, while unified under the general umbrella of explicit memory management, diverge in technique, application domain, and theoretical guarantees.

1. Factorization Memory Architecture for Language Modeling

Factorization Memory, sometimes referred to as "MemFactory" (in the context of (Xiong et al., 31 Oct 2025)), is an efficient RNN architecture that achieves Transformer-comparable performance on short-context tasks and exhibits superior extrapolation on extended-context sequences. This model factorizes the hidden state into $m$ parallel slots of dimension $d_\text{memory}$, replacing the traditional $d_\text{model}$-dimensional state.

Core Recurrence

At each timestep $t$, the model computes memory-slot affinities, update ratios, and merges representations: $\alpha_t = \softmax(W_\alpha x_t) \in \mathbb{R}^m$

$$\eta_t = \sigma(w_\eta^\top x_t),\quad \mu_t = \sigma(w_\mu^\top x_t)$$

$$\theta_t = \eta_t \alpha_t,\qquad \phi_t = \mu_t \alpha_t$$

Input is projected to memory space: $\bar{x}_t = W_i x_t$ The recurrent update across slots: $h_t = \diag(1 - \theta_t) h_{t-1} + \theta_t \otimes \bar{x}_t$ Output is produced by merging normalized slot states: $y_t = W_o \left(\norm(h_t)\right)^\top \phi_t$

Complexity and Sparsity

The per-token complexity is $d_\text{memory}$0; under sparse slot activation (top-$d_\text{memory}$1 routing), computational cost reduces to $d_\text{memory}$2 without degrading performance.

Empirical Results

Factorization Memory matches or slightly lags Transformer models on short-context tests and surpasses state-space models (e.g., Mamba-2) in both FLOP-normalized performance and resistance to context-length degradation at up to 128K tokens. Downstream average accuracy (1B parameter, English/JP): FM: 31.0 vs Transformer: 29.5 vs Mamba-2: 29.1.

Practical Integration

Factorization Memory layers can directly replace self-attention blocks in Transformer stacks, enabling constant memory inference and efficient large-context deployment. Custom CUDA/Triton kernels support both dense and sparse regimes (Xiong et al., 31 Oct 2025).

2. MemFactory: Unified Training and Inference for Memory-Augmented Agents

MemFactory (Guo et al., 31 Mar 2026) is an open-source, modular framework for constructing, training, and evaluating memory-augmented LLM agents, particularly those operating over long-term or externalized memory via reinforcement learning.

Modular Architecture

MemFactory abstracts the memory agent lifecycle into four core atomic modules:

• MemoryExtractor: selects discrete memory candidates from unstructured input.
• MemoryUpdater: determines CRUD (Create, Read, Update, Delete) operations on the memory bank, learnable via RL.
• MemoryRetriever: fetches top-$d_\text{memory}$3 relevant memories using embedding or LLM reranking.
• AgentModule: encapsulates end-to-end policies (e.g., latent recurrent memory modules).

Each module conforms to standardized generate, rollout, and inference interfaces, with "Lego-like" composition to assemble custom agents.

Policy Optimization: Group Relative Policy Optimization (GRPO)

MemFactory natively implements GRPO, which eliminates the need for a separate value network. For each group of $d_\text{memory}$4 trajectories: $d_\text{memory}$5

$d_\text{memory}$6

This yields memory-efficient RL with outcome-driven multi-dimensional rewards, including format-based, coverage, and LLM-judged signals.

Supported Paradigms

MemFactory ships with plug-and-play implementations for:

• Memory-R1: simple pipeline with modular extraction, update, and vector search.
• RMM: combines incremental extraction and reranked retrieval, reward by citation accuracy.
• MemAgent: recurrent latent memory, reward by retrieved/used memory accuracy.

Empirical Performance

On the MemAgent release suite, RL-trained agents with MemFactory yield up to 14.8% improvement in average task metrics for 1.7B parameter models, and 7.3% OOD improvement for 4B-Instruct variants (see detailed table below).

Model	Setting	eval_50	eval_100	eval_fwe_16384	Average
Qwen3-1.7B	Base checkpoint	0.4727	0.4297	0.0332	0.3118
	+ MemFactory RL	0.5684	0.4863	0.0195	0.3581
Qwen3-4B-Instruct	Base checkpoint	0.6523	0.5645	0.6270	0.6146
	+ MemFactory RL	0.7051	0.6309	0.6426	0.6595

3. Memory Factor Networks and Proactive Message Passing

Memory Factor Networks (MFNs) (Eschenfeldt et al., 2016) are bipartite graphical models with memory and evidence factors that encode structural or exemplar-driven constraints. Associated Proactive Message Passing (PMP) performs convergent, political-ballot-style message passing to minimize joint costs over variable and factor "votes."

Formal MFN Structure

Variables $d_\text{memory}$7 take values in (possibly heterogeneous) alphabets $d_\text{memory}$8. Each factor-to-variable edge $d_\text{memory}$9 has a "vote" $d_\text{model}$0 and weight $d_\text{model}$1. The global cost to minimize: $d_\text{model}$2 with selection costs $d_\text{model}$3 (either enforcing table-matches or subspace projections).

PMP Algorithm

PMP minimizes $d_\text{model}$4 by iteratively activating abstaining factors, updating local votes via: $d_\text{model}$5 where $d_\text{model}$6 sums the minimal mismatch plus neighbors' effects. Closed-form efficient computation is enabled by maintaining consensus statistics/histograms as local summary messages ($d_\text{model}$7).

Theoretical Properties

PMP guarantees monotonic reduction in abstentions and global cost, converging in finitely many rounds to a local minimum (unlike generic loopy BP).

Applications

MFNs with PMP have demonstrated state-of-the-art or competitive results on structured reconstruction (face images, CIFAR-10 restoration), sequence interpolation (music spectrograms), and hierarchical classification (MNIST), e.g., 96.41% MNIST accuracy with 17 parallel rounds, and 91.9% perfect restoration rate on CIFAR-10 corrupted images (Eschenfeldt et al., 2016).

4. Comparative Complexity and Practical Considerations

System	Core Complexity	Memory Usage	Inference Cost
Factorization Memory	$d_\text{model}$8, sparse: $d_\text{model}$9	$t$0 recurrent slots	Constant per token
MemFactory (RL Agents)	Dominated by base LLM forward + RL group rollout	Configurable via environment, no value net in GRPO	Bounded by LLM + module cost
MFN + PMP	$t$1	Table or factor subspace storage	Parallelizable per update

Practical implementation of Factorization Memory and MemFactory utilizes custom GPU kernels and batching, while MFNs benefit from strategic caching, parallel batch voting, and local optimization structures for memory-factor subproblems.

5. Extensibility, Limitations, and Future Directions

Current MemFactory release covers three main Memory-RL paradigms, with modular interfaces poised for community extension—e.g., hierarchical, multi-granular, or multi-modal memory modules, and human-in-the-loop RL. Reward quality remains a limitation due to reliance on format-judged and LLM-judged proxies. Future development directions identified include curriculum RL, continual and lifelong learning, and combining GRPO with parameter-efficient fine-tuning (Guo et al., 31 Mar 2026).

For Factorization Memory, scaling $t$2 (slot count) and $t$3 (per-step sparsity) tunes capacity and efficiency, while custom slot-normalization and router temperature trading off sharp slot selection in long vs short context.

MFN/PMP methods are extensible to mixed variable types and complex data domains, providing a provably convergent alternative to standard BP. Efficient rollout to new problem classes requires careful factor and memory construction, balancing memory-table and subspace representations for task fidelity.

6. Summary and Research Significance

"MemFactory" thus denotes either a neural ("Factorization Memory") or software-centric (modular RL pipeline) architecture for scalable, efficient memory in modern AI, and the foundational message-passing structures used in probabilistic inference over memory-augmented models. These frameworks underpin robust long-horizon sequence modeling, enable reproducible experimentation in memory-driven agent RL, and provide scalable, convergent inference over structured data. Distinct, but complementary, these MemFactory approaches have established new empirical and theoretical baselines for memory management in neural and graphical models (Xiong et al., 31 Oct 2025, Guo et al., 31 Mar 2026, Eschenfeldt et al., 2016).