---
title: 'Mem-α: Learned Memory Construction'
url: https://www.emergentmind.com/topics/mem-alpha
type: topic
---

# Mem-α: Learned Memory Construction

Mem-α is a reinforcement learning framework for training large language model agents to construct, update, and compress external memory while processing long information streams. It is motivated by the observation that finite context windows constrain long-term information understanding, while conventional memory-augmented agents typically rely on pre-defined instructions and tools for memory updates rather than learning which information to store, how to structure it, and when to revise it. In Mem-α, memory construction is treated as a sequential decision-making problem, and the agent is optimized by downstream question-answering performance over the full interaction history rather than by hand-crafted memory heuristics [2509.25911].

## 1. Motivation and problem setting

Large language models such as GPT-4 or Qwen-32B are described as fundamentally limited by finite context windows, with the paper giving 32 K tokens as an example. The motivating scenarios are very long information streams, including multi-document question answering, conversational histories, and entire novels. The paper explicitly contrasts two broad approaches: increasing context size through sparse attention or positional interpolation, and attaching an external memory system. It argues that merely expanding context either encounters computational barriers or does not teach the model how to prioritize and organize what to remember [2509.25911].

The central problem is therefore not only storage capacity but memory construction. The paper identifies three unresolved questions for memory-augmented agents: which facts should be written into memory, when an existing memory entry should be updated or pruned, and how different kinds of information should be organized, including personal profile, episodic events, and declarative knowledge. Mem-α addresses these questions by moving from instruction-driven memory management to learned policy optimization.

A common misconception, implicitly challenged by the framework, is that a vector database or key-value store is sufficient for long-horizon competence. The formulation instead treats the existence of external memory as only the substrate; the substantive difficulty lies in deciding what the memory should contain and how it should evolve. This suggests that memory quality, not only memory availability, is a first-order determinant of long-context performance.

## 2. Sequential decision formulation

Mem-α casts memory construction as a sequential decision-making problem. The environment is a sequence of information chunks,
$$
C=(c_1,\ldots,c_n),
$$
drawn from multi-turn interactions such as documents, dialogues, classification examples, and story chunks. At step $t$, the state is
$$
S_t = (c_t, \mathcal{M}_{t-1}),
$$
where $c_t$ is the current chunk and $\mathcal{M}_{t-1}$ is the current external memory [2509.25911].

The action space consists of a possibly empty sequence of memory operations,
$$
a_t=(a_t^{(1)},\ldots,a_t^{(K_t)}),
$$
where each operation belongs to
$$
\{\texttt{memory\_insert}, \texttt{memory\_update}, \texttt{memory\_delete}\}
$$
together with arguments such as entry id, memory type, and textual content. Memory transitions are defined by
$$
\mathcal{M}_t = T(\mathcal{M}_{t-1}, a_t),
$$
with each function call applied in turn.

The reward is computed after the full sequence has been processed, when the final memory $\mathcal{M}_n$ is frozen and evaluated by a retrieval-augmented generation pipeline over a set of questions $Q=\{q_1,\ldots,q_m\}$. Four reward components are combined:

1. **Correctness**:
   $$
   r_1 = \frac{1}{m}\sum_{j=1}^m I[\text{metric}(g(q_j;\phi(\mathcal{M}_n,q_j)), r_j)]
   $$
   where $g$ is a frozen generator, specified as Qwen-32B, $\phi$ is BM25 retrieval, and the metric is dataset-specific, including exact-match, F1, or accuracy.

2. **Function-call format success**:
   $$
   r_{2,t} = \frac{1}{K_t}\sum_{k=1}^{K_t} s(a_t^{(k)}),
   $$
   where $s(\cdot)=1$ if the call parses and executes successfully, else $0$.

3. **Compression**:
   $$
   r_3 = 1 - \frac{|\mathcal{M}_n|}{\sum_t |c_t|},
   $$
   encouraging compact memory.

4. **Semantic validity**:
   $$
   r_{4,t} = \frac{1}{K_t}\sum_{k=1}^{K_t} v(a_t^{(k)}),
   $$
   where $v(\cdot)=1$ if a language-model judge confirms that the operation’s content is semantically appropriate to its memory type.

The per-step reward is
$$
r_t = r_1 + r_{2,t} + \beta \cdot r_3 + \gamma \cdot r_{4,t},
$$
with typical values $\beta=0.05$ and $\gamma=0.1$. The optimization target is the expected return
$$
J(\theta)=E_{C\sim P(C), A\sim \pi_\theta}\left[\sum_{t=1}^n \gamma^t r_t\right],
$$
and training uses a clipped policy-gradient algorithm, Group Relative Policy Optimization (GRPO), with advantages computed using group-normalized returns [2509.25911].

This formulation is notable because the reward is anchored in downstream QA accuracy over the entire interaction history. A plausible implication is that the agent is not directly supervised to imitate a human-designed memory trace; instead, it is optimized for memory utility.

## 3. Structured memory design

Mem-α instantiates a three-component memory,
$$
\mathcal{M}=\{\text{Core}, \text{Episodic}, \text{Semantic}\},
$$
and the paper presents this organization as a memory architecture with differentiated representational roles [2509.25911].

| Component | Content type | Operations |
|---|---|---|
| Core Memory | Short summary of profile, task summary, or learned rules | `memory_update` only |
| Episodic Memory | Chronological list of timestamped events | `memory_insert`, `memory_update`, `memory_delete` |
| Semantic Memory | Bag of discrete fact-statements | `memory_insert`, `memory_update`, `memory_delete` |

Core Memory is defined as a single short summary of at most 512 tokens that captures the user’s profile, ongoing task summary, or learned rules, and it is always kept in the LLM’s prompt. Its only operation is `memory_update`; the agent rewrites the summary holistically when critical new information arrives.

Episodic Memory is a chronological list of timestamped events. The paper gives sentence-level examples such as “At 2023-03-08 01:55, user asked X; assistant replied Y.” Semantic Memory is a bag of discrete fact-statements, with examples including “John is 18 years old” and “Rule: Label 1 means ….” Both episodic and semantic stores support insert, update, and delete operations.

Formally, for $M=\{M_{\text{core}}, M_{\text{epi}}, M_{\text{sem}}\}$, the transition $T(M,a)=M'$ applies the action to a specified sub-memory and entry. At evaluation time, retrieval is restricted to episodic and semantic pools: for a question $q$, the retriever $\phi(M,q)$ returns the top-$k$ entries by BM25 ranking, and the generator $g$ answers using Core plus the retrieved entries as context. The paper also states that consolidation, such as merging similar episodic entries, can be implemented as a post-processing update action if the policy learns to invoke `memory_update` or `memory_delete` appropriately.

The architecture is therefore hierarchical in a concrete sense: persistent prompt-level summary in Core, event trace in Episodic, and decontextualized facts or rules in Semantic. This suggests a division between compressed global state, temporally indexed local state, and reusable declarative state.

## 4. Training data and optimization procedure

To train memory-management behavior, the authors assemble a specialized multi-turn interaction dataset of 4,139 instances spanning three task families [2509.25911].

**Accurate Retrieval (AR)** comprises SQuAD, HotpotQA, Personal LTQA, and a LongMemEval train split. Each instance contains 10–15 sequential chunks of text or dialogue plus 4–100 QA pairs requiring single-hop or multi-hop retrieval.

**Test-Time Learning (TTL)** consists of classification tasks including PubMed-RCT, NLU, and TREC-C, where each chunk presents labeled examples and the agent must learn category rules in memory in order to classify held-out examples.

**Long-Range Understanding (LRU)** uses BookSum chapters segmented into 10–20 chunks, with the agent required to build a summary of the chapter via memory.

Because reinforcement learning is costly and the dataset is class-imbalanced, training uses a stratified sample of 562 instances, with a held-out validation set of 463 instances. The training loop is specified in five stages: initialize $\mathcal{M}_0 \leftarrow \varnothing$; for each chunk, append the chunk and a universal memorization instruction to the prompt, sample actions from $\pi_\theta(\cdot \mid \mathcal{M}_{t-1}, c_t)$, and update memory; after all chunks, freeze $\mathcal{M}_n$ and run RAG QA; compute global rewards $r_1,r_3$ and per-step rewards $r_{2,t},r_{4,t}$; then compute advantages and update $\theta$ via GRPO.

The reported training configuration uses Qwen3-4B as backbone, 205 gradient steps, 32 H100 GPUs, approximately 3 days of training time, learning rate $1\times10^{-6}$, batch size 32, and rollout 8. These details are significant because the paper later identifies RL training as compute-intensive and sensitive to reward-weight hyperparameters.

## 5. Empirical performance and length extrapolation

The empirical evaluation spans nine tasks covering single-hop and multi-hop QA, few-shot classification, and long-document summarization. On the validation-set average, the reported scores are 0.588 for Long-Context, defined as Qwen3-32B truncated to 32 K; 0.567 for RAG-Top2, defined as BM25 plus Qwen3-32B; 0.236 for MemAgent; 0.111 for MEM1; and 0.642 for Mem-α. The paper summarizes the Mem-α result as approximately +9% over Long-Context and +13% over RAG-Top2 [2509.25911].

On the out-of-distribution MemoryAgentBench average, the reported scores are 0.461 for Long-Context, 0.502 for RAG-Top2, 0.198 for MemAgent, 0.071 for MEM1, and 0.592 for Mem-α, corresponding to +28% versus Long-Context and +18% versus RAG-Top2.

| Setting | Baseline comparison | Mem-α result |
|---|---|---|
| Validation-set average | 0.588 Long-Context; 0.567 RAG-Top2 | 0.642 |
| MemoryAgentBench average | 0.461 Long-Context; 0.502 RAG-Top2 | 0.592 |
| Validation memory footprint | 10.8 K Long-Context; 11.3 K RAG-Top2 | 7.9 K |
| Test-suite memory footprint | 207 K RAG-Top2 | 129 K |

The memory-footprint results are important because the reward includes an explicit compression term. On validation, Mem-α’s balanced memory is approximately 7.9 K tokens, compared with 10.8 K for Long-Context and 11.3 K for RAG-Top2. On the test suite, which includes sequences up to 474 K tokens, Mem-α uses approximately 129 K tokens versus 207 K for RAG-Top2. The paper interprets this as efficient compression linked to $r_3$.

The length generalization result is among the most distinctive claims: although trained exclusively on instances with a maximum length of 30 K tokens, the agents generalize to sequences exceeding 400 K tokens, more than 13 times the training length, without further fine-tuning. The paper states that this indicates that the learned policies capture general memory-management principles rather than dataset-specific heuristics. A cautious restatement is that the reported evidence supports extrapolation in the tested offline settings.

## 6. Interpretation, limitations, and future directions

The paper’s principal conclusion is that reinforcement learning can teach LLM agents how to build, update, and compress structured memory more effectively than relying on hand-crafted prompts or static tool instructions [2509.25911]. It also identifies the hierarchical memory design—core summary, episodic events, and semantic facts—as providing the expressivity needed across retrieval, test-time learning, and summarization tasks. In the paper’s interpretation, the reward design is critical because correctness, function format, compression, and semantic validity jointly regulate both what is remembered and how that remembered information is structured.

The work also delineates several limitations. Reinforcement learning training is described as compute-intensive and sensitive to the reward-weight hyperparameters $\beta$ and $\gamma$. The experiments are conducted in a simulated, offline dataset rather than a live tool-enabled environment. The paper therefore leaves open integration with production databases, safety constraints, and latency budgets. These caveats are central to evaluating practical deployment claims.

Future directions are framed as architectural and environmental extensions. The authors state that further memory architectures, including graph memories such as MIRIX, or adaptive retrieval strategies could be incorporated because the RL training is agnostic to the memory’s internal design. This suggests that Mem-α is best understood less as a fixed memory structure than as a training framework for memory policies over explicit state.

A broader misconception addressed by the results is that long-context scaling alone solves long-horizon reasoning. The reported comparison with Long-Context and RAG-Top2 indicates that memory organization and update policy can matter as much as, or more than, raw accessible token count in the evaluated regimes. At the same time, the offline nature of the experiments means that claims about lifelong, real-world memory should be treated as prospective rather than demonstrated.

## 7. Position within memory-augmented LLM research

Mem-α is situated within research on memory-augmented agents but differentiates itself by replacing pre-defined memory instructions with learned memory construction. The framework is therefore neither a pure long-context model nor a conventional retrieval wrapper. Its defining move is to optimize an agent policy over memory operations using downstream task reward [2509.25911].

Conceptually, the framework bridges several strands of work: retrieval-augmented generation through the use of BM25 and a frozen generator; tool-using LLM agents through explicit function calls for insert, update, and delete; and reinforcement learning for sequence-level policy optimization with GRPO. The paper presents this combination as a path between fixed prompt engineering and more general lifelong intelligent systems.

Within that framing, Mem-α can be characterized as a method for learned memory construction under context-window constraints. Its significance lies not simply in adding memory modules, but in formalizing memory management itself as a trainable policy over structured external state.

Source: https://www.emergentmind.com/topics/mem-alpha