MemexRL: Indexed Memory for LLM Agents

• MemexRL is a reinforcement-learning framework for long-horizon LLM agents that uses an indexed experience memory to separate a compact working context from an external full-fidelity archive.
• It employs explicit memory tools to compress and retrieve evidence, enabling agents to maintain precise summaries while recovering exact past interactions on-demand.
• Empirical results indicate a task success jump from 24% to 85% with reduced working context size and optimized memory operations under complex long-horizon tasks.

MemexRL is a reinforcement-learning framework for long-horizon LLM agents built around Memex, an indexed experience memory mechanism that keeps a compact working context while storing full-fidelity past interactions in an external experience database under stable indices. Instead of relying on truncation or running summaries, Memex maintains concise structured summaries and stable indices in-context, and allows the agent to decide when to dereference an index and recover the exact past evidence needed for the current subgoal. In the specific formulation introduced in "Memex(RL): Scaling Long-Horizon LLM Agents via Indexed Experience Memory," memory operations are first-class actions optimized with reward shaping under a context budget, so the agent learns what to summarize, what to archive, how to index it, and when to retrieve it (Wang et al., 4 Mar 2026).

1. Concept and problem setting

MemexRL addresses the long-horizon failure mode in which trajectories accumulate large tool outputs, intermediate reasoning traces, and early facts that remain decision-relevant later, while the context window remains finite. The motivating claim is that simply appending everything makes prompts huge and eventually impossible to fit, truncation discards evidence outright, and running or hierarchical summaries are fundamentally lossy because exact tool outputs, IDs, and fine-grained artifacts are replaced by paraphrases (Wang et al., 4 Mar 2026).

The core distinction of Memex is the separation between a compact working context and an external full-fidelity archive. The working context contains a concise indexed summary of progress and state, while the external database stores exact artifacts such as logs, code snippets, directory listings, or environment observations under stable keys. This design departs both from summary-only memory and from ordinary retrieval-augmented memory with semantic search: the summary is pointer-heavy rather than evidence-heavy, and retrieval is deterministic dereferencing of named indices rather than approximate similarity search (Wang et al., 4 Mar 2026).

A plausible implication is that MemexRL treats memory management not as passive storage but as an explicit control problem. In that sense, it sits in the same broader research line as systems that cast memory retrieval, pruning, or reuse as learned decision-making, but its defining emphasis is indexed, auditable recovery of exact prior evidence rather than fuzzy retrieval or purely compressed memory.

2. Indexed experience memory architecture

The Memex agent consists of an LLM policy, a working context, an external experience database, and two memory tools: CompressExperience and ReadExperience. The full prompt at step $t$ is

$\mathbf{M} = [m_0, u, \mathbf{M}_{\mathrm{work}}],$

where $m_0$ is the fixed system prompt, $u$ is the task instruction, and $\mathbf{M}_{\mathrm{work}}$ is the dynamic working context. The external store is a key–value database

$$\mathcal{D} : \mathsf{index} \mapsto \mathsf{content},$$

which holds full-fidelity artifacts under stable indices (Wang et al., 4 Mar 2026).

The in-context memory object is an Indexed Summary

$\sigma = (s, \mathcal{I}),$

where $s$ is a compact, actionable progress state and

$$\mathcal{I} \triangleq \{(\mathsf{index}, \mathsf{description})\}$$

is a finite set of index–description pairs. The index map tells the agent what each stored block contains, while keeping the full artifact off-context. After a compression event, the working context is reduced to the system prompt, the task, and the indexed summary, while the earlier reasoning and tool outputs are removed from the prompt and written into $\mathcal{D}$ (Wang et al., 4 Mar 2026).

CompressExperience(IndexedSummary, MemoryBlocks) supports two archival modes. In explicit authoring, the model writes db_content directly for a given db_index. In anchor-based extraction, the model provides start_anchor, mid_anchor, and end_anchor, and the system stores the exact span between the anchors after verifying the midpoint anchor. This mechanism is designed for exact preservation of long logs, code excerpts, or other verbatim artifacts without retyping them (Wang et al., 4 Mar 2026).

ReadExperience(index) dereferences a stable key and appends the retrieved content to the prompt. The agent therefore alternates between compact semantic control state and exact evidence recovery. A common pattern in the paper’s examples is to keep the summary focused on plans, verified facts, and index descriptions, while storing exact IDs, file excerpts, and environment observations in db_blocks, then retrieving them only when needed (Wang et al., 4 Mar 2026).

3. Reinforcement learning formulation and memory actions

MemexRL treats memory usage as part of the action space. The LLM emits thinking text $\mathbf{M} = [m_0, u, \mathbf{M}_{\mathrm{work}}],$0 and a tool call $\mathbf{M} = [m_0, u, \mathbf{M}_{\mathrm{work}}],$1 from a tool set

$\mathbf{M} = [m_0, u, \mathbf{M}_{\mathrm{work}}],$2

State is the current prompt state $\mathbf{M} = [m_0, u, \mathbf{M}_{\mathrm{work}}],$3, including the indexed summary, retrieved evidence, recent tool outputs, and a context status message reporting working-context token usage relative to a threshold (Wang et al., 4 Mar 2026).

The episode-level reward is

$\mathbf{M} = [m_0, u, \mathbf{M}_{\mathrm{work}}],$4

Here $\mathbf{M} = [m_0, u, \mathbf{M}_{\mathrm{work}}],$5 is the task-success signal, $\mathbf{M} = [m_0, u, \mathbf{M}_{\mathrm{work}}],$6 penalizes context overflow, $\mathbf{M} = [m_0, u, \mathbf{M}_{\mathrm{work}}],$7 penalizes redundant non-memory tool calls with identical arguments when no state-modifying operation intervenes, and $\mathbf{M} = [m_0, u, \mathbf{M}_{\mathrm{work}}],$8 penalizes malformed tool invocations (Wang et al., 4 Mar 2026).

The context penalty is defined as

$\mathbf{M} = [m_0, u, \mathbf{M}_{\mathrm{work}}],$9

where $m_0$0 is the working-context token length and $m_0$1 is the threshold. The redundancy penalty is

$m_0$2

and the format penalty is

$m_0$3

Operationally, this reward shaping encourages proactive compression before overflow, discourages re-running tools when archived evidence can be dereferenced, and enforces syntactically correct memory operations (Wang et al., 4 Mar 2026).

Optimization uses a GRPO-style PPO objective. For a group of rollouts, the normalized group-relative advantage is

$m_0$4

and token-level updates use a clipped surrogate with KL regularization:

$m_0$5

Because compression changes the prompt prefix discontinuously, trajectories are segmented at each compression boundary and flattened for training, while all segments share the same terminal reward. This provides credit assignment across compress decisions without keeping the full uncompressed history in a single training sequence (Wang et al., 4 Mar 2026).

4. Theoretical characterization

The paper formalizes the ideal Memex regime through the notion of a decision-sufficient indexed summary. Let $m_0$6 be the full message history, and let $m_0$7 be the induced Memex state at step $m_0$8. The indexed summary $m_0$9 is called $u$0-bounded decision-sufficient if there exist an index selector $u$1 and a decision function $u$2 such that $u$3 and

$u$4

This means that an optimal full-history policy can, in principle, be reproduced from the summary plus at most $u$5 retrieved blocks (Wang et al., 4 Mar 2026).

Under that assumption, the paper states that there exists a Memex policy $u$6 that conditions only on $u$7, uses at most $u$8 ReadExperience calls per step, and matches the expected return of the full-context optimal policy:

$u$9

This proposition does not claim that MemexRL always learns such a policy; it states that indexed summaries with bounded dereferencing can, in principle, preserve optimal decision quality (Wang et al., 4 Mar 2026).

The second result concerns bounded working context. If $\mathbf{M}_{\mathrm{work}}$0, $\mathbf{M}_{\mathrm{work}}$1, and each retrieved block satisfies $\mathbf{M}_{\mathrm{work}}$2, then

$\mathbf{M}_{\mathrm{work}}$3

Consequently, the compression ratio

$\mathbf{M}_{\mathrm{work}}$4

grows as the full history grows, while working-context size remains bounded. This suggests that effective in-context computation can remain bounded even as overall interaction history becomes arbitrarily long, provided summaries and dereferencing remain sufficiently disciplined (Wang et al., 4 Mar 2026).

5. Empirical behavior on long-horizon tasks

MemexRL is evaluated in a modified ALFWorld setting designed to stress long-horizon memory. The benchmark removes default lists of admissible commands, hides the initial observation that normally contains all location IDs, limits look to once per episode, and truncates the summary field of CompressExperience to 300 tokens. These changes make exact recovery of IDs and early observations dependent on proper memory archival and retrieval (Wang et al., 4 Mar 2026).

Training uses Qwen3-30B-A3B-Thinking-2507 as the base model, the Slime RL framework, INT4 rollout with quantization-aware training, a 32K context window, an 8K working-context penalty threshold, batch size 32, group size 8, learning rate $\mathbf{M}_{\mathrm{work}}$5, and a training set of 3553 tasks. The reported training curves show rollout task success increasing from about 20% to above 90%, while the total penalty improves from about $\mathbf{M}_{\mathrm{work}}$6 to about $\mathbf{M}_{\mathrm{work}}$7, indicating that the agent learns both task solving and proper memory-tool usage (Wang et al., 4 Mar 2026).

The main reported result is a jump in task success from 24.22% before RL to 85.61% after MemexRL. Peak working-context length drops from 16,934.46 tokens to 9,634.47 tokens. At the same time, mean CompressExperience calls per episode decrease from about 6.5 to about 3, while mean ReadExperience calls increase from about 1 to about 6–7. The behavioral interpretation given in the paper is that the trained agent compresses less frequently but more strategically, and increasingly uses explicit retrieval via ReadExperience instead of repeating environment interactions or keeping all detailed evidence in-context (Wang et al., 4 Mar 2026).

A recurrent qualitative pattern is that summaries remain compact and plan-oriented, whereas exact IDs, code excerpts, room descriptions, or object lists are stored in db_blocks such as ctx_locations or ctx_units_code_excerpt_001. Later actions then recover these exact blocks by stable name. This suggests that MemexRL is most effective when the summary carries semantic control state and the external database carries precision-critical evidence (Wang et al., 4 Mar 2026).

6. Position within the broader memory-RL design space

MemexRL belongs to a broader family of systems that treat memory operations as learned control rather than static storage, but its mechanism is distinct. MemRL formulates runtime, non-parametric reinforcement learning over an episodic memory of Intent–Experience–Utility triplets and uses a Two-Phase Retrieval mechanism that first filters by semantic relevance and then reranks by learned Q-values (Zhang et al., 6 Jan 2026). MemSearcher instead maintains a bounded textual memory inside the interaction loop and uses multi-context GRPO to jointly optimize reasoning, search strategies, and memory management (Yuan et al., 4 Nov 2025). MemCtrl attaches a trainable memory head $\mathbf{M}_{\mathrm{work}}$8 to an embodied MLLM and uses offline expert supervision or online RL to decide whether observations should be retained or discarded during exploration (Dorbala et al., 28 Jan 2026).

Other systems emphasize different parts of the same design space. APEX-EM uses non-parametric online learning over a structured procedural-episodic experience memory and hybrid retrieval combining semantic search, structural signature matching, and graph traversal (Banerjee et al., 31 Mar 2026). MemR$\mathbf{M}_{\mathrm{work}}$9 treats memory retrieval as a closed-loop controller with a router over retrieve, reflect, and answer actions, coordinated by a global evidence–gap tracker (Du et al., 23 Dec 2025). Retrospex does not inject memories into the LLM prompt at inference time; instead, it trains an offline RL critic on past trajectories and combines LLM action likelihoods with critic-estimated values through dynamic action rescoring (Xiang et al., 17 May 2025).

This comparison suggests a useful taxonomy. MemexRL is primarily an indexed evidence-preservation framework; MemRL is a value-aware episodic retrieval framework; MemSearcher is an end-to-end compact-memory search framework; MemCtrl is an online pruning controller; and APEX-EM is a structured experience replay framework. A plausible implication is that future systems may hybridize these approaches, for example by combining Memex-style stable indices with MemRL-style Q-values or APEX-EM-style structural retrieval.

7. Limitations and open directions

MemexRL’s strongest claims are conditional on the quality of the learned summaries, indices, and dereferencing policy. The theoretical results assume decision-sufficient summaries and bounded dereferencing, but the practical system can still fail when the agent archives the wrong artifact, writes an ambiguous index description, or neglects to retrieve evidence that remains necessary later. Over-retrieval can also push the working context back toward the threshold, while under-retrieval can force redundant environment interaction or unsupported reasoning (Wang et al., 4 Mar 2026).

The empirical setting is also specialized. The ALFWorld experiments use environment-specific prompting and memory conventions, including explicit instructions that exact location IDs and object IDs should be stored in db_blocks rather than in the truncated summary. This indicates that MemexRL, as currently instantiated, still depends on domain-specific interface design and reward shaping (Wang et al., 4 Mar 2026).

Open directions are strongly suggested by adjacent work. Memex-style stable indices could be combined with learned retrieval utilities as in MemRL (Zhang et al., 6 Jan 2026), with structured procedural traces as in APEX-EM (Banerjee et al., 31 Mar 2026), or with explicit validation-grounded controllers and delayed credit assignment as in PYTHALAB-MERA (Iscan, 8 May 2026). Another plausible direction is to combine Memex’s deterministic dereferencing with reflective or rubric-guided memory refinement, as in R$$\mathcal{D} : \mathsf{index} \mapsto \mathsf{content},$$0-Mem’s experience distillation for deep memory search (Wang et al., 13 May 2026). The broader research question is whether long-horizon agents will benefit most from compressing memory, valuing memory, structuring memory, or indexing memory—or from systems that do all four simultaneously.