Memex(RL): Scaling Long-Horizon LLM Agents via Indexed Experience Memory

Abstract: LLM agents are fundamentally bottlenecked by finite context windows on long-horizon tasks. As trajectories grow, retaining tool outputs and intermediate reasoning in-context quickly becomes infeasible: the working context becomes prohibitively long, eventually exceeds the context budget, and makes distant evidence harder to use even when it is still present. Existing solutions typically shorten context through truncation or running summaries, but these methods are fundamentally lossy because they compress or discard past evidence itself. We introduce Memex, an indexed experience memory mechanism that instead compresses context without discarding evidence. Memex maintains a compact working context consisting of concise structured summaries and stable indices, while storing full-fidelity underlying interactions in an external experience database under those indices. The agent can then decide when to dereference an index and recover the exact past evidence needed for the current subgoal. We optimize both write and read behaviors with our reinforcement learning framework MemexRL, using reward shaping tailored to indexed memory usage under a context budget, so the agent learns what to summarize, what to archive, how to index it, and when to retrieve it. This yields a substantially less lossy form of long-horizon memory than summary-only approaches. We further provide a theoretical analysis showing the potential of the Memex loop to preserve decision quality with bounded dereferencing while keeping effective in-context computation bounded as history grows. Empirically, on challenging long-horizon tasks, Memex agent trained with MemexRL improves task success while using a significantly smaller working context.

• The paper introduces Indexed Experience Memory, which pairs compact in-context summaries and stable pointers with a full-fidelity external archive so agents can recover decisive evidence instead of relying on lossy compression.
• The paper presents MemGRPO, an RL framework that learns when to compress and retrieve through segmented credit assignment and rewards for task success, context efficiency, redundancy, and tool-call correctness.
• The paper raises modified ALFWorld success from 24.22% to 85.61% and reduces peak working context from 16,934 to 9,634 tokens, while leaving generalization beyond one model and benchmark unresolved.

Overview

"Memex(RL): Scaling Long-Horizon LLM Agents via Indexed Experience Memory" (2603.04257) addresses the context-window bottleneck of long-horizon LLM agents. The authors, from the Center for Advanced AI at Accenture, argue that existing context-management approaches—truncation, rolling summaries, and learned compression schemes such as MEM1, MemAgent, Memory-R1, ReSum, and AgentFold—are fundamentally lossy: they compress or discard past evidence itself, so information that becomes decisive many steps later may be unrecoverable. Their alternative is Indexed Experience Memory, which compresses the working context without discarding evidence by pairing a compact in-context indexed summary with a full-fidelity external key–value archive that the agent can explicitly dereference.

The paper makes four contributions: the Indexed Experience Memory interface; MemGRPO, a GRPO-style RL framework that trains both memory write and read behaviors under a context budget; a theoretical analysis of when indexed summaries plus bounded dereferencing preserve decision quality; and an empirical study on a modified ALFWorld benchmark showing large gains in task success with reduced peak working context.

Indexed Experience Memory

The mechanism maintains two structures: a working context $\mathbf{M} = [m_0, u, \mathbf{M}_{\text{work}}]$ (system prompt and task instruction are never compressed) and an external experience store $$\mathcal{D}: \mathsf{index} \mapsto \mathsf{content}$$. The in-context state is an indexed summary $\sigma = (s, \mathcal{I})$, where $s$ is a compact actionable progress state and $\mathcal{I}$ binds semantic descriptions to stable indices into $\mathcal{D}$. Two memory operations are exposed as first-class tools alongside environment tools:

• CompressExperience archives $(\mathsf{index}, \mathsf{content})$ blocks into $\mathcal{D}$ and rewrites the working context to $[m_0, u, \sigma]$. Archived content supports either explicit authoring or anchor-based verbatim extraction using start/mid/end anchors, where the mid-anchor serves as a verification checkpoint against false matches.
• ReadExperience(index) dereferences an index and appends the exact archived block back into the working context.

This design makes memory access precise and auditable—an index points to a concrete artifact rather than an approximate semantic match—contrasting with similarity-based retrieval over pools of near-duplicate fragments, which the authors characterize as brittle for long-horizon tool use. A concrete before/after example shows six rounds of reasoning and tool outputs (~4,000 tokens) collapsed into a ~300-token summary plus six stable indices pointing to archived repo snapshots, code excerpts, and a reproducer script.

MemGRPO: learning memory behavior

Because the utility of a compression or index entry is revealed only when a downstream step needs to retrieve exactly that artifact, memory decisions pose a delayed-credit-assignment problem that prompt rules cannot enforce. MemGRPO treats memory operations as actions in the same space as environment tools and optimizes an episode-level return combining task success with three normalized penalties:

$$R = R_{\text{task}} - P_{\text{context}} - P_{\text{redundancy}} - P_{\text{format}}$$

where $$\mathcal{D}: \mathsf{index} \mapsto \mathsf{content}$$0 penalizes accumulated working-context overflow beyond threshold $$\mathcal{D}: \mathsf{index} \mapsto \mathsf{content}$$1, $$\mathcal{D}: \mathsf{index} \mapsto \mathsf{content}$$2 penalizes repeated identical tool calls without intervening state changes, and $$\mathcal{D}: \mathsf{index} \mapsto \mathsf{content}$$3 penalizes malformed tool invocations. Two training-design choices are notable. First, segmented trajectory processing splits trajectories at compression boundaries into independently tokenized segments that all share the identical terminal reward, so group-relative advantage estimation propagates credit to earlier compression decisions even though their effects appear only in later segments—compression changes the conditioning prefix, not merely its length. Second, soft triggering replaces hard forced-compression rules: a deterministic ContextStatus message reports token usage each step, and the agent learns when to compress based on task semantics rather than arbitrary token counts, e.g., deferring compression if completion is imminent. Training warm-starts from supervised demonstrations of well-formed summaries before RL refinement.

Theoretical analysis

The analysis characterizes sufficient conditions rather than claiming the method always learns such summaries. Under a $$\mathcal{D}: \mathsf{index} \mapsto \mathsf{content}$$4-bounded decision-sufficiency assumption—that there exist an index selector $$\mathcal{D}: \mathsf{index} \mapsto \mathsf{content}$$5 with $$\mathcal{D}: \mathsf{index} \mapsto \mathsf{content}$$6 and a decision function $$\mathcal{D}: \mathsf{index} \mapsto \mathsf{content}$$7 reproducing the optimal full-context policy—the first proposition shows a Memex policy conditioning only on $$\mathcal{D}: \mathsf{index} \mapsto \mathsf{content}$$8 with at most $$\mathcal{D}: \mathsf{index} \mapsto \mathsf{content}$$9 dereferences per step achieves $\sigma = (s, \mathcal{I})$0: no loss in decision quality relative to conditioning on the entire history. The second proposition shows that if $\sigma = (s, \mathcal{I})$1, $\sigma = (s, \mathcal{I})$2, and each block satisfies $\sigma = (s, \mathcal{I})$3, then the working context is uniformly bounded by $\sigma = (s, \mathcal{I})$4, so the compression ratio grows without bound as history accumulates. Both proofs are constructive and straightforward; their force lies entirely in the decision-sufficiency assumption, which the empirical section does not verify directly.

Empirical results

Experiments use Qwen3-30B-A3B-Thinking-2507 (MoE, ~3B active parameters), trained with Slime using INT4 quantized rollouts with QAT-style BF16 gradient accumulation, truncated importance sampling (clip ratio 2.0), and a token-in-token-out loop to avoid re-tokenization mismatch. The evaluation environment is a deliberately harder variant of ALFWorld with hidden admissible commands, hidden initial observations, look restricted to once per episode (forcing reliance on ReadExperience for location IDs thereafter), and 300-token summary truncation forcing detailed storage in db_blocks. Training uses 3,533 tasks, a 32K context window, an 8K penalty threshold, batch size 32, and GRPO group size 8.

The headline results are strong. On evaluation, Memex(RL) raises task success from 24.22% to 85.61% (a >3.5× improvement) while reducing peak working context from 16,934 to 9,634 tokens (~43% reduction), approaching but not meeting the 8K training threshold. During training, rollout success climbs from roughly 20% to over 90% while total penalty improves from about −0.4 to −0.1. Behavioral analysis supports the intended operating mode: mean CompressExperience calls per episode fall from ~6.5 to ~3 while ReadExperience calls rise from ~1 to ~6–7, indicating the learned policy compresses more selectively and increasingly relies on explicit retrieval rather than repeated context rewriting or redundant tool re-execution. The authors note this shows the gain is not merely more aggressive compression.

Limitations and open questions

Several limitations deserve plain statement. Evaluation rests on a single model family and a single, albeit modified, environment; ALFWorld tasks, even hardened, involve modest state spaces compared with open-ended coding or research workflows, and generalization to settings where decision-sufficiency may fail is untested. The theoretical guarantees hold only under the $\sigma = (s, \mathcal{I})$5-bounded decision-sufficiency assumption, and the paper provides no empirical measurement of whether learned summaries satisfy it. Peak working context after training (9,634 tokens) still exceeds the 8K penalty threshold, suggesting imperfect learned timing. The reward depends on hand-designed penalties whose weights could interact with task difficulty in unstudied ways, and the soft-triggering mechanism's advantage over hard thresholds is asserted rather than ablated. Whether indexed memory scales to hundreds-of-steps horizons with much larger archives—and how index quality degrades as the store grows—remains an open question the paper leaves unanswered.

Conclusion

Memex(RL) reframes agent memory as indexed experience management: keep a compact pointer-heavy control state in context, archive full-fidelity artifacts externally, and learn both writing and dereferencing via RL with memory-aware reward shaping and segmented credit assignment. The combination yields a substantially less lossy memory regime than summary-only compression, with concrete gains of +61 percentage points in task success and ~43% lower peak context on modified ALFWorld. The results position summarization, indexing, and dereferencing as a complementary scaling axis for persistent LLM agents, contingent on the decision-sufficiency conditions the theory formalizes but practice has yet to validate broadly.