MemoPilot: Memory Copilot for Frozen LLMs

• MemoPilot is a memory copilot system that trains a separate memory model to update a bounded textual memory for enhancing a frozen LLM's performance.
• It employs multi-turn Group Relative Policy Optimization with one-step rewards to refine memory updates, yielding significant improvements in strategic tasks like RPS and Texas Hold’em.
• The structured 3-tier memory format—dividing reasoning, knowledge base, and final strategy—ensures that actionable insights are efficiently leveraged during sequential interactions.

MemoPilot is a plug-in memory copilot for LLM agents that explicitly trains the memory update process to improve a frozen LLM’s performance across sequential interactions. Instead of adapting the main agent, it treats memory itself as the adaptive object: after each interaction, a separate memory model updates a bounded textual memory that is then passed into the next interaction. The system formulates memory updating as a multi-turn decision problem, optimizes it end-to-end with multi-turn GRPO, and evaluates the resulting test-time learning behavior primarily in repeated strategic environments such as multi-round Rock–Paper–Scissors and Limit Texas Hold’em (Cai et al., 7 Jun 2026).

1. Conceptual framing and formal setting

MemoPilot is designed for long-running settings in which an LLM agent repeatedly interacts with an environment or user and is expected to improve at test time by learning from experience. Its defining constraint is that the main agent LLM, termed the “player,” is frozen: its parameters never change. The only component that evolves across interactions is an explicit textual memory, bounded to 512 tokens in the reported experiments, that summarizes what has been learned so far (Cai et al., 7 Jun 2026).

The memory update process is formalized as

$$m_t = G_\theta(e_t, m_{t-1}), \quad m_0 = \emptyset,$$

where $e_t$ is the current game trajectory and $G_\theta$ is the trainable memory model. On the next game, the frozen player $\pi$ receives $m_t$ as part of its prompt. MemoPilot therefore treats memory updating, rather than policy updating, as the locus of test-time adaptation (Cai et al., 7 Jun 2026).

The paper’s central motivation is that most prior test-time learning systems update memory with hand-designed prompting rules—“write a reflection,” “summarize errors,” or “extract skills”—without optimizing those updates for downstream long-horizon reward. MemoPilot instead casts memory updating as an MDP $$\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R})$$, where the state is

$s_t = (e_t, m_{t-1}),$

the action is the new memory $m_t$, and the reward is the outcome of the current game. An episode consists of $T$ games, with return

$R(\tau) = \sum_{t=1}^{T-1} r_{t+1},$

and the objective is to maximize expected return over opponent strategies drawn from a strategy pool $e_t$0 (Cai et al., 7 Jun 2026).

A common misconception is that MemoPilot fine-tunes the player. It does not. The player remains frozen throughout; only the external memory copilot $e_t$1 is trained. This architectural decoupling is central to the method’s claim that test-time learning can be improved without modifying the core LLM (Cai et al., 7 Jun 2026).

2. Architecture and memory format

MemoPilot is explicitly decoupled into a frozen player $e_t$2 and a trainable memory copilot $e_t$3. In the reported experiments, the player can be Qwen2.5-14B-Instruct or Qwen3-235B-A22B, while the memory copilot is a separate Qwen2.5-14B-Instruct model fine-tuned via RL. The copilot observes the full game trajectory $e_t$4 together with the previous memory $e_t$5, produces a new memory $e_t$6 in a structured textual format, and never interacts with the environment directly; its only influence on behavior is through the player’s prompt (Cai et al., 7 Jun 2026).

The memory is not free-form notes. MemoPilot uses a fixed 3-tier cheatsheet format, wrapped in a <cheatsheet> block, with three sections:

• <reasoning>: a diagnostic “Think” layer containing analysis of opponent behavior and the last game outcome.
• <knowledge_base>: a “State” layer containing persistent hypotheses, evidence, observation counts, success counts, and a status field such as Hypothesis, Verified, or Confirmed.
• <final_strategy_prompt>: an “Action” layer containing concise, actionable guidance for the next game.

Only the <final_strategy_prompt> is passed into the player’s context as explicit guidance. The full cheatsheet is preserved and fed back to MemoPilot on the next turn, allowing the copilot to maintain and revise its own internal state across games (Cai et al., 7 Jun 2026).

This design separates persistent evidential bookkeeping from executable advice. The paper’s analysis argues that MemoPilot’s learned memory behaves less like a passive note store and more like a policy over how to exploit the opponent: the system maintains hypotheses, updates them with evidence, and distills them into instructions that the frozen player can immediately act on. A plausible implication is that the utility of the memory lies not only in its content but in its formatting and role separation (Cai et al., 7 Jun 2026).

3. Multi-turn GRPO and credit assignment

MemoPilot trains the memory copilot with a multi-turn variant of Group Relative Policy Optimization. For each training iteration, the method samples an opponent strategy, generates $e_t$7 parallel episodes of length $e_t$8, and records the sequence of memory generations $e_t$9 for each rollout $G_\theta$0. Each episode begins from $G_\theta$1, and after each game the frozen player’s trajectory and reward are observed before the memory model produces the next memory (Cai et al., 7 Jun 2026).

The paper’s key credit-assignment move is a turn-wise one-step reward:

$G_\theta$2

A memory update at turn $G_\theta$3 is therefore credited only with the outcome of the next game, not the cumulative return over all later games. The rationale is that cumulative returns in games such as Limit Texas Hold’em introduce large variance from unrelated future randomness. Empirically, the paper reports that training with cumulative reward yields LHE@5 $G_\theta$4, whereas one-step reward yields $G_\theta$5, with cumulative-reward training showing instability and collapse (Cai et al., 7 Jun 2026).

For each turn $G_\theta$6, MemoPilot computes a context-independent, turn-level advantage across the group of rollouts:

$G_\theta$7

and applies that same turn-level advantage to all tokens $G_\theta$8 of the memory $G_\theta$9. This avoids learning a separate value function and yields finer-grained supervision for memory writing in multi-turn settings (Cai et al., 7 Jun 2026).

At the token level, MemoPilot uses a PPO-style clipped surrogate objective with importance weights

$\pi$0

and optimizes the corresponding clipped objective over group, turn, and token indices. In the LHE setting, the method additionally uses the same random seed across rollouts in a GRPO group so that compared episodes see the same card sequence at each turn, which reduces variance when comparing returns (Cai et al., 7 Jun 2026).

This training setup distinguishes MemoPilot from prompt-engineered memory writers. The memory-writing rule is not merely instructed; it is optimized against downstream performance.

4. Benchmarks, baselines, and empirical results

MemoPilot is evaluated on two synthetic strategic environments—multi-round Rock–Paper–Scissors from TextArena and 2-player Limit Texas Hold’em from RLCard—and on StreamBench, which includes sequential CoSQL and DS-1000 tasks. The paper also reports Elo rankings based on held-out test opponents and measures transfer from the training player Qwen2.5-14B-Instruct to the much stronger Qwen3-235B-A22B (Cai et al., 7 Jun 2026).

Before the main result table, the paper organizes baselines into several categories: No Memory, Full History, Ground-Truth Opponent Strategy, Human-Written Counter-Strategy, prior prompt-based memory methods such as Reflexion, ExpeL, MemoryBank, AWM, and ReasoningBank, prompt-based memory writers using different LLMs including DeepSeek-V3.2 and Gemini-3.0-Flash, and finally RL-trained “Memory w/ MemoPilot” (Cai et al., 7 Jun 2026).

The principal quantitative outcomes are as follows:

Setting	Baseline or comparator	MemoPilot
RPS@5, Qwen2.5-14B-Instruct	No Memory: 0.43	3.28
LHE@5, Qwen2.5-14B-Instruct	No Memory: -1.36	2.03
RPS@5, Qwen3-235B-A22B	No Memory: 0.44	3.27
LHE@5, Qwen3-235B-A22B	No Memory: -1.46	1.31
Elo ranking	—	1590 on RPS, 1762 on LHE
StreamBench overall pass@4	No Memory: CoSQL 69.5, DS-1000 50.0	CoSQL 73.5, DS-1000 56.3

Several ablations sharpen the interpretation. A 3-tier memory format without RL yields LHE@5 $\pi$1, whereas free-form memory with RL reaches $\pi$2, and full 3-tier memory with RL reaches $\pi$3. RL is therefore reported as crucial, and the structured memory format adds value on top of RL. Likewise, one-step reward outperforms cumulative reward, and longer-horizon training with 5-turn rollouts produces better cumulative performance than 2-turn training, especially after the first few games (Cai et al., 7 Jun 2026).

The paper also addresses a common assumption that “correct descriptive information” should be sufficient. It reports that ground-truth opponent strategy and human-written counter-strategy both underperform MemoPilot; moreover, taking MemoPilot’s own memories and rewriting them into polished English with DeepSeek-V3.2 reduces performance from RPS@5 $\pi$4 to $\pi$5 and from LHE@5 $\pi$6 to $\pi$7. This suggests that the learned memory’s behavioral usefulness depends on more than factual correctness alone (Cai et al., 7 Jun 2026).

5. Relation to adjacent memory paradigms

MemoPilot belongs to a broader family of LLM memory systems, but its defining feature is RL over memory updates for a frozen player. Closely related work varies along three axes: whether memory is internal or external, whether it is optimized by supervision or RL, and whether it is intended for embodied agents, dialogue, prompt optimization, or human recall.

MemPO likewise treats memory as part of the policy’s action space, introducing <mem> summaries and a dedicated memory-effectiveness reward, but it differs structurally because the policy model itself autonomously summarizes and manages memory during interaction, whereas MemoPilot keeps the player frozen and trains an external copilot (Li et al., 28 Feb 2026). MemoBrain also separates memory from execution, but it does so through an executive memory model that constructs a dependency-aware graph of thoughts and manages context with fold and flush operations rather than by updating a bounded textual cheatsheet between games (Qian et al., 12 Jan 2026). MemoChat, by contrast, uses a memorize–retrieve–response cycle with self-composed memos for long-range open-domain conversation, emphasizing instruction tuning over memo writing, memo retrieval, and memo-conditioned response rather than RL over multi-turn returns (Lu et al., 2023).

Other systems extend memory beyond MemoPilot’s game setting. MemAPO reconceptualizes prompt optimization as continual memory formation using dual memories of strategy templates and error patterns (Liang et al., 23 Mar 2026). Mem-PAL’s H$\pi$8Memory combines device logs and dialogues in a hierarchical and heterogeneous memory for personalized service-oriented interaction (Huang et al., 17 Nov 2025). T-Mem targets long-term conversational recall with descriptive and associative triggers generated at write time (Guo et al., 13 Jun 2026). MemoCue focuses on helping humans recall their own memories by transforming vague queries into cue-rich ones through a 5W Recall Map, 15 recall strategies, and MCTS-guided cue generation (Zhao et al., 31 Jul 2025).

Several papers explicitly use “MemoPilot-like” as a design reference when discussing memory-centric systems. MemCompiler describes many agent frameworks and “a lot of ‘MemoPilot’-like designs” as instances of Ahead-of-time Monolithic Memory Injection (Ding et al., 8 May 2026). MemAPO, MemoBrain, Memento, MemPO, memorAIs, and “AI-native Memory 2.0: Second Me” each present design lessons for “MemoPilot-like” systems in domains ranging from embodied agents to AR assistants, medication reminders, and AI-native personal memory (Liang et al., 23 Mar 2026, Qian et al., 12 Jan 2026, Kim et al., 24 Jan 2026, Shaveet et al., 2023, Wei et al., 11 Mar 2025). This suggests that MemoPilot has become, at least in adjacent work, a shorthand for a memory copilot that mediates between persistent experience and task-time behavior.

6. Limitations, misconceptions, and broader significance

The paper explicitly notes three main limitations. First, MemoPilot depends on informative experience and a reasonably dense scalar reward; extremely sparse or low-information feedback may limit learning. Second, memory capacity is bounded to 512 tokens in the experiments, so very long single-task trajectories may require summarization. Third, non-stationary or adaptive opponents can cause the system’s hypothesize-and-verify loop to become stale, especially when conditions change every two games or when opponents themselves adapt based on memory (Cai et al., 7 Jun 2026).

A second misconception is that larger prompt-based memory writers should obviate RL. The reported results do not support that view. Prompt-based “Memory w/ DeepSeek-V3.2” improves RPS@5 to 1.64 for Qwen2.5-14B-Instruct but still trails MemoPilot’s 3.28, and on LHE it remains negative at -0.78 versus MemoPilot’s 2.03. Likewise, Full History often underperforms No Memory. The paper’s interpretation is that hand-written or prompt-induced memory rules are not reliably aligned with downstream objectives over multi-step horizons (Cai et al., 7 Jun 2026).

The broader significance of MemoPilot lies in its reframing of test-time learning. Rather than updating the main policy, the method learns a memory-writing policy that maintains evolving hypotheses, evidence counts, and executable prompts for a frozen agent. The paper characterizes this as having a meta-learning flavor: given trajectories and outcomes, MemoPilot learns generally useful ways to infer opponent policies and explain them to the player in a usable form. Its transfer to a much stronger player and its gains on StreamBench suggest that the learned memory policy is not merely overfitting to one backbone or one game (Cai et al., 7 Jun 2026).

In that sense, MemoPilot occupies a specific position in the emerging memory-augmented-agent literature. It does not treat memory as passive storage, nor as a generic retrieval layer, nor as a static persona sheet. It treats memory updating as a trainable control problem over sequential interactions, and shows that RL over memory can measurably improve a frozen LLM’s behavior across games and related long-horizon tasks (Cai et al., 7 Jun 2026).