From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory

Abstract: LLM agents are increasingly deployed in long-running settings where improving through experience at test time becomes important. A common approach is to update an explicit memory after each interaction to guide future decisions. However, most existing methods rely on hand-designed prompting rules, making it difficult to align memory updates with downstream objectives over multi-step horizons consistently. We propose MemoPilot, a plug-in memory copilot that explicitly trains the memory update process to improve a frozen LLM's performance across sequential interactions. We formulate memory updating as a multi-turn decision problem and optimize it end-to-end with multi-turn GRPO. Our training recipe introduces (i) a turn-wise reward signal and (ii) a context-independent, turn-level advantage estimation across rollouts, enabling finer-grained credit assignment and more stable training in multi-turn settings. We evaluate MemoPilot on two testbeds: multi-round Rock-Paper-Scissors (RPS) and Limit Texas Hold'em (LHE). Across both environments, MemoPilot substantially improves test-time learning of a frozen player over strong baselines, ranking first in Elo ratings on both games (1762 on LHE and 1590 on RPS) and outperforming all baseline memory methods and proprietary models, including DeepSeek-V3.2.

• The paper shows that an RL-trained memory copilot enhances test-time learning without modifying the frozen LLM agent.
• The methodology uses multi-turn GRPO to optimize memory updates, leading to significant Elo score improvements in RPS and LHE.
• The approach enables plug-and-play adaptation for LLMs, offering practical deployment benefits and advancing theoretical insights in online learning.

Enhancing Test-Time Learning in LLM Agents via Reinforcement-Learned Memory Updates

Introduction and Problem Motivation

LLMs deployed as agents in interactive, sequential settings require adaptive mechanisms to effectively leverage past experiences during test time. The observed bottleneck is the inability of such agents to consistently improve behaviors across sequential tasks without explicit online adaptation mechanisms. While prior work has attempted to inject explicit memory, most integrate hand-crafted or prompt-based update heuristics, yielding suboptimal adaptation and limited alignment between memory evolution and downstream utility.

Methodology: RL-Driven Memory Copilot

The central contribution of the paper is a plug-in memory copilot architecture trained via multi-turn reinforcement learning, explicitly optimizing memory update policies for downstream performance. Rather than modifying the frozen LLM player $\pi$, the framework revolves around a separate, trainable memory model $G_\theta$. After each sequential interaction, $G_\theta$ updates a structured memory state $m_t$, which the frozen policy conditions on in subsequent games.

Figure 1: The memory copilot $G_\theta$ updates memory $m_t$ after observing interaction $e_t$, guiding the fixed player $\pi$ in future games; all cross-game learning is funneled through memory, without modifying $\pi$.

Memory updating is framed as a Markov Decision Process, where the state comprises the last trajectory and previous memory, and the action is a new textual memory state. Rewards are given per game, enabling direct alignment of memory updates with episodic outcomes. Training employs multi-turn Group Relative Policy Optimization (GRPO), introducing a one-step, turn-level advantage estimation to mitigate high variance credit assignment and stabilize optimization under stochastic environments.

Figure 2: Multi-turn GRPO processes parallel rollouts, attributing the next game's outcome to the most recent memory update via one-step proxy rewards, and computes group-normalized per-turn advantages for token-level optimization.

The memory format is structured into three tiers: diagnostic reasoning, maintained hypothesis state, and actionable guidance, enabling efficient hypothesis accumulation, verification, and downstream control.

Experimental Setup

The framework is evaluated on two strategic domains:

• Multi-round Rock–Paper–Scissors (RPS): Players interact over six rounds per game, with designed opponent strategy pools varying in difficulty and behavioral mechanisms.
• Limit Texas Hold'em (LHE): Partial information with explicit action spaces and natural stochasticity, providing a demanding testbed for memory-driven long-horizon adaptation.

Opponent pools are meticulously constructed for diversity and difficulty calibration, with Elo rating estimates confirming the pools span a broad competence range.

Figure 3: RPS opponent pool Elo ratings establish a broad, relatively uniform difficulty spectrum.

Figure 4: LHE opponent pool Elo ratings display similarly controlled difficulty spread.

Test-time learning is measured as cumulative per-game performance over consecutive games, and methods are compared under fixed memory budgets.

Main Results

Performance Improvements

The trained memory copilot achieves substantial gains over all baselines on both RPS and LHE. On RPS@5, it achieves a mean score of 3.28, versus 0.43 with No Memory and 1.64 with the best prior memory method. On LHE@5, the gain is similarly marked at 2.03, far exceeding alternatives. Notably, the memory copilot ranks first in Elo ratings among all compared methods: 1590 (RPS), 1762 (LHE).

Figure 5: The memory copilot achieves the highest Elo ranking versus all evaluated memory architectures on both RPS and LHE.

Prompt- or history-based memory updates are dramatically less effective; naively providing longer unfiltered histories impairs learning, highlighting the criticality of selective, distilled, and reward-aligned memory updates.

Learning Dynamics and Generalization

Adaptation is efficient: within a few games, cumulative performance rises sharply and plateaus at significantly higher levels than baselines.

Figure 6: In LHE, the memory model (left) consistently outperforms all baselines in cumulative performance, and generalizes to a stronger frozen player (right) with negligible degradation.

Figure 7: The same rapid learning trend holds in RPS, with superior cumulative scores and strong generalization to larger, stronger frozen players.

Importantly, the trained memory module exhibits robust zero-shot generalization: when paired with a much larger frozen player ($\pi$), it yields comparable or improved performance relative to paired training.

Ablations and Analysis

• Reward attribution: Per-turn, next-game rewards for memory policy optimization lead to superior stability and downstream return relative to cumulative (long-horizon) return assignments, mitigating noise due to stochastic environment transitions.
• Structured memory vs. free-form: Structured, tiered memory formats provide an inductive bias for effective hypothesis management and downstream guidance, yielding higher scores than free-form, RL-trained variants.
• Cross-opponent stream robustness: The system maintains effective adaptation even when opponent strategies switch mid-stream, with memory policy rapidly adjusting and revising its maintained hypotheses.
• Failure modes: Performance declines with highly non-stationary or adaptive opponents, as evidence accumulation for robust hypotheses is at odds with the need for rapid revision.

  Figure 8: Longer training rollouts during memory policy optimization result in improved stability and long-horizon test-time gains, as reflected in cumulative performance curves.

Practical and Theoretical Implications

The results demonstrate that end-to-end RL training for agent memory policies dramatically raises the test-time learning ceiling of frozen LLM-based agents. By eliminating reliance on hand-tuned or prompting-based memory evolution and focusing all cross-game adaptation into an external, compositional memory module, the framework enables plug-and-play test-time learning with closed-source or large-scale frozen models.

This has substantial implications:

• Practical deployment: Existing LLM agents, even when frozen or proprietary, can acquire robust online adaptation capability via externally trained memory copilot modules, avoiding costly model finetuning or re-deployment cycles.
• Theoretical value: The approach decouples RL-driven adaptation from policy parameters, centering research on optimal external experience compression and retrieval strategies, and advancing understanding of indirect online learning dynamics in language agents.

Limitations and Future Directions

Several limitations are acknowledged. The method assumes sufficiently informative reward and experience signals; extreme reward sparsity or low-information tasks may require auxiliary signals. Fixed memory budgets limit accumulated context, potentially requiring chunking or further abstraction for very long interaction streams. Rapidly changing or adaptive environments expose tradeoffs between robust hypothesis maintenance and flexible revision; meta-learning or change-point detection strategies may be mandatory for further robustness.

Future avenues include:

• Integrating denser auxiliary feedback signals for challenging, sparse environments.
• Investigating meta-learning enhancements for more efficient adaptation to non-stationary distributions.
• Scaling to complex, multi-agent, or open-ended scenarios to evaluate emergent, cooperative, or competitive agent memory behaviors.

Conclusion

The presented memory copilot framework establishes a rigorous, RL-based solution for enabling effective test-time learning in LLM agents, realizing substantial empirical improvements over existing methods. By treating memory update as a trainable, reward-aligned process, the approach unlocks rapid and robust adaptation capabilities for frozen agents in stochastic, sequential domains, setting a new standard for research and practical development in this area.