Meta-RL Induces Exploration in Language Agents (2512.16848v1)

Abstract: Reinforcement learning (RL) has enabled the training of LLM agents to interact with the environment and to solve multi-turn long-horizon tasks. However, the RL-trained agents often struggle in tasks that require active exploration and fail to efficiently adapt from trial-and-error experiences. In this paper, we present LaMer, a general Meta-RL framework that enables LLM agents to actively explore and learn from the environment feedback at test time. LaMer consists of two key components: (i) a cross-episode training framework to encourage exploration and long-term rewards optimization; and (ii) in-context policy adaptation via reflection, allowing the agent to adapt their policy from task feedback signal without gradient update. Experiments across diverse environments show that LaMer significantly improves performance over RL baselines, with 11%, 14%, and 19% performance gains on Sokoban, MineSweeper and Webshop, respectively. Moreover, LaMer also demonstrates better generalization to more challenging or previously unseen tasks compared to the RL-trained agents. Overall, our results demonstrate that Meta-RL provides a principled approach to induce exploration in language agents, enabling more robust adaptation to novel environments through learned exploration strategies.

• The paper introduces LAMER, a meta-reinforcement learning framework that enhances exploration and rapid adaptation through cross-episode credit assignment and textual reflection.
• It employs in-context adaptation by leveraging reflective practices and discounted cross-episode rewards to balance exploration and exploitation effectively.
• Empirical results demonstrate significant performance improvements on tasks like Sokoban and MineSweeper, showcasing robust generalization and adaptability in challenging scenarios.

Meta-RL Induced Exploration in Language Agents: Summary and Implications

Introduction

The paper "Meta-RL Induces Exploration in Language Agents" (2512.16848) introduces LAMER, a meta-reinforcement learning (Meta-RL) framework to train LLM agents for improved exploration and adaptation in interactive, multi-turn environments. The motivation arises from limitations observed in standard RL-trained LLM agents in long-horizon decision-making: they typically under-explore at test time and fail to exploit trial-and-error feedback for rapid adaptation. LAMER is presented as a mechanism to operationalize active exploration and in-context policy adaptation, thereby bridging the gap between offline policy imitation and high-frequency online adaptation in sequential tasks.

Methodology

Meta-RL Framework

LAMER formalizes agent-environment interactions as multi-episode Markov Decision Processes (MDPs), redefining exploration-exploitation balancing over episode sequences. Unlike conventional RL where episodes are independent, LAMER enforces cross-episode linkage by:

• Cross-episode training: Agents are trained on consecutive episodes, leveraging early exploratory episodes for environment knowledge acquisition and subsequent exploitation.
• Discounted cross-episode reward: Trajectory-level discounted returns ($\gamma_{traj}$) prioritize either early or late episode rewards, tuning the exploration-exploitation trade-off.
• In-context adaptation via reflection: Policy adaptation is performed in-context through textual reflections, enabling agents to modify behavior based on previous outcomes without gradient updates at test time.

Optimization and Implementation

LAMER's meta-objective is optimized using policy gradient methods compatible with standard RL algorithms (e.g., PPO, GRPO, GiGPO). The framework is validated on Qwen3-4B and Llama3.1-8B-Instruct architectures, demonstrating model-agnostic applicability. Key innovations include explicit credit assignment across episodes and flexible in-context memory buffers for adaptation, with efficient ablations on memory content (trajectory-only, reflection-only, combined).

Empirical Results

Performance

LAMER consistently outperforms RL and prompting-based baselines across a range of environments (Sokoban, MineSweeper, Webshop, ALFWorld):

• Sokoban pass@3: 55.9% (LAMER) vs. 44.1% (best RL) vs. 12.9% (prompting)
• MineSweeper pass@3: 74.4% (LAMER) vs. 55.1% (best RL)
• Webshop pass@3: 89.1% (LAMER) vs. 75.2% (best RL)

Test-time scaling behavior is markedly superior under LAMER, exhibiting higher gains across attempts (pass@1→pass@3).

Exploration and Diversity

Trajectory entropy analyses confirm that Meta-RL agents preserve diversity from base models while improving success rates, achieving a more favorable exploration-exploitation equilibrium than RL agents, which tend toward determinism.

Generalization

LAMER maintains strong performance under increased task difficulty (e.g., more boxes/mines in Sokoban/MineSweeper) and out-of-distribution generalization (ALFWorld), registering up to 10% improvement in hardest settings. Out-of-distribution tasks present substantial gains: on ALFWorld's Cool and Pick2 tasks, LAMER achieves 23% and 14% higher success rates than RL.

Ablations

• Discount factor tuning ($\gamma_{traj}$): Intermediate values generally yield optimal performance; higher values foster exploration but may reduce exploitation efficacy depending on environment.
• Inter-episode memory: Reflection-only memory surpasses combined configurations, suggesting concise contextual feedback enhances adaptation.

Training Efficiency

LAMER matches RL baselines in total trajectory consumption per gradient update, but sequential episode dependency limits parallelism, incurring approximately 2x training time. Asynchronous rollout strategies are proposed for future efficiency improvements.

Theoretical and Practical Implications

Theoretical Advances

LAMER demonstrates that Meta-RL principles, previously established in non-agentic settings, are directly translatable to agentic LLM training, especially when leveraging in-context adaptation. The cross-episode credit assignment and reflective adaptation represent advances in unifying meta-learning with reinforcement learning for language agents. The work underscores the importance of explicit exploration strategies, as opposed to mere imitation or single-episode exploitation, in achieving robust generalization and rapid adaptation.

Practical Impact

LAMER enables LLM agents to perform autonomously in dynamic environments requiring exploration, reasoning, and iterative learning. This is critical for applications in embodied AI, web navigation, and complex planning where trial-and-error and environmental feedback are intrinsic. The framework is model-agnostic, does not require gradient updates at test time, and supports generalization outside the training distribution, thus paving the way for more resilient, scalable agent designs.

Future Directions

The sequential dependency inherent in LAMER suggests avenues for parallelization and asynchronous training. Combining Meta-RL frameworks with advanced advantage estimation or more expressive reasoning models could further enhance agent performance. The observed generalization opens prospects for universalist agents capable of adapting to entirely novel domains, including multimodal or non-verbal environments. Integration with richer self-reflection protocols and more efficient credit assignment mechanisms warrants exploration.

Conclusion

LAMER—by leveraging meta-reinforcement learning and in-context policy adaptation—empirically and theoretically advances the training of exploratory, adaptive LLM agents. It delivers quantifiable performance improvements, superior exploration, and broader generalization, illuminating a path toward building versatile, autonomous language agents capable of robust decision-making in challenging, interactive environments (2512.16848).