ProAct: Agentic Lookahead in Interactive Environments

Abstract: Existing LLM agents struggle in interactive environments requiring long-horizon planning, primarily due to compounding errors when simulating future states. To address this, we propose ProAct, a framework that enables agents to internalize accurate lookahead reasoning through a two-stage training paradigm. First, we introduce Grounded LookAhead Distillation (GLAD), where the agent undergoes supervised fine-tuning on trajectories derived from environment-based search. By compressing complex search trees into concise, causal reasoning chains, the agent learns the logic of foresight without the computational overhead of inference-time search. Second, to further refine decision accuracy, we propose the Monte-Carlo Critic (MC-Critic), a plug-and-play auxiliary value estimator designed to enhance policy-gradient algorithms like PPO and GRPO. By leveraging lightweight environment rollouts to calibrate value estimates, MC-Critic provides a low-variance signal that facilitates stable policy optimization without relying on expensive model-based value approximation. Experiments on both stochastic (e.g., 2048) and deterministic (e.g., Sokoban) environments demonstrate that ProAct significantly improves planning accuracy. Notably, a 4B parameter model trained with ProAct outperforms all open-source baselines and rivals state-of-the-art closed-source models, while demonstrating robust generalization to unseen environments. The codes and models are available at https://github.com/GreatX3/ProAct

• The paper presents a two-stage framework that combines GLAD for efficient lookahead reasoning with MC-Critic for low-variance value estimation in reinforcement learning.
• It leverages environment-grounded MCTS to compress extensive search trees into concise, causal reasoning chains, reducing hallucinations and context drift.
• Empirical evaluations on 2048 and Sokoban demonstrate state-of-the-art performance, highlighting robust planning in both stochastic and deterministic settings.

ProAct: A Two-Stage Paradigm for Agentic Lookahead in Interactive Environments

Introduction and Motivation

Lookahead reasoning is essential for autonomous agents operating in interactive, long-horizon environments where actions have compounding effects on future states. LLM agents, while exhibiting robust generalization in static and short-turn tasks, are limited by simulation drift and compounding errors when tasked with multi-step planning. These weaknesses are pronounced in stochastic or deterministic environments such as 2048 and Sokoban, where hallucinated rollouts quickly degrade decision quality. ProAct directly addresses these gaps via a unified two-stage training paradigm: (1) Grounded Lookahead Distillation (GLAD) for supervised learning of compressed, environment-grounded reasoning; and (2) a Monte-Carlo Critic (MC-Critic) for plug-and-play, low-variance value estimation during reinforcement learning (RL) fine-tuning.

Figure 1: Overview of ProAct's two-stage framework: GLAD compresses MCTS search into reasoning chains; MC-Critic provides efficient, low-variance value estimates to stabilize RL training.

The ProAct Framework

Stage 1: Grounded Lookahead Distillation (GLAD)

GLAD formalizes the lookahead supervision process by directly probing the environment (e.g., via MCTS) to generate ground-truth search trajectories that reveal both optimal and suboptimal decision branches. Instead of naively cloning extended search trees—which is inefficient and misaligned with LLM priors—ProAct compresses these trees into concise, causal reasoning chains. This distillation process preserves explicit multi-step counterfactual reasoning, imposes natural language constraints, and actively conveys trade-off analyses and risk assessments inferred from real environment feedback. The supervised fine-tuning thus teaches the agent a robust world model of environment dynamics, grounding the agent’s future predictions in verifiable outcomes.

Figure 2: ProAct’s architecture: Stage 1 (GLAD) distills MCTS into high-quality, explicit reasoning chains; Stage 2 integrates RL with MC-Critic for accurate, stable policy optimization.

GLAD thus mitigates hallucination and context-drift, endowing agents with internalized lookahead capabilities without the computational burden of inference-time search.

Stage 2: RL with Monte-Carlo Critic (MC-Critic)

Conventional parametric critics in LLM-based RL are hampered by high variance and poor sample efficiency, especially in long-horizon domains. ProAct replaces parametric critics with a parameter-free MC-Critic, which dynamically estimates the state value by Monte Carlo averaging over lightweight, environment-based rollouts (using a random policy as a surrogate for the LLM policy). Critically, MC-Critic provides a low-variance, efficiently-computable auxiliary signal, and is agnostic to the underlying policy optimization algorithm (integrating with either PPO or GRPO).

This estimator is robust to sparse rewards (Sokoban) and long-horizon, noisy returns (2048). Ablation on hyperparameters $M$ (number of rollouts) and $T$ (horizon per rollout) reveals domain sensitivity: denser reward environments benefit from larger $M$/$T$, while environments with sparse rewards or short optimal solutions require conservative settings to avoid gradient dilution. Empirically, MC-Critic consistently outperforms parametric approaches and yields stable, monotonic policy improvements in both RL from-scratch and RL-from-GLAD initializations.

Empirical Evaluation

Benchmarks and Metrics

ProAct is evaluated on 2048 (stochastic, long-horizon, cumulative score maximization) and Sokoban (deterministic, sparse-reward, box-to-goal navigation), including multiple OOD test sets (variant grid sizes, tile values, action space permutations, symbol mapping changes).

The Qwen3-4B-Instruct LLM backbone is used for all models and ablations to ensure comparability to both state-of-the-art open and closed-source baselines.

GLAD’s Effect on Reasoning Quality

Supervised GLAD training confers a substantial performance advantage relative to generic instruction tuning. In representative case studies, GLAD-trained LLMs generate compact, correct, and forward-looking reasoning chains, reliably simulate the impact of candidate actions, and explicitly reject suboptimal moves based on projected environment feedback. In contrast, base instruction-tuned models exhibit context drift, redundant steps, and hallucinated transitions.

Figure 3: Model outputs on 2048—(a) baseline model fixates on generic "thinking" with simulation errors; (b) GLAD-trained model achieves accurate, concise lookahead with explicit trade-off reasoning.

RL and MC-Critic Results

On all benchmarks, RL equipped with MC-Critic (MC-PPO/MC-GRPO) dominates both vanilla Step-GRPO/Step-PPO and brute-force Traj-GRPO, especially as task horizon and reward sparsity increase. Fine-tuning from GLAD SFT checkpoints yields further state-of-the-art gains, and these improvements persist under environment randomization and OOD variants.

For example, in the 2048 4×4 standard setting, the ProAct-enhanced model (GLAD+MC-Critic) achieves a cumulative score of 4503.8, outperforming all open-source baselines and rivaling larger closed-source models. Similarly, in Sokoban, the average "boxes on target" rises from 0.39 (base) to 0.94 (ProAct), with robust transfer to variant levels and action spaces.

Figure 4: PPO on 2048: MC-Critic consistently improves training stability and asymptotic performance compared to vanilla PPO.

Figure 5: PPO on 2048: Step-wise performance improvements are more pronounced at higher horizons, highlighting ProAct’s effectiveness in long-term planning.

Hyperparameter analysis demonstrates that, for 2048, MC-GRPO with large $M$ and $T$ is optimal, while in Sokoban, smaller $M$ avoids excessive averaging over near-zero returns, preserving discrimination between optimal and suboptimal moves.

Figure 6: MC-GRPO sensitivity to $M$ (number of rollouts) and $T$ (rollout length): trade-off between variance and discriminability is environment-dependent.

Discussion and Implications

ProAct’s paradigm directly addresses the principal bottlenecks of LLM-centric agents in long-horizon planning: simulation drift, compounding errors, and high-variance value estimation. By decoupling search-based foresight from inference and compressing it into language-aligned reasoning supervisors, GLAD enables efficient System-1-like policy execution. MC-Critic, by providing an unbiased, environment-grounded low-variance baseline, alleviates optimization instability inherent in multi-turn RL for LLMs.

Practically, this removes dependence on expensive inference-time search and opens the door to scalable training of agentic LLMs for complex, interactive scenarios, including program synthesis, autonomous navigation, and text-based games. Theoretically, ProAct suggests that explicit environment calibration and systematic reasoning compression are critical to bridging the System 1-System 2 gap in agentic cognition.

Immediate research avenues include: (1) adaptive search compression mechanisms integrating task and environment priors; (2) hierarchical MC-Critics for hierarchical RL; (3) joint training with world model distillation; and (4) scaling to multi-agent and multimodal environments.

Conclusion

ProAct establishes a robust framework for internalizing accurate lookahead reasoning and stable policy optimization in LLM-driven agents through its two-stage GLAD+MC-Critic method. Empirical evidence across complex benchmarks demonstrates that ProAct-trained agents achieve state-of-the-art performance and generalize robustly to OOD tasks, rivaling larger proprietary systems in both stochastic and deterministic environments. This framework represents a significant advancement towards practical, scalable training of agentic LLMs under realistic, interactive conditions.

---

Reference:

"ProAct: Agentic Lookahead in Interactive Environments" (2602.05327)