Multi-Turn Agentic Planning

• Multi-turn agentic planning is a process where LLM-based agents iteratively interact with environments to achieve complex objectives through stateful reasoning and tool calls.
• It leverages RL algorithms like PPO and ReBN to enable dynamic credit assignment and uses modular architectures to decouple planning from execution.
• The approach is formalized via MDPs/POMDPs and evaluated through specialized benchmarks, asynchronous execution, and scalable curricula for real-world applications.

Multi-turn agentic planning is the process by which learning agents, typically based on LLMs, engage in extended interaction with environments or users, making a series of reasoning steps and tool calls across multiple dialogue or execution turns to achieve complex objectives. Unlike single-turn planning, which often relies on static datasets or isolated actions, multi-turn agentic planning requires strategies for stateful interaction, incremental information gathering, adaptive subgoal selection, and dynamic credit assignment. Recent work formalizes this paradigm via Markov decision processes (MDPs) or partially observable MDPs (POMDPs), emphasizing efficient interface design, stable reinforcement learning (RL) protocols, modularity, and robust evaluation. The field is supported by specialized environments, data generation pipelines, tool-centric datasets, and scalable RL algorithms.

1. Formal Foundations and Environment–Agent Interfaces

Multi-turn agentic planning is typically modeled as an RL or POMDP problem. In frameworks such as GEM ("General Experience Maker") (Liu et al., 1 Oct 2025), each episode comprises states $s_t$ that encode the full history of environment messages and prior actions, with actions $a_t$ produced by the agent as entire LLM-generated responses up to an EOS. The transition kernel $P(s_{t+1}|s_t, a_t)$ evolves an internal environment state and emits a new observation; the reward function $R(s_t, a_t)$ supports both dense per-turn rewards and sparse, episode-level rewards.

Environments can range from text-based games (TextWorld, ALFWorld), code reasoning platforms, QA with tool augmentation, to complex orchestration in synthetic DAGs (OrchDAG). Modular interfaces—often function-call based APIs or containerized endpoints—support heterogeneous execution and facilitate extensibility (Zhang et al., 5 Oct 2025). State and action spaces may be further abstracted via wrappers for observation types, action parsing, and tool integration.

2. RL Algorithms and Credit Assignment Strategies

Credit assignment across multiple turns is central to agentic planning. Naive trajectory-level updates lead to poor gradient signal for long-horizon reasoning tasks. Key RL approaches include:

• REINFORCE plus Return Batch Normalization (ReBN): Normalizes per-transition returns $G_i$ over batches, providing dense, gradient-rich updates. Empirically, ReBN improves stability and convergence—even for binary reward tasks where vanilla REINFORCE plateaus (Liu et al., 1 Oct 2025).
• PPO (Proximal Policy Optimization): Turn-level PPO computes advantages using a learned critic $V_\phi(s_t)$ and applies a clipped surrogate objective. Token-level credit assignment via GAE is used for granular feedback; however, turn-level PPO improves robustness in long-horizon settings (Li et al., 18 Dec 2025).
• Group Relative Policy Optimization (GRPO): Aggregates total trajectory rewards into a group-normalized advantage, but collapses to suboptimal behavior in multi-turn tasks with per-turn rewards (Liu et al., 1 Oct 2025, Zhao et al., 26 Aug 2025).
• Turn-Level MDPs: Defining the MDP at the semantic turn-level (rather than token-level) enables homogeneous transitions, reduces variance, and stabilizes critic learning for extended conversations and tool-use sequences (Li et al., 18 Dec 2025).
• Flow-GRPO: Broadcasts the final trajectory-level outcome to all turns, effectively optimizing single-turn updates in live, multi-turn environments. This resolves credit-assignment bottlenecks without requiring intermediate shaped rewards (Li et al., 7 Oct 2025).

3. Modular Architectures and Tool-Oriented Pipelines

Agentic systems increasingly employ a modular architecture, decoupling planning and execution. Notable designs include:

• Multi-module agents: Architectures such as AgentFlow coordinate planner, executor, verifier, and generator modules, each interacting through an evolving structured memory. This supports transparency, context compression, and reliable multi-turn reasoning (Li et al., 7 Oct 2025).
• Decoupled planning and generation: AI-SearchPlanner separates a lightweight, trainable search planner (LLM$_{\text{plan}}$) from a frozen answer generator (LLM$_{\text{gen}}$), optimizing only the planning loop via RL while maintaining high answer quality (Mei et al., 28 Aug 2025).
• Orchestration via DAGs: OrchDAG models multi-turn tool execution as a directed acyclic graph, enabling systematic evaluation of orchestrated plans and explicit dependency management. Graph-based rewards credit partial correctness and drive RL optimization (Lu et al., 28 Oct 2025).
• Caching and memory: Datasets like T1 stress explicit management of cache state for tool outputs, allowing dynamic replanning, cost-aware tool reuse, and multi-session persistence (Chakraborty et al., 22 May 2025).

4. Data Generation, Curriculum Design, and Evaluation Benchmarks

Multi-turn agentic planning demands high-quality, verifiable data capturing realistic agent–environment and agent–user dynamics:

• Synthetic trajectory generation: APIGen-MT introduces a two-phase protocol: committee-reviewed blueprint generation and simulated human–agent interplay, producing validated, diverse multi-turn datasets for downstream training (Prabhakar et al., 4 Apr 2025).
• Benchmarks: TravelBench and COMPASS target real-world domains (travel planning), providing multi-turn dialogues, structured tool ecosystems, and robust evaluation protocols encompassing preference optimization, constraint satisfaction, coordinator scoring, and tool-usage penalties (Cheng et al., 27 Dec 2025, Qin et al., 8 Oct 2025).
• Ambiguity and clarification: ClarifyMT-Bench studies multi-turn clarification, decomposed into slot perception, persona forecasting, state tracking, and planning. Empirical results show dramatic gains in ask–answer decision accuracy using agentic structures (Luo et al., 24 Dec 2025).

5. Practical Training Recipes, Scaling, and Infrastructure

Multiple research efforts distill reproducible recipes and infrastructure strategies for effective multi-turn agentic RL:

• Co-design across environment, reward, and policy: Empirical analyses recommend pairing dense turn-level rewards with robust RL algorithms (preferably PPO at token or turn level), leveraging supervised fine-tuning for data-efficient initialization, and structuring mixed-task curricula to boost generalization (Wang et al., 1 Oct 2025).
• Asynchronous, vectorized execution: AgentRL and GEM demonstrate that asynchronous, coroutine-based rollouts paired with autoreset and high-throughput batching yield 2–3× speedup in episode collection, essential for scaling RL to large models and diverse environments (Zhang et al., 5 Oct 2025, Liu et al., 1 Oct 2025).
• Normalization and exploration: Per-task advantage normalization and cross-policy sampling counteract inter-task variance and policy collapse, contributing to more stable multi-task learning (Zhang et al., 5 Oct 2025).
• Dynamic replanning, credit alignment, and best practices: Modular memory, explicit dependency graphs, and outcome-driven alignment simplify context management and error recovery in multi-tool settings. Empirical benchmarks consistently highlight the need for iterative plan refinement, explicit constraint propagation, and post-hoc reranking to close optimality gaps (Chakraborty et al., 22 May 2025, Lu et al., 28 Oct 2025).

6. Frontier Applications and Future Directions

Multi-turn agentic planning underpins current research into tool-augmented QA, code reasoning, travel and logistics, multi-agent system interaction, and conversational clarification. Noteworthy developments include long-horizon Text-to-SQL agents (Guo et al., 12 Oct 2025), preference-optimizing planners (Qin et al., 8 Oct 2025), and robust multi-agent route planning via unified frameworks spanning full POMDP recursion to scalable, myopic heuristics (Zhu et al., 13 Feb 2025). Ongoing challenges remain in credit assignment for sparse rewards, multi-domain coordination, scalable curriculum design, and comprehensive benchmarking. Future research is pointed toward hierarchical decomposition, semi-automatic policy extraction, multi-modal integration, dynamic subgoal management, and tighter alignment between RL protocols and real-world agentic system requirements.