Multi-Turn Reinforcement Learning

• Multi-Turn Reinforcement Learning is a framework that formalizes agent-environment interactions over extended sequences to achieve cumulative objectives in long-horizon tasks.
• It leverages scalable policy optimization methods like PPO and GRPO, integrating neural architectures and structured reward shaping to handle sparse or multi-component rewards.
• MT-RL has demonstrated success in domains such as clinical dialogue, GUI/web navigation, and multi-agent collaboration, consistently outperforming single-turn approaches.

Multi-Turn Reinforcement Learning (MT-RL) is a framework in which autonomous agents interact over a sequence of discrete decision points with dynamic, feedback-driven environments, learning to maximize cumulative or sequence-level objectives not attainable in single-step settings. In contrast to single-turn RL, which addresses one-shot or immediate-reward environments, MT-RL is uniquely suited for tasks involving long-horizon planning, language/dialogue, multi-step tool use, navigation, or multi-agent collaboration. The MT-RL paradigm formalizes agent–environment interaction as a Markov Decision Process (MDP) or Partially Observable MDP (POMDP) with trajectories comprising multiple alternating observations and actions, sparse or structured reward signals, and policies that must integrate sequential context for optimal performance. Recent advances leverage scalable policy optimization algorithms (e.g., PPO, GRPO), neural agent architectures (transformers, multimodal LLMs), and problem-specific reward shaping to achieve strong empirical and theoretical gains in domains such as dialogue, clinical reasoning, GUI/web navigation, software engineering, and scientific discovery.

1. Formalism and Problem Formulation

In MT-RL, the task is typically cast as an episodic finite-horizon MDP or POMDP:

$\mathcal{M}=(\mathcal{S},\mathcal{A},P,R,T)$

where:

• $\mathcal{S}$: state space; $s_t$ encodes the full history and current environment observation at step $t$.
• $\mathcal{A}$: action space; $a_t$ may be a discrete action, a structured language utterance, or a tool call, depending on domain.
• $P(s_{t+1}|s_t,a_t)$: transition kernel; possibly deterministic (GUI navigation) or stochastic (dialogue, user simulation).
• $R(s_t,a_t)\in\mathbb{R}$: reward function; may be sparse (terminal-only), dense (per-step), or multi-component/shaped.
• $T$: horizon length (episode terminates at $t=T$ or upon a stopping criterion).

The agent selects actions according to a (possibly stochastic) policy $\pi_\theta(a_t\,|\,s_t)$, with the objective:

$$J(\pi_\theta) = \mathbb{E}_{\pi}\Bigl[\sum_{t=0}^{T}\gamma^t R(s_t,a_t)\Bigr]$$

Dialogue-based and agentic domains often formulate the state as the entire interaction history, e.g., $H_t = (o_0,a_0,o_1,\dots,a_{t-1},o_t)$, and actions as natural language sequences or tool invocations (Wang et al., 1 Oct 2025, Feng et al., 26 May 2025, Yan et al., 2 Dec 2025). Multi-agent MT-RL generalizes this to joint action spaces and possible competition or collaboration (Liu et al., 30 Jun 2025).

2. Algorithmic Frameworks and Policy Optimization

Contemporary MT-RL adopts scalable policy-gradient optimization, extending single-turn PPO and related algorithms to cope with long-horizon, high-dimensional, and often language-mediated environments. Key exemplars include:

Proximal Policy Optimization (PPO)

• Online or batched rollouts of trajectories $\tau$, advantage estimation (possibly via GAE), and KL penalization to stabilize exploration.
• Clipping surrogate objective:

$$L^{CLIP}(\theta) = \mathbb{E}[\min(r_t(\theta)\,A_t,\,\mathrm{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\,A_t)]$$

where $$r_t(\theta)=\pi_{\theta}(a_t|s_t)/\pi_{\theta_{old}}(a_t|s_t)$$ (Wang et al., 1 Oct 2025, Abdulhai et al., 2023, Feng et al., 26 May 2025).

Group Relative Policy Optimization (GRPO)

• Groups rollouts by prompt or query, normalizes advantages within group, and uses group-relative baselines in gradient computation.
• Surrogate combines group-clipped ratios and (optionally) KL or entropy regularization:

$$\mathcal{J}_{\rm GRPO}(\theta) = \mathbb{E}\biggl[ \frac{1}{G} \sum_{i=1}^G \sum_{t} \min(r_{i,t}\hat{A}_{i,t}, \mathrm{clip}(r_{i,t},1-\epsilon,1+\epsilon)\hat{A}_{i,t}) - \beta\,\mathrm{D}_{\mathrm{KL}}[\pi_\theta \| \pi_{\rm ref}] \biggr]$$

(Feng et al., 26 May 2025, Yan et al., 2 Dec 2025, Zhang et al., 16 Sep 2025).

Value-Based and Unbiased Estimators

• State-value or advantage networks, or REINFORCE variants lacking learned critics, used in certain complex or sparse-reward regimes (Wang et al., 1 Oct 2025, Abdulhai et al., 2023).

Curriculum, Filtering, and Stabilization

• Curriculum learning over solution length/horizon or environment complexity (Yan et al., 2 Dec 2025, Wang et al., 1 Oct 2025).
• Trajectory filtering by variance, outcome, or quality to stabilize updates and prevent policy collapse in long-horizon settings (Wang et al., 24 Apr 2025, Feng et al., 26 May 2025, Xu et al., 29 Oct 2025).

3. Reward Design and Shaping

Reward structures in MT-RL are engineered to address the credit assignment, exploration, and alignment challenges inherent in long-horizon domains:

• Sparse Terminal Rewards: Reward provided only at episode end (task success, dialogue outcome), sufficient in deterministic settings or with trajectory-level preference comparisons (Yan et al., 2 Dec 2025, Wei et al., 22 May 2025, Zhao et al., 26 Aug 2025).
• Dense Turn-Level or Shaped Rewards: Intermediate rewards for subgoals, compliance, information gain, tool validity, or reasoning structure (Feng et al., 26 May 2025, Liu et al., 30 Jun 2025, Zhou et al., 2023, Sun et al., 14 Aug 2025).
• Gated Reward Accumulation (G-RA): Stepwise rewards are conditional on success of higher-level objectives, preventing reward hacking and ensuring global alignment (Sun et al., 14 Aug 2025).
• Preference-Based Rewards: Human or model-derived preferences over whole trajectories or intermediate steps, enabling policy optimization when scalar rewards are unavailable (Shani et al., 23 May 2024, Wang et al., 26 Sep 2025).

Multi-component reward design incorporating task-specific, compliance, information, and protocol structure has been shown to substantially improve efficiency and policy robustness (Feng et al., 26 May 2025, Sun et al., 14 Aug 2025). A typical instantiation for clinical dialogue evaluation is:

$$R(s_t,a_t) = R_{\rm accuracy}(H_T) + \sum_{i=1}^T R_{\rm info}^i + \sum_{i=1}^T R_{\rm compliance}^i$$

with component-wise shaping as in (Feng et al., 26 May 2025).

4. Applications and Domain-Specific Adaptations

MT-RL has demonstrated state-of-the-art performance across a variety of domains requiring temporally extended planning, reasoning, and interaction:

Clinical Dialogue and Multi-Agent Reasoning

• DoctorAgent-RL frames the doctor–patient consultation as a multi-turn MDP and uses GRPO to learn efficient questioning, outperforming single-turn and SFT-only agents on multi-turn diagnostic accuracy (Feng et al., 26 May 2025).

GUI and Web Navigation

• GUI Exploration Lab applies MT-RL for screen graph exploration, using sparse goal rewards and staged learning to generalize navigation policies across interface layouts. Multi-turn policies discover backtracking and unseen path strategies absent in single-turn RL (Yan et al., 2 Dec 2025).
• WebAgent-R1 leverages asynchronous multi-turn rollouts, minimal binary rewards, and dynamic context compression for real-world web tasks, dramatically improving success rates over BC and single-turn baselines (Wei et al., 22 May 2025).

Vision-and-Language Navigation (VLN)

• ActiveVLN combines short imitation learning with multi-turn GRPO on episodic 3D navigation, achieving >10% SR gains over prior IL+DAgger or RL-only approaches via active exploration and early-stopping heuristics (Zhang et al., 16 Sep 2025).

Dialogue, Negotiation, and Preference-RL

• Supporter (Zhou et al., 2023) and other mixture-of-expert or modular architectures use turn-level reward composition (emotion elicitation, coherence, future dialogue match) and trajectory-level value estimation to drive policy improvement in conversational agents.
• Multi-turn preference RL methods optimize policies against global or trajectory-level preferences, enabling alignment and planning in the absence of direct scalar reward (Shani et al., 23 May 2024, Wang et al., 26 Sep 2025).

Tool Use, Task Planning, and Scientific Optimization

• MUA-RL integrates user simulation into the MT-RL loop, exposing agents to dynamic, stochastic queries and requiring simultaneous communication and tool invocation (Zhao et al., 26 Aug 2025).
• POLO applies Preference-Guided MT-RL for molecular lead optimization, learning from both trajectory-level strategic RL and dense intra-trajectory preference comparisons for superior sample efficiency (Wang et al., 26 Sep 2025).
• In planning domains, single-turn RL on decomposed expert state–action pairs is proven to amplify multi-turn task completion probability under GRPO, with zero-shot subtask generalization (Hu et al., 24 Sep 2025).

5. Empirical Results and Theoretical Guarantees

Empirical studies demonstrate that MT-RL confers substantial improvements over single-turn RL, imitation learning, and purely supervised fine-tuning, particularly as task complexity and horizon length increase:

Domain/Task	MT-RL Metric	SFT/Base	Single-Turn RL	MT-RL (Best)	Reference
Clinical dialogue (avg score)	Diagnosis+recom (%)	46.3–49.4	—	53.9	(Feng et al., 26 May 2025)
GUI navigation (OOD acc.)	Pass@1 (%)	14.3–17.2	17.2	17.5–25.2	(Yan et al., 2 Dec 2025)
Web navigation (SR)	Success rate (%)	20–24	—	33.9–44.8	(Wei et al., 22 May 2025)
Vision–Language Nav (VLN)	SR (R2R val-unseen %)	38.5 (IL)	—	50.1	(Zhang et al., 16 Sep 2025)
Lead optimization	Success rate (%)	—	~67	84 (single-prop)	(Wang et al., 26 Sep 2025)
Task planning	SR (Burger, >30-steps)	0.00–0.70	—	0.70	(Hu et al., 24 Sep 2025)

Theoretical results in multi-turn settings include:

• Convergence of preference-based MT-RL to Nash equilibria in tabular and policy-parametric settings (Shani et al., 23 May 2024).
• Proof that single-turn GRPO improvements on decomposed expert states amplify exact-path multi-turn task success (e.g., Theorem 3.5, (Hu et al., 24 Sep 2025)).
• In multi-agent zero-sum games, role-conditioned advantage normalization enables variance-reduced policy gradients and transfer of emergent reasoning patterns (Liu et al., 30 Jun 2025).

6. Stabilization, Generalization, and Future Directions

Key stabilization approaches for MT-RL include uncertainty-based trajectory filtering, asymmetric surrogate clipping, and explicit curriculum strategies, which prevent echo-traps and collapse in long-horizon environments (Wang et al., 24 Apr 2025, Xu et al., 29 Oct 2025). Best practices consistently recommend:

• Warm-starting from strong supervised or expert imitation priors;
• Progressive horizon/scenario scaling (curricula);
• Careful reward shaping or tiered gating of intermediate subgoals to avoid reward hacking (Sun et al., 14 Aug 2025);
• Extensive ablation to determine optimal SFT–RL allocation, advantage estimator variants, and reward density (Wang et al., 1 Oct 2025, Abdulhai et al., 2023).

Open research problems include sample efficiency in highly sparse-reward, expensive domains, reward/model alignment in open-ended settings, extending to multi-modal and multi-agent contexts, and integrating richer user/human-in-the-loop feedback.

7. Impact, Benchmarks, and Toolkits

MT-RL has catalyzed the development of benchmark suites (LMRL-Gym (Abdulhai et al., 2023), GUI-Exploration Lab (Yan et al., 2 Dec 2025), MTMedDialog (Feng et al., 26 May 2025)), domain-standard public codebases, and policy optimization libraries supporting language, vision, and tool-use agents. Empirically, MT-RL unlocks intentional, temporally-coherent agent behavior—such as targeted clinical conversations, robust navigation under distribution shift, and multi-step scientific optimization—outperforming both open and proprietary single-turn LLMs and Imitation Learning (IL)-based architectures. These results establish MT-RL as foundational for agentic LLM research and deployment in real-world, interactive domains.