Turn-PPO: Turn-Level RL for LLM Agents

Updated 3 July 2026

Turn-PPO is a reinforcement learning algorithm that redefines policy optimization at the turn level, aligning updates with natural action boundaries to reduce variance.
It computes turn-level importance ratios using a geometric mean of token probabilities and leverages Generalized Advantage Estimation for stable credit assignment.
Empirical results show Turn-PPO outperforms token-level methods, achieving higher returns and enhanced stability in multi-turn, long-horizon tasks.

Turn-PPO is a family of reinforcement learning (RL) algorithmic variants applied to LLM agents operating in multi-turn, multi-stage environments. These environments commonly require agents to perform sequences of tool calls, reason over extended horizons, and act in discrete, semantically meaningful “turns” (e.g., agent-initiated actions or responses). Turn-PPO departs from standard token-level Proximal Policy Optimization (PPO) by aligning the granularity of policy optimization and credit assignment with environment-defined turns, addressing the instability and credit assignment issues endemic to long-horizon LLM agent training (Li et al., 18 Dec 2025, Li et al., 25 Nov 2025, Wei et al., 17 May 2025).

1. Motivation and Foundations

The dominant approach in LLM RL fine-tuning has been token-level PPO, applying policy ratio calculation and advantage estimation at every generation step. In multi-turn tasks—such as dialog, interactive tool use, or sequential reasoning—this introduces two critical sources of instability:

Temporal mismatch: Natural MDP transitions occur at the turn (macro-action) level, but token-level PPO aggregates policy updates and credit over micro-steps, ignoring the semantic structure (Li et al., 18 Dec 2025, Li et al., 25 Nov 2025).
High-variance gradients: Lack of fine-grained, process-level credit assignment (e.g., one scalar reward for the entire trajectory) leads to unreliable and high-variance advantage estimates at the token level, especially for long, sparse-reward tasks (Wei et al., 17 May 2025, Li et al., 25 Nov 2025).

Turn-PPO reinterprets the RL problem at the turn level—each turn comprising one or more generated tokens corresponding to a distinct logical or interaction stage—thereby stabilizing credit assignment and aligning the RL update step with environment semantics.

2. Formal Algorithmic Structure

Turn-PPO operates by segmenting each agent trajectory into discrete turns, computing policy updates and advantage estimation per turn rather than per token.

Let each trajectory $y=(y^1; y^2;\dots;y^K)$ be partitioned into $K$ turns, where $y^k$ denotes the token sequence for turn $k$ . The core elements are:

Turn-level state: $s_k$ denotes the state preceding turn $k$ , i.e., the full dialog history or previous actions and received observations (Li et al., 18 Dec 2025, Li et al., 25 Nov 2025).
Turn-level action: $a_k = y^k$ is the full response produced in turn $k$ .
Turn-level importance ratio:

$w_k^{\mathrm{turn}}(\theta) = \left(\frac{\pi_\theta(y^k \mid s_k)}{\pi_{\theta_{\mathrm{old}}}(y^k \mid s_k)}\right)^{1/|y^k|} = \exp\left(\frac{1}{|y^k|} \sum_{t\in k}\log\frac{\pi_\theta(y_t|s_t)}{\pi_{\theta_{\mathrm{old}}}(y_t|s_t)}\right)$

All tokens in turn $k$ share this common importance weight, reducing variance associated with per-token ratios (Li et al., 25 Nov 2025).

Advantage estimation: Rather than assigning the same advantage to every token in the trajectory (as in GRPO), or using high-variance sample-based methods, Turn-PPO computes turn-level advantages, typically employing Generalized Advantage Estimation (GAE):

$K$ 0

$K$ 1

with $K$ 2 a learned turn-value critic (Li et al., 18 Dec 2025).

Objective: The turn-level surrogate loss is

$K$ 3

where all tokens $K$ 4 in turn $K$ 5 inherit the same ratio $K$ 6, and typically the same (possibly averaged) advantage $K$ 7 (Li et al., 25 Nov 2025).

3. Algorithmic Implementation Details

The practical Turn-PPO training loop is as follows (Li et al., 25 Nov 2025, Wei et al., 17 May 2025):

Trajectory Collection: Sample batches of multi-turn episodes, segmenting each into turns (using dialogue structure, special markers, or task-defined boundaries).
Advantage Estimation: For each turn, compute turn-level GAE advantages, using a learned value critic and observed turn-level or shaped rewards.
Turn-Level Ratios: Compute a geometric mean of per-token importance weights within each turn, assign this ratio to all turn tokens.
Policy and Critic Updates: Minimize the surrogate PPO objective at turn granularity; update critic using MSE loss on returns.
Hyperparameters: Typical settings include $K$ 8, actor LR $K$ 9, critic LR $y^k$ 0, GAE parameters $y^k$ 1, $y^k$ 2, with entropy penalty for exploration.

A representative pseudocode for Turn-PPO simplifies to:

$y^k$ 3

4. Empirical Performance and Benchmarks

Turn-PPO and its variants have been validated across a broad range of long-horizon, multi-turn tasks:

Environment / Model	GRPO	Token-PPO	Turn-PPO
WebShop, Qwen2.5-3B	0.72	0.73	0.75
Sokoban, Qwen2.5-7B	Crash	2.90	3.74
HotpotQA (Qwen-2.5-1.5B)	~60%	~70%	~78%

Turn-PPO achieves substantially higher average returns and lower gradient norms compared to token-level PPO and GRPO, both in textual environments (WebShop) and combinatorial reasoning tasks (Sokoban). Notably, it prevents the training collapse observed in baseline methods on long-horizon reasoning and sparse reward problems (Li et al., 18 Dec 2025, Li et al., 25 Nov 2025, Wei et al., 17 May 2025).

Empirical studies attribute these improvements to:

More stable value estimation: Turn-level transitions are homogeneous, aiding critic learning.
Variance reduction: Shared importance ratios and advantages across turn tokens lessen update volatility.
Fine-grained credit assignment: Turn-level or shaped rewards ensure that policy updates reflect the causal effect of individual turns on task success.

The Turn-PPO paradigm has inspired extensions and hybrid approaches. Notable variants include:

ST-PPO: Combines turn-level importance sampling with additional clipping-bias correction to address high-variance advantages from off-policy replay, especially for large models (Li et al., 25 Nov 2025). ST-PPO achieves robust stability and monotonic improvement over both turn-only and token-level variants.
MT-PPO: Augments Turn-PPO with explicit turn-level reward shaping from verifiable signals or LLM-judged rubrics, further enhancing convergence speed and output validity in search-augmented QA (Wei et al., 17 May 2025).
Tree-based/Process-level methods: Frameworks such as AT²PO (Zong et al., 8 Jan 2026), A²TGPO (Chen et al., 7 May 2026), and GTPO (Ding et al., 18 Nov 2025) introduce structured rollouts, fine-grained credit assignment mechanisms, or adaptive clipping, but almost universally incorporate turn-level ratios or rewards, positioning Turn-PPO as a foundational component.

6. Practical Considerations, Pitfalls, and Future Extensions

Despite its empirical strengths, Turn-PPO exposes several open issues:

High-dimensional action spaces: Each action corresponds to an entire textual span (turn), so probability estimation and policy evaluation become more complex than for single-timestep actions (Li et al., 18 Dec 2025).
Reward sparsity: While turn-level shaping partially mitigates sparse credit, environments with only delayed terminal rewards may still challenge value function learning (Wei et al., 17 May 2025).
Turn boundary detection: Implementations require reliable segmentation of agent behaviors into turns, which is trivial for protocol-driven interaction (e.g., dialog) but less so for free-form text or tool-calls.

Opportunities for further improvement include adaptive GAE parameter tuning, hierarchical decomposition of turns into sub-tasks, and integration with process-level intrinsic reward signals (e.g., information gain (Chen et al., 7 May 2026)) to enhance credit assignment in more unstructured domains.

7. Significance and Impact

Turn-PPO and its derivatives have become central in contemporary LLM RL agent training for scenarios that require robust, scalable optimization over long-horizon, multi-turn interactions. By aligning RL updates with environment- and task-defined turns, Turn-PPO sidesteps core limitations of token-level credit assignment, delivers lower-variance policy updates, and empirically achieves state-of-the-art success rates and stability (Li et al., 18 Dec 2025, Li et al., 25 Nov 2025, Wei et al., 17 May 2025). Its adoption underpins recent advances in tool-augmented QA, compositional reasoning, and interactive LLM agent design.

Principal references: (Li et al., 18 Dec 2025, Li et al., 25 Nov 2025, Wei et al., 17 May 2025, Zong et al., 8 Jan 2026, Chen et al., 7 May 2026, Ding et al., 18 Nov 2025)