Turn-Level Advantage Estimation

• Turn-level advantage estimation is a reinforcement learning technique that assigns distinct per-turn credit, enabling precise reward decomposition in multi-turn tasks.
• It leverages methods such as direct advantage estimation, graph-based aggregation, and bi-level GAE to reduce variance and stabilize policy updates.
• The approach improves sample efficiency and convergence in complex, long-horizon environments, benefiting agents like large language and vision-language models.

Turn-level advantage estimation is a family of reinforcement learning (RL) methods that assign distinct advantage values to each interaction turn or step within an agent’s trajectory, as opposed to coarse trajectory-level assignment. This granularity is central for complex, multi-turn tasks—especially in LLM and vision-LLM (VLM) agents—where effective credit assignment under sparse or delayed reward regimes directly impacts stability, convergence, and overall agent performance. The development of turn-level advantage estimators addresses fundamental challenges in credit assignment, variance reduction, and sample efficiency in long-horizon domains. Recent methodological innovations span graph-based merging, dual/segmental advantage construction, information-theoretic signals, hybrid trajectory-turn credit decomposition, and direct advantage learning.

1. Motivation and Conceptual Foundations

Classic RL approaches with sparse trajectory-level rewards distribute the same feedback uniformly across all steps, resulting in gradient conflict, suboptimal policy updates, and instability, especially for long-horizon or multi-turn tasks. Group-based methods such as Group Relative Policy Optimization (GRPO) suffer acutely: beneficial and detrimental actions are intertwined, and assigning undifferentiated advantages to all steps suppresses learning signal (Li et al., 22 Oct 2025). This inaccuracy is considerably amplified as sequence lengths increase (the “advantage collapse” problem), which motivates finer granularity credit assignment.

Formally, the advantage function $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$ quantifies the marginal causal effect of an action at a particular step on expected return, unmixing the direct effect of per-step (“skill”) versus environment stochasticity (“luck”) (Pan et al., 2024). Turn-level advantage estimation exploits this per-step decomposition to deliver precise, low-variance updates.

2. Methodological Approaches to Turn-Level Advantage Estimation

Recent research advances various turn-level estimation methodologies. The principal strategies include:

• Graph-based aggregation: SALT forms a trajectory graph over all steps from sampled group rollouts; edges representing identical actions under equivalent context are merged, and per-step advantages are computed as a mean of the group-normalized trajectory rewards attached to each edge. This directly leverages cross-trajectory structure for refining local credit signal (Li et al., 22 Oct 2025).
• Direct Advantage Estimation (DAE): Rather than regressing value or $Q$, DAE fits a parameterized $\pi$-centered advantage function by minimizing the variance between trajectory returns and the sum of per-step advantages. This estimation holds under both on-policy and off-policy settings (the latter with auxiliary “luck” correction terms for stochastic transitions), yielding unbiased, local, and stable per-turn effects (Pan et al., 2021, Pan et al., 2024).
• Turn-level GAE and segmentation: Extensions of Generalized Advantage Estimation to turn granularity compute TD errors and propagate advantage backwards only at turn (or segment) boundaries. Segmental Advantage Estimation (SAE) analytically reduces bias versus token-level GAE by aligning bootstrapping to information-rich turn delimiters rather than every token (Gong et al., 12 Jan 2026).
• Intrinsic and extrinsic hybridization: Approaches such as IGPO define per-turn intrinsic rewards by measuring marginal information gain in the model’s answer likelihood after each turn. Turn-wise normalized and discounted cumulative advantages are then formed directly from these dense, intrinsic signals combined with explicit outcome rewards (Wang et al., 16 Oct 2025).
• Dual-level and hierarchical estimators: MatchTIR constructs both trajectory-level and per-turn normalized returns, then uses a composite advantage—integrating global and local credit—to each token in a turn. Fine-grained per-turn rewards are defined by bipartite matching (hard/soft assignments) between predicted and ground-truth tool traces (Qu et al., 15 Jan 2026).
• Bi-Level GAE: For visual agents, a turn-aware “bi-level” GAE first computes turn-level GAE, then injects this at each turn boundary into a token-level GAE within the turn, facilitating hierarchical credit propagation in partially observable domains (Wang et al., 19 Oct 2025).
• Turn-level MDP parametrization: Turn-PPO reforms the MDP such that each environment-agent “turn” is treated as a single MDP step, with full-response actions and rewards. This smooths critic learning, further lowering the variance of advantage calculations (Li et al., 18 Dec 2025).

3. Mathematical Formalisms and Key Algorithms

Let $K$ denote the number of dialogue turns, with $(s_t, a_t, r_t)$ as the state, action, and reward at turn $t$.

• Turn-level value and advantage:

$$V^\pi(s_t) = \mathbb{E}\left[\sum_{k=t}^{K}\gamma^{k-t} r_k \mid s_t\right]$$

$$A_t^{\mathrm{TD}} = r_t + \gamma V^\pi(s_{t+1}) - V^\pi(s_t)$$

$$A_t^{\mathrm{GAE}(\gamma,\lambda)} = \sum_{\ell=0}^{K-t} (\gamma\lambda)^\ell \delta_{t+\ell}$$

with $$\delta_t = r_t + \gamma V^\pi(s_{t+1}) - V^\pi(s_t)$$ (Wei et al., 17 May 2025).

• Group-normalized per-turn advantages:

$$\widehat{A}_{i,j} = \frac{R_{i,j} - \mu_{R,j}}{\sigma_{R,j}}$$

where $R_{i,j}$ is the discounted return at turn $j$ for rollout $i$ (Ding et al., 18 Nov 2025).

• Direct estimation (π-centering):

$$\hat{A}_\theta(s,a) = f_\theta(s,a) - \sum_{a'} \pi(a'|s) f_\theta(s,a')$$

The objective is

$$L(\theta) = \mathbb{E}\left[\left(G(\tau) - \sum_{t=0}^T \gamma^t \hat{A}_\theta(s_t,a_t)\right)^2\right]$$

(Pan et al., 2021, Pan et al., 2024).

• Trajectory-graph merging (SALT): For each edge $e$, define

$$\hat{A}'(e) = \mathrm{mean}\{\widehat{A}(e') : e' \text{ merges with } e\}$$

This value is assigned to all occurrences of the step in the batch (Li et al., 22 Oct 2025).

• Segmental and bi-level GAE:

Recursive credit propagation is adapted so that aggregation and bootstrapping align to the semantic turn boundaries (Wang et al., 19 Oct 2025, Gong et al., 12 Jan 2026).

4. Practical Implementation and Algorithmic Workflows

Turn-level advantage estimation frameworks share a standardized RL policy optimization loop incorporating the following stages:

1. Batch rollout generation: For each sampled prompt, produce $G$ trajectories using the current policy.
2. Per-turn reward assignment: Design local rewards using outcome feedback, information gain, verifiable rules, LLM-based rubrics, or bipartite matching for tool interactions. In group-based RL, dense shaping (e.g., format or partial correctness) is frequently introduced (Ding et al., 18 Nov 2025, Wang et al., 16 Oct 2025, Wei et al., 17 May 2025, Qu et al., 15 Jan 2026).
3. Advantage computation: Calculate discounted per-turn returns, normalize (often $z$-score across batch and/or turn index), and/or propagate via GAE or direct estimation according to the framework.
4. Policy/critic updates: Employ per-turn (and possibly per-token) advantages in PPO/GRPO/clipped objectives, with KL penalty if using reference anchoring (Li et al., 18 Dec 2025, Ding et al., 18 Nov 2025).
5. Auxiliary components: For off-policy correction, DAE/skill–luck decomposition use explicit latent variable models for unbiased advantage regression (Pan et al., 2024); for graph-based methods, step merging (merge/diverge) is performed using efficient lookup or hash-based aggregation (Li et al., 22 Oct 2025).

Table 1: Comparison of Core Turn-Level Advantage Estimators

Method	Advantage Granularity	Key Innovation
SALT	Step (graph-edge)	Trajectory graph, merge-based edge averaging
IGPO	Turn	Information gain as intrinsic reward
GTPO	Turn	Group-normalized, shaped per-turn credit
MatchTIR	Turn+Trajectory	Bipartite reward assignment, dual-level adv.
Bi-Level GAE	Turn+Token	Hierarchical, turn-injected GAE
SAE	Segment/Turn	Variable-length segment GAE, boundary-aware
DAE	Turn	Direct, π-centered advantage, causal reg.
Turn-PPO	Turn	MDP at turn granularity, turn-critic

5. Computational Complexity and Empirical Properties

Turn-level methods increment compute cost minimally relative to baseline group-normalized return approaches, as shown for the SALT framework where the combined cost of trajectory graph construction and step-advantage computation accounts for ≪1% of total RL iteration time (e.g., ~0.15s per iteration vs 270s for RL update with Qwen2.5-1.5B) (Li et al., 22 Oct 2025). The dominant cost in most frameworks remains environment rollouts and model forward passes.

Empirically, turn-level advantage estimation yields:

• Performance improvement over trajectory-level baselines across a diverse suite of benchmarks including ALFWorld, WebShop, AppWorld, BFCL, ToolHop, and multi-turn QA/search environments (Li et al., 22 Oct 2025, Ding et al., 18 Nov 2025, Wei et al., 17 May 2025, Qu et al., 15 Jan 2026).
• Reduced gradient conflict and more precise credit propagation, reflected in both faster convergence and higher plateau performance.
• Variance/bias trade-off is tunable via turn/segment granularity: segment-length in SAE, merge history in SALT, discount factors in dual-level schemes.
• Superior correlation to ground-truth per-step advantage: SAE demonstrates higher correlation with Monte Carlo ground-truth advantage than any token-level GAE(λ), justifying reduced bias (Gong et al., 12 Jan 2026).
• Robustness to hyperparameters and model scale: Properly tuned turn-level estimators scale from 1.5B up to 32B models and maintain benefit at longer horizons (Ding et al., 18 Nov 2025, Li et al., 22 Oct 2025, Qu et al., 15 Jan 2026).
• Ablations confirm that group size, history/segment length, and normalizer selection are significant, but turn-level granularity per se is the dominant explanatory variable for empirical gains.

6. Limitations, Design Considerations, and Future Directions

Turn-level advantage approaches are predicated on meaningful reward decomposition, which may require careful engineering (e.g., bipartite matching, LLM-judge construction, or verifiable rubric definition). Some methods—such as MatchTIR—rely on the availability of ground-truth tool traces, which may restrict applicability in open-ended or creative domains (Qu et al., 15 Jan 2026). Another consideration is the choice of turn segmentation: too fine (token-level) reverts to traditional GAE with high variance; too coarse loses crucial temporal structure and delays feedback.

Open research directions include:

• Generalizing fine-grained reward decomposition to settings without explicit expert traces or external judges.
• Scaling dual-level and hierarchical estimation methods to larger models and non-textual modalities.
• Exploring meta-learning of credit assignment boundaries (e.g., automatically inferring optimal segmentation for advantage propagation).
• Bias-variance characterization of adaptive segmentation and segment-length scheduling (as in SAE).

7. Impact and Empirical Outcomes in Agent Training

Experiments consistently show that turn-level advantage estimation accelerates policy improvement, increases sample efficiency, and yields higher accuracy benchmarks across agentic LLM and VLM tasks. In ALFWorld, incorporation of SALT boosts overall GRPO success from 81.8% to 85.2%; on AppWorld, turn-level estimation lifts GRPO (Test-N TGC) from 61.5% to 66.2% (Li et al., 22 Oct 2025). In multi-turn TIR, GTPO achieves ≥3% overhead reduction over classic GRPO, while MatchTIR demonstrates that dual-level advantage schemes yield outsized gains with increasing horizon (Ding et al., 18 Nov 2025, Qu et al., 15 Jan 2026). In the VLM domain, bi-level GAE within VAGEN systems achieves 3× improvement over untrained baselines and outperforms proprietary agents in long-horizon, world-modeling benchmarks (Wang et al., 19 Oct 2025).

Turn-level advantage estimation thus enables robust, scalable, and fine-grained credit assignment in modern multi-turn RL, serving as a foundational tool for current and future agent-centric learning paradigms.