---
title: Multi-Turn GRPO for Efficient Sequential RL
url: https://www.emergentmind.com/topics/multi-turn-group-relative-policy-optimization-mtgrpo
type: topic
---

# Multi-Turn GRPO for Efficient Sequential RL

Multi-Turn Group Relative Policy Optimization (mtGRPO) generalizes group-based policy optimization to multi-turn sequential decision problems involving complex credit assignment. It provides dense, turn-level, group-relative feedback for each interaction step, supporting sample-efficient and stable reinforcement learning (RL) in scenarios where rewards are sparse or delayed. mtGRPO is a central innovation for high-performance multi-turn reasoning and tool-use with large language models (LLMs) and multimodal agents, as demonstrated in domains such as autonomous driving, tool-integrated reasoning, and multi-agent collaboration [2601.22930][2602.03025][2511.14846][2511.13288][2509.20616].

## 1. Foundations and Motivation

Classical RL algorithms such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO) perform policy gradient updates using trajectory-level or episode-level rewards. In multi-turn interaction settings—where an agent must reason and act iteratively over several turns—rewards are typically sparse, delayed, and not attributable to individual decisions within the sequence. Standard approaches suffer from two key issues: (1) feedback sparsity, in which most actions receive no learning signal, and (2) poor credit assignment, where reward cannot distinguish which turns contributed positively or negatively to the final outcome [2601.22930][2511.14846].

mtGRPO addresses these limitations by redefining the RL signal at each interaction step: it introduces per-turn reward collection and computes the *group-relative advantage* for each turn by contrasting the agent’s performance against a batch-level baseline within the same turn. This structure increases the density and specificity of learning signals, substantially enhancing convergence and generalization in long-horizon, multi-turn regimes [2601.22930][2511.14846].

## 2. Multi-Turn Decision Process and Algorithmic Formulation

An mtGRPO episode is represented as a $T$-turn Markov decision process (MDP), where at each turn $t\in \{1, \dots, T\}$, the agent observes a state $s_t$, takes an action $a_t$, and immediately receives a reward $r_t$. The state comprises all relevant information, including environmental context, interaction history, and turn-wise feedback. The objective is to maximize cumulative discounted rewards:
\[
G_t = \sum_{k=t}^T \gamma^{k-t} r_k
\]
where $\gamma$ is the discount factor [2601.22930][2511.14846].

mtGRPO proceeds as follows:
- For each batch of $N$ rollouts and at each turn $t$, the algorithm collects per-turn rewards $\{r_{i,t}\}_{i=1}^N$.
- It computes the turn-specific batch mean $b_t = (1/N)\sum_{i=1}^N r_{i,t}$ and standard deviation $\sigma_t$.
- The group-relative advantage for each rollout at turn $t$ is $\hat{A}_{i,t} = (r_{i,t} - b_t) / \sigma_t$.

The per-token surrogate objective extends the PPO/GRPO loss to sum over both tokens and turns, with the group-relative advantage applied to each action token generated at the corresponding turn:
\[
J_{\text{mtGRPO}}(\theta) = \mathbb{E}\Bigg[ \frac{1}{N}\sum_{i=1}^N \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \min\Big( r_t(\theta) \hat{A}_{i,t}, \text{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat{A}_{i,t} \Big) - \beta D_{\rm KL}[\pi_\theta \| \pi_{\rm ref}] \Bigg]
\]
where $r_t(\theta)$ is the importance weight, $\epsilon$ is the clipping parameter, $\beta$ is the KL penalty, and $\pi_{\rm ref}$ is a reference (e.g., SFT) policy [2601.22930][2511.14846][2511.13288].

Pseudocode for mtGRPO (single-agent case) is as follows [2601.22930]:
```python
for each RL iteration:
    # Sample N rollouts under old policy
    for each rollout i in N:
        for each turn t in T:
            a_i_t = agent.act(s_i_t)
            r_i_t = reward(a_i_t)
    for each turn t:
        b_t = mean([r_i_t for i in 1..N])
        sigma_t = std([r_i_t for i in 1..N])
        for each rollout i:
            A_hat_i_t = (r_i_t - b_t) / sigma_t
    Optimize J_mtGRPO using collected (states, actions, advantages)
    Periodically update reference policy
```

## 3. Extensions: Multi-Turn Reasoning, Tool Use, and Multi-Agent Systems

mtGRPO has been extended across several domains, each adapting the group-relative framework to domain-specific challenges.

- **Multi-Turn Trajectory Refinement in Autonomous Driving**: In MTDrive, mtGRPO supports an MLLM-based agent performing iterative trajectory refinement, where each turn’s reward is derived from perceptual driving model (PDM) metrics (e.g., collision, drivable area, time-to-collision) and is directly assigned to the tokens generated at that turn, providing efficient and targeted credit assignment [2601.22930].
- **Multi-Turn Tool-Calling Agents**: mtGRPO is integrated as Reward-Conditioned GRPO (RC-GRPO), which injects reward-mode tokens (e.g., `<|high_reward|>`/`<|low_reward|>`) at each rollout to promote within-group diversity and combat reward variance collapse. This yields robust policy updates even in environments with sparse or bimodal reward distributions [2602.03025].
- **Tool-Integrated Reasoning with LLMs**: Referred to as Group Turn Policy Optimization (GTPO), the method includes per-turn reward shaping (e.g., partial rewards for code-similar negative trajectories), turn-level returns, and discounting, which leads to superior question-answering and reasoning performance on mathematical benchmarks. Here, the group-relative advantage is computed at each reasoning step [2511.14846].
- **Hierarchical Multi-Agent Systems**: In M-GRPO, the multi-turn framework is extended to handle a main agent (planner) and subordinate tool agents with different frequencies and response times. Group-relative advantages and PPO-style loss are computed separately for each agent, with batch-wide trajectory alignment to maintain synchronization despite stochastic agent invocation counts [2511.13288].

## 4. Reward Assignment, Advantage Estimation, and Credit Sharpening Strategies

A central property of mtGRPO is the densification and sharpening of credit assignment via group-relative normalization at each turn.

- **Per-Turn Reward Collection**: mtGRPO collects individual step-wise rewards, often aggregating multiple informational channels (e.g., environmental signals, formatting, tool usage metrics), yielding $r_t = w_p \cdot p_t + w_f \cdot f_t$ in driving [2601.22930], or $r_{i,j} = r_{\rm acc} + r_{\rm format}$ in tool reasoning [2511.14846].
- **Relative Advantage Computation**: By normalizing each rollout’s return by the contemporaneous batch mean and standard deviation at the same turn, mtGRPO mitigates low-variance scenarios and ensures sustained gradient updates, even when the policy becomes peaked after extensive SFT [2602.03025].
- **Reward Shaping**: In tool-based reasoning, partial rewards are awarded for code similarity between failed and successful runs, based on embedding similarity (e.g., Titan Text Embeddings), further densifying the learning signal [2511.14846].
- **Reward-Conditioned Sampling**: In RC-GRPO, explicit conditioning on reward tokens structurally introduces reward variance within each batch group, guaranteeing non-degenerate group normalization and improved advantage spread [2602.03025].

## 5. System-Level Optimizations and Implementation

mtGRPO’s adoption in high-throughput multimodal RL settings necessitates system-level engineering [2601.22930].

- **Inter-Process Streaming Serialization (IPSS)** enables immediate tensor serialization and streaming to training workers as soon as a rollout is completed, minimizing idle time and maximizing device utilization.
- **Intra-Process Tensor Cache (IPTC)** consolidates multimodal embeddings and tokenization across co-located modules (actor, reference, log-prob computation), reducing redundant deserialization and memory copies.
- Both optimizations jointly achieve a $\sim2.5\times$ speedup in wall-clock training throughput, critical for large-scale RL with high-dimensional inputs and multi-turn sequences.

In multi-agent distributed training, M-GRPO adopts a decoupled pipeline in which each agent operates independently, sharing only scalar reward statistics and trajectory identifiers via a lightweight database, ensuring scalable deployment without cross-server backpropagation or parameter sharing [2511.13288].

## 6. Empirical Results and Comparative Performance

mtGRPO and its extensions have demonstrated significant empirical gains across diverse benchmarks:

| Domain            | Baseline (SFT/GRPO)       | mtGRPO Variant         | Reported Gain                |
|-------------------|---------------------------|------------------------|------------------------------|
| Autonomous Driving| SFT: 88.1, GRPO: 94.2     | mtGRPO: 96.2 (oracle)  | Exceeds VLM-driving/human    |
| Tool Calling      | SFT+GRPO: 48.75           | mtGRPO: 85.00 (Qwen)   | Surpasses closed-API models  |
| Reasoning (GTPO)  | GRPO: 49.78               | GTPO: 51.26            | +3.0% average                |
| Multi-Agent       | Single: 54-58             | M-GRPO: 68-72          | +7% over main-only/frozen    |
| Task Planning     | Larger model: 0%-SR       | mtGRPO 1.5B: 70%-SR    | Outperforms 14B models       |

In ablation studies, per-turn advantage normalization is identified as critical for convergence and non-vanishing policy gradients. Qualitative analyses demonstrate that error correction (e.g., in trajectory rollouts) is localized and successive turns target previously problematic steps [2601.22930][2511.14846][2509.20616].

## 7. Theoretical Guarantees and Application Implications

Under appropriate assumptions (unique minimal-turn expert trajectories, dense verifiable rewards), mtGRPO provides formal guarantees: improvements in the group-based single-turn objective provably translate to higher multi-turn success probabilities and more sample-efficient policies, as established by backward induction arguments and explicit bounds on success probability [2509.20616]. Empirical evidence supports strong cross-task generalization, minimal task completion times, and stable learning curves.

A plausible implication is that future research extending mtGRPO to more open-ended, partially observable, or hierarchical tasks will need to further refine group assignment, baseline computation, and advantage normalization techniques to accommodate the increased complexity of credit assignment.

---

**Key References**
- MTDrive: Multi-turn Interactive Reinforcement Learning for Autonomous Driving [2601.22930]
- RC-GRPO: Reward-Conditioned Group Relative Policy Optimization for Multi-Turn Tool Calling Agents [2602.03025]
- Empowering Multi-Turn Tool-Integrated Reasoning with Group Turn Policy Optimization [2511.14846]
- Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPO [2511.13288]
- Training Task Reasoning LLM Agents for Multi-turn Task Planning via Single-turn Reinforcement Learning [2509.20616]

Source: https://www.emergentmind.com/topics/multi-turn-group-relative-policy-optimization-mtgrpo