Trajectory-wise Group Relative Policy Optimization

• TGRPO is a reinforcement learning framework that generalizes groupwise policy optimization to full trajectory decision-making for tasks with sparse or episodic rewards.
• It integrates trajectory grouping, normalized advantage estimation, and KL regularization to ensure stable and efficient policy improvements in multi-step, high-dimensional settings.
• TGRPO has demonstrated robust performance in robotics, image captioning, and generative modeling, highlighting its effectiveness in complex, sequential control tasks.

Trajectory-wise Group Relative Policy Optimization (TGRPO) is a reinforcement learning framework that extends the group relative policy optimization (GRPO) approach to temporally extended, trajectory-based decision-making problems. TGRPO is specifically designed for high-dimensional, multi-step control and sequence modeling tasks, such as those encountered in robotic manipulation, continuous control, and generative modeling with temporal or sequential structure. By integrating trajectory-level and group-based advantage estimation, TGRPO enables robust, sample-efficient, and stable policy improvement in domains where feedback is sparse, delayed, or only available at the episodic or outcome level.

1. Foundational Principles

TGRPO generalizes the core concept of GRPO—which estimates the policy improvement signal by normalizing rewards within groups of sampled outputs—to the trajectory level. In trajectory-based environments, an agent generates entire sequences (trajectories) of actions and receives cumulative or episodic rewards that may be only weakly attributable to individual steps. TGRPO structures learning around the following principles:

• Trajectory grouping: For each episode or prompt, multiple full trajectories are sampled from the current or recent policy. Rewards for these trajectories are collectively normalized and used for advantage estimation and policy updates.
• Advantage normalization: The reward—or an aggregation of stepwise rewards—on each trajectory is compared to the group mean and variance, yielding a normalized advantage signal. Policy improvement is then driven by these relative, rather than absolute, signals.
• KL regularization and trust-region constraints: As with PPO and GRPO, TGRPO includes an explicit step-size control via KL divergence penalties or clipping, ensuring updates remain close to the current or reference policy for stability.

This paradigm enables policy optimization to leverage relative performance within a group, reducing susceptibility to degenerate baselines and high-variance updates that arise in traditional actor-critic or self-critical methods.

2. Methodological Framework

The core TGRPO algorithm operates via iterative sampling and policy updates over groups of trajectories. Each iteration consists of the following steps:

1. Group Sampling: For each input or environment instance (e.g., image, prompt, or initial state), a set of $G$ trajectories $\{o_i\}_{i=1}^G$ is sampled using the current (or previously snapshot) policy $\pi_{\theta_{\mathrm{old}}}$.
2. Reward Computation: Each trajectory $o_i$ is assigned a reward $r_i$. This reward can be outcome-based (e.g., task success), cumulative (sum of stepwise rewards), or constructed from dense, process-aware signals when available.
3. Advantage Estimation: The group-relative advantage for trajectory $i$ is computed as

$$A_i = \frac{r_i - \operatorname{mean}(\{r_1, \ldots, r_G\})}{\operatorname{std}(\{r_1, \ldots, r_G\})}$$

Variants such as adaptive or zero-variance-corrected estimators are also employed to ensure gradients even in degenerate groups.

1. Surrogate Policy Update: For each trajectory, a likelihood ratio between the new and old policy is computed, and a clipped (PPO-style) surrogate loss is optimized, typically of the form:

$$\mathcal{J}_{\text{TGRPO}}(\theta) = \mathbb{E}_{i}\big\{ \min\big( \frac{\pi_\theta(o_i)}{\pi_{\theta_{\mathrm{old}}}(o_i)}A_i,~ \operatorname{clip}\big( \frac{\pi_\theta(o_i)}{\pi_{\theta_{\mathrm{old}}}(o_i)}, 1-\epsilon, 1+\epsilon \big)A_i \big) \big\} - \beta D_\mathrm{KL}(\pi_\theta \| \pi_{\mathrm{ref}})$$

1. Gradient Step: Model parameters are updated via stochastic gradient descent using the above objective, often batched over multiple inputs or environments.

This procedure is repeated, often with periodic resampling of the old policy and adaptation of the KL constraint for stability.

3. Key Mathematical Formulations

The TGRPO framework unifies groupwise and trajectory-level policy optimization through a set of characteristic formulas:

• Group-normalized advantage (trajectory level):

$A_i = \frac{r_i - \mu}{\sigma}$

where $\mu$ and $\sigma$ are the mean and standard deviation of rewards across the group.

• Integration with stepwise/trajectory advantage fusion (as in vision-language-action tasks):

$$\operatorname{Adv}_{i,t} = \alpha_1 S_{i,t} + \alpha_2 T_i$$

Where $S_{i,t}$ and $T_i$ are the normalized advantages at step $t$ and for the entire trajectory, respectively.

• Clipped surrogate loss:

$$\mathbb{E}_{i} \left[ \min \left( r_{i,t}(\theta) \operatorname{Adv}_{i,t},~ \operatorname{clip}(r_{i,t}(\theta), 1-\varepsilon, 1+\varepsilon)\operatorname{Adv}_{i,t} \right) \right]$$

where $$r_{i,t}(\theta) = \pi_\theta(o_{i,t}|q, o_{i,<t}) / \pi_{\theta_{\mathrm{old}}}(o_{i,t}|q, o_{i,<t})$$.

• KL regularization constraint:

$D_{\mathrm{KL}}(\pi_\theta \| \pi_{\mathrm{ref}})$

penalizes deviation from a stable reference, preserving diversity and preventing overfitting to group-specific reward noise.

4. Advances and Performance Characteristics

TGRPO demonstrates several empirically validated advantages over non-grouped and stepwise-only alternatives:

• Stability and sample efficiency: By leveraging groupwise comparison, TGRPO avoids instability associated with unreliable or degenerate baselines, as seen in self-critical or single-greedy-reference methods. In online robotic manipulation tasks, TGRPO outperforms supervised fine-tuning (SFT) and classic actor-critic baselines by significant margins (e.g., 4.4% higher average success rate in LIBERO-Object evaluations) and shows greater robustness and efficiency.
• Length and efficiency control: Modifications such as adaptive advantage estimation and length-based rewards, as in AGPO, enable TGRPO to yield concise and efficient generation without sacrificing accuracy.
• Fine-grained and process-level optimization: Extensions such as TreeRPO (2506.05183) build on TGRPO by assigning dense, step-level group rewards through tree-based sampling, yielding substantially improved accuracy and response efficiency in reasoning tasks.
• Generality across modalities: TGRPO applies to trajectory-based RL tasks in robotics, multimodal reasoning in LLMs and MLLMs, sequence generation (image captioning, speech), and autoregressive generative modeling. This is reflected in successful applications spanning vision-language-action models, next-scale visual autoregressive models, and multimodal reasoning agents.
• Synergy with variance reduction methods: TGRPO is compatible with off-policy sampling, partial trajectory reuse, and other sample-efficient RL techniques (e.g., VRER, MLR), extending its practical reach in distributed or low-data regimes.

5. Comparative Analysis and Applications

TGRPO addresses limitations present in both classic trajectory optimization and group-based RL methods:

• Compared to classic model-free trajectory optimization (e.g., MOTO (1606.09197)), TGRPO maintains groupwise stability and is adaptable to settings where Q-functions or rewards are sparse or available only on trajectories.
• In comparison with step-wise group policy optimization, TGRPO prioritizes stable, temporally-coherent policy updates on long-horizon tasks, which is essential for robotics and closed-loop control.
• Empirically, TGRPO provides:
  • Improved generalization and robustness when applied as online RL fine-tuning for VLA models across a diverse suite of tasks.
  • Enhanced sample and computational efficiency due to compatibility with group sampling and modern hardware-accelerated inference.
  • Superior outcome and process efficiency in educational, medical, and decision-making LLMs/MLLMs.

6. Limitations and Future Directions

TGRPO poses several open questions and research opportunities:

• Advantage weighting and hyperparameterization: Current approaches for fusing step and trajectory-level signals (e.g., $\alpha_1, \alpha_2$) are task-specific and require empirical tuning. Automated, adaptive weighting or meta-optimization could further improve portability and generalization.
• Credit assignment in very long or partially observable trajectories: The propagation and normalization of group advantages in highly stochastic or hierarchical environments remains a challenge.
• Group formation strategies: Most implementations group by prompt or environment instance; dynamic grouping by context, phase, or event could further improve sample efficiency and generalization.
• Scaling and distributed training: Off-policy, replay, and adaptive masking mechanisms could be further integrated for efficient distributed or federated RL with TGRPO in large-scale or high-throughput environments.

A notable direction is the fusion of dense tree-based credit assignment (TreeRPO) or process-level reward propagation for reasoning, and off-policy/partial trajectory reuse for sample efficiency in continuous control domains.

7. Representative Implementations and Empirical Benchmarks

A variety of research works and implementations exemplify TGRPO and its trajectory-wise, group-relative paradigm:

Domain	Key Benefit	Implementation/Result
VLA Model Online RL Fine-tuning (2506.08440)	Stable learning on multi-stage, long-horizon manipulation (LIBERO-Object)	+4.4% avg success, denser credit assignment
Image Captioning (2503.01333)	Diversity & stable policy improvement	Outperforms SCST on BLEU, CIDEr, SPICE
Autoregressive Image Generation (2505.23331)	Efficient alignment to nuanced, CLIP-based human rewards	Style generalization beyond ImageNet, improved aesthetics
Reasoning with LLMs (2503.12937, 2506.05183)	Dense, process-level feedback, reduced token cost	+2.9%–16.5% accuracy boosts, concise outputs
Robotic Continuous Control (2505.13549)	Robustness to high-dimensional instability	Rapid, stable learning on 26-DoF Unitree H1-2

Practical implementations and released codebases are available for TGRPO in image captioning (MindSpore), voice MoE transformers (PyTorch), VLA online RL (OpenVLA), and TreeRPO (Qwen-2.5-Math).

---

TGRPO serves as a foundation for scalable, robust, and sample-efficient RL across a broad spectrum of temporally-extended, group-structured decision problems. Its generalization of groupwise policy improvement to the trajectory level enables stable RL optimization in both simulated and real-world environments, with ongoing research focused on enhancing dynamic advantage fusion, adaptive group formation, and scalable, distributed learning strategies.