Trajectory-Group Policy Optimization (T-GRPO)

• T-GRPO is a trajectory-centric policy optimization framework that replaces uniform credit assignment with structured, group-level signals.
• It employs diverse formulations—from verifiable process supervision to tree-based temporal grouping—to improve variance reduction and update stability.
• The approach has been applied across reasoning tasks, continuous control, vision-language-action manipulation, and combinatorial optimization with notable performance gains.

Trajectory-Group Relative Policy Optimization (T-GRPO) denotes a family of GRPO-derived policy optimization schemes in which the trajectory becomes the primary unit for grouping, normalization, correction, or credit assignment. In current usage, the label does not refer to a single standardized algorithm. Instead, it covers several distinct constructions: segment-wise verifiable process supervision for reasoning LLMs, trajectory-level importance correction for critic-free alignment, trajectory clustering and state-aware normalization for continuous control, and trajectory-grouped PPO-style training for neural combinatorial optimization. Across these formulations, the common objective is to retain GRPO’s critic-free or baseline-free character while replacing indiscriminate or poorly scaled credit assignment with trajectory-structured signals (Wang et al., 22 Apr 2026, Pang et al., 4 Aug 2025, Khanda et al., 25 Jul 2025, Sepúlveda et al., 9 Jun 2026).

1. GRPO substrate and the trajectory-level shift

The common starting point is Group Relative Policy Optimization (GRPO). For a prompt or instance $x$, GRPO samples a group of $G$ trajectories and computes a relative advantage within that group. In one formulation, used for verifiable reasoning tasks, the trajectory-level advantage is

$A^i = r^i - \frac{1}{G}\sum_{j=1}^G r^j,$

where $r^i \in \{0,1\}$ denotes final correctness. In another formulation, common in PPO-style variants, the group-relative signal is a z-score,

$$\hat A_i = \frac{R(\tau_i)-\mu_G}{\sigma_G+\delta},$$

or equivalently with $\max(\sigma_G,\epsilon)$ in the denominator. Standard GRPO then applies the same trajectory-level advantage to every token or action in that rollout (Wang et al., 22 Apr 2026, Pang et al., 4 Aug 2025, Sepúlveda et al., 9 Jun 2026).

T-GRPO variants arise from dissatisfaction with that uniform assignment. The reported failure modes are different across domains but structurally related: indiscriminate credit assignment for intermediate reasoning steps, high-variance estimation at early denoising stages, instability from baseline dependence, bias from token-level importance weighting, and weak discrimination in long-horizon control. T-GRPO therefore shifts the optimization granularity from “one scalar reward per whole completion” to trajectory segments, trajectory clusters, descendant sets in rollout trees, or single trajectory-level likelihood ratios (Wang et al., 22 Apr 2026, Lyu et al., 30 Nov 2025, Pang et al., 4 Aug 2025, Khanda et al., 25 Jul 2025).

2. Principal formulations in the literature

Formulation	Defining mechanism	Representative source
Verifiable-process T-GRPO	Hybrid segment advantage $A^i + \alpha \Delta C_k^i$ from confidence probes on partial reasoning	(Wang et al., 22 Apr 2026)
TIC-GRPO / T-GRPO	Replace token-level importance ratios with one trajectory-level ratio $R(\tau;\theta,\theta_{\mathrm{old}})$	(Pang et al., 4 Aug 2025)
Continuous-control T-GRPO	Trajectory feature clustering, DBSCAN state clustering, regularized clipped objective	(Khanda et al., 25 Jul 2025)
NCO T-GRPO	Intra-group z-score advantages over sampled solution trajectories, no external baseline	(Sepúlveda et al., 9 Jun 2026)
Temporal-grouping T-GRPO	Tree-based rollout with descendant-based rewards and per-group normalization	(Lyu et al., 30 Nov 2025)

These formulations share vocabulary but differ materially in objective construction. In some papers, “trajectory” means the unit at which importance correction is computed. In others, it means the unit on which clustering is performed, the object whose descendants define local value estimates, or the episode whose global return is fused with step-level signals. This suggests that T-GRPO is best understood as a trajectory-centric design space built on GRPO rather than a single canonical update rule.

3. Credit assignment along reasoning and decision trajectories

A major T-GRPO direction addresses temporal credit assignment directly. In “GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning” (Wang et al., 22 Apr 2026), a generated trajectory is segmented into $M$ contiguous chunks $z_k$. After segment $G$0, the model is probed with the known correct answer $G$1, producing

$G$2

The hybrid segment-level advantage is then

$G$3

Positive $G$4 marks a reasoning segment that increased the model’s belief in the correct answer; negative $G$5 marks a harmful segment. The paper reports up to 2.6-point accuracy improvements and 13.7% reasoning-length reductions on math tasks, and up to 2.4 points and 4% on general-domain tasks, while adding up to $G$6 extra forward-only probes per trajectory when $G$7 (Wang et al., 22 Apr 2026).

A closely related credit-fusion idea appears in “TGRPO :Fine-tuning Vision-Language-Action Model via Trajectory-wise Group Relative Policy Optimization” (Chen et al., 10 Jun 2025). There, step-level relative advantages $G$8 and trajectory-level normalized returns $G$9 are combined as

$A^i = r^i - \frac{1}{G}\sum_{j=1}^G r^j,$0

The experimental ablation isolates the effect of horizon fusion: step-only $A^i = r^i - \frac{1}{G}\sum_{j=1}^G r^j,$1 yields 73.6% average success over 10 LIBERO-Object tasks, trajectory-only $A^i = r^i - \frac{1}{G}\sum_{j=1}^G r^j,$2 yields 86.8%, and full TGRPO yields 91.0%. The same study reports SFT at 86.4% and PPO at 86.6%, indicating that trajectory-level signals carry most of the gain while the step-level term improves stabilization (Chen et al., 10 Jun 2025).

In diffusion and flow-based generation, temporal grouping is instantiated through rollout trees rather than explicit segment probes. The trajectory-grouping component of “Multi-GRPO: Multi-Group Advantage Estimation for Text-to-Image Generation with Tree-Based Trajectories and Multiple Rewards” (Lyu et al., 30 Nov 2025) branches only at selected early denoising steps, defines descendant-based rewards

$A^i = r^i - \frac{1}{G}\sum_{j=1}^G r^j,$3

and normalizes them within each temporal group. Early branching improved PickScore from 23.65 for Flow-GRPO to 24.24 in the single-reward ablation, and an early branching rule $A^i = r^i - \frac{1}{G}\sum_{j=1}^G r^j,$4 outperformed a late rule with the same leaf count (Lyu et al., 30 Nov 2025).

4. Trajectory-level correction, normalization, and stability mechanisms

A distinct T-GRPO line targets bias in the policy-gradient estimator itself. “On the Theory and Practice of GRPO: A Trajectory-Corrected Approach with Fast Convergence” argues that token-level importance sampling in GRPO estimates $A^i = r^i - \frac{1}{G}\sum_{j=1}^G r^j,$5 rather than $A^i = r^i - \frac{1}{G}\sum_{j=1}^G r^j,$6, and introduces Trajectory level Importance Corrected GRPO (TIC-GRPO) (Pang et al., 4 Aug 2025). The core substitution is

$A^i = r^i - \frac{1}{G}\sum_{j=1}^G r^j,$7

used in a trajectory-level clipped surrogate. Under KL-smoothness, bounded reward, and finite-variance assumptions, the paper gives the convergence statement

$A^i = r^i - \frac{1}{G}\sum_{j=1}^G r^j,$8

and reports that removing token-level importance sampling entirely yields performance nearly identical to standard GRPO when $A^i = r^i - \frac{1}{G}\sum_{j=1}^G r^j,$9 is refreshed frequently (Pang et al., 4 Aug 2025).

In combinatorial optimization, stability is pursued through self-contained normalization rather than trajectory-ratio correction. “Baseline-Free Policy Optimization for Neural Combinatorial Optimization” uses grouped solution trajectories, defines returns as $r^i \in \{0,1\}$0, normalizes them within each instance-specific group, and optimizes a PPO-clip objective without any rollout baseline or critic (Sepúlveda et al., 9 Jun 2026). The study uses group size $r^i \in \{0,1\}$1, inner PPO epochs $r^i \in \{0,1\}$2, and anneals sampling temperature from $r^i \in \{0,1\}$3 to $r^i \in \{0,1\}$4 over the first 30% of training epochs to avoid degenerate $r^i \in \{0,1\}$5. Empirically, GRPO avoids the TSP-100 collapse observed with REINFORCE, where cost degrades from 9.8 to 52.1 immediately after the warmup phase, and remains within 2% of POMO at matched gradient updates (Sepúlveda et al., 9 Jun 2026).

Safety-constrained dual-horizon normalization appears in “Can the Environment Speak for Itself? $r^i \in \{0,1\}$6-GRPO: A Turn-Trajectory Group Relative Policy Optimization for Caregiver Agents” (Song et al., 7 Jun 2026). That framework separates environment-grounded turn rewards from trajectory-level rubric scores, applies centered-rank normalization to each channel independently, and imposes a hard veto: $r^i \in \{0,1\}$7 The hard veto ensures that a catastrophic safety violation cannot be offset by good performance elsewhere in the rollout (Song et al., 7 Jun 2026).

A related but differently named LLM variant, GTPO, addresses instability through conflict-token masking and entropy filtering rather than trajectory ratios or trajectory clusters. It suppresses negative updates on conflict tokens, doubles positive ones, and filters trajectories whose average entropy exceeds the threshold $r^i \in \{0,1\}$8 (Simoni et al., 5 Aug 2025).

5. Domains of application and reported results

Domain	Representative variant	Reported outcome
Mathematical and general-domain reasoning	Verifiable-process T-GRPO (Wang et al., 22 Apr 2026)	Up to 2.6-point accuracy improvement and 13.7% reasoning-length reduction on math; up to 2.4 points and 4% on general-domain tasks
Vision-language-action manipulation	TGRPO (Chen et al., 10 Jun 2025)	91.0% average success over 10 tasks; step-only 73.6%; trajectory-only 86.8%
Neural combinatorial optimization	T-GRPO (Sepúlveda et al., 9 Jun 2026)	Best epoch-matched performance on TSP-50, TSP-100, CVRP-50, CVRP-100; within 2% of POMO at matched updates
Dementia caregiver dialogue	$r^i \in \{0,1\}$9-GRPO (Song et al., 7 Jun 2026)	Caregiving quality 0.82 vs. 0.74 for GDPO; safety violation rate 2.1% vs. 7.6%; dialogue quality 0.83
Text-to-image generation	Temporal grouping in Multi-GRPO (Lyu et al., 30 Nov 2025)	PickScore 24.24 vs. 23.65 for Flow-GRPO in the single-reward ablation
Image-to-video generation	TAGRPO (Wang et al., 9 Jan 2026)	~0.06–0.10 Q-Save gain at 320p, ~0.09–0.13 at 720p; convergence in 1.9–2.5 GPU-hours vs. 2.4–3.2 h for DanceGRPO

The domain spread is unusually broad. In language-model reasoning, trajectory structure is used to probe latent progress toward a verifier-known answer. In robotics and VLA training, it is used to combine local transition rewards with global episode outcomes. In NCO, it replaces external baselines with intra-group normalization over full solution rollouts. In multimedia generation, it supports tree-based temporal grouping, direct trajectory alignment, or latent-space pruning. In domain-specific sequence prediction, ShipTraj-R1 reformulates ship trajectory prediction as text generation and applies a clipped GRPO objective with group-normalized rewards, using Qwen3-8B, group size $$\hat A_i = \frac{R(\tau_i)-\mu_G}{\sigma_G+\delta},$$0 or $$\hat A_i = \frac{R(\tau_i)-\mu_G}{\sigma_G+\delta},$$1, clipping $$\hat A_i = \frac{R(\tau_i)-\mu_G}{\sigma_G+\delta},$$2, and KL coefficient $$\hat A_i = \frac{R(\tau_i)-\mu_G}{\sigma_G+\delta},$$3 (Zhan et al., 3 Mar 2026).

The continuous-control formulation in “Extending Group Relative Policy Optimization to Continuous Control: A Theoretical Framework for Robotic Reinforcement Learning” is notable because it is presented as a theoretical framework rather than an empirical benchmark paper. It identifies HalfCheetah-v4, Walker2d-v4, Ant-v4, FrankaReach, HandManipulateBlock, and ShadowHandCatch as target applications, and grounds the algorithm in trajectory-based clustering, DBSCAN state clustering, temporal smoothness, and inter-group diversity regularization (Khanda et al., 25 Jul 2025).

6. Theoretical status, nomenclature, and unresolved design axes

Theoretical treatment is still uneven. The continuous-control T-GRPO framework proves convergence to a stationary point under bounded rewards, Lipschitz policies, bounded gradient norm, Robbins–Monro learning-rate conditions, and sufficiently large $$\hat A_i = \frac{R(\tau_i)-\mu_G}{\sigma_G+\delta},$$4, and gives the sample-complexity statement

$$\hat A_i = \frac{R(\tau_i)-\mu_G}{\sigma_G+\delta},$$5

for achieving an $$\hat A_i = \frac{R(\tau_i)-\mu_G}{\sigma_G+\delta},$$6-optimal policy with probability at least $$\hat A_i = \frac{R(\tau_i)-\mu_G}{\sigma_G+\delta},$$7 (Khanda et al., 25 Jul 2025). TIC-GRPO, by contrast, provides the first convergence analysis explicitly for GRPO-style language-model optimization with trajectory-level importance correction (Pang et al., 4 Aug 2025).

The nomenclature remains unsettled. Alongside T-GRPO, the literature contains “Trajectory-wise Group Relative Policy Optimization” (TGRPO) for VLA fine-tuning (Chen et al., 10 Jun 2025), “Turn-Trajectory Group Relative Policy Optimization” ($$\hat A_i = \frac{R(\tau_i)-\mu_G}{\sigma_G+\delta},$$8-GRPO) for caregiver agents (Song et al., 7 Jun 2026), “Trajectory-Group Relative Policy Optimization” as temporal grouping inside Multi-GRPO (Lyu et al., 30 Nov 2025), “TAGRPO” for image-to-video direct trajectory alignment (Wang et al., 9 Jan 2026), “GTPO” for conflict-aware LLM optimization (Simoni et al., 5 Aug 2025), and “Pro-GRPO” for expand-and-prune trajectory filtering in generative models (Ge et al., 17 Dec 2025). This suggests that “trajectory” has become a general design axis in GRPO research rather than a single recipe.

Several design axes recur across these papers. Group construction may be prompt-level, rollout-level, tree-structured, or feature-clustered. Advantage estimation may be binary-relative, z-score normalized, centered-rank normalized, or fused across turn, step, segment, and trajectory horizons. Update control may rely on PPO clipping, KL anchoring, entropy filtering, or hard veto constraints. Auxiliary structure may come from verifier-known answers, simulator state transitions, latent descendants, or trajectory feature vectors. A plausible implication is that future work will continue to specialize T-GRPO-like methods by domain, with the principal questions centered on which trajectory decomposition yields the best variance–bias trade-off, which normalization preserves heterogeneous reward channels most faithfully, and how far critic-free optimization can scale before explicit value estimation again becomes advantageous.