Token-Sequence GRPO (TS-GRPO)

• TS-GRPO is a reinforcement learning framework that merges token-level and sequence-level policy optimization to reconcile fine-grained credit assignment with global reward stability.
• It employs a mixed importance ratio—blending GRPO and GSPO techniques—to balance detailed token updates with overall sequence-level performance.
• Empirical tests on mathematical reasoning benchmarks show that TS-GRPO (via DHPO variants) significantly outperforms traditional GRPO and GSPO methods.

Token-Sequence GRPO (TS-GRPO) denotes a reinforcement-learning-with-verifiable-rewards (RLVR) design in which token-level and sequence-level policy optimization are combined within a single autoregressive training objective. In the formulation most directly mapped to this name, TS-GRPO corresponds to Dynamic Hybrid Policy Optimization (DHPO), introduced as a method that bridges Group Relative Policy Optimization (GRPO) and Group Sequence Policy Optimization (GSPO) by mixing token-level and sequence-level importance ratios inside one clipped surrogate objective (Min et al., 9 Jan 2026). Its central motivation is that purely token-level or purely sequence-level optimization alone is inadequate for complex reasoning tasks: GRPO preserves fine-grained credit assignment but suffers from high variance and instability, whereas GSPO better matches sequence-level rewards and is more stable, but sacrifices token-wise credit assignment.

1. RLVR setting and the token-sequence problem

RLVR optimizes LLMs on tasks where solutions are automatically checked by rule-based verifiers. In mathematical and programming settings, the reward is typically sequence-level: an entire response is marked correct or incorrect after generation. Within this setup, a prompt $q$ is paired with a group of $G$ sampled responses, each response $o_i$ receives a scalar reward $r_i$, and the reward is normalized within the group to form a group-relative advantage that is then assigned uniformly to all tokens in that response (Min et al., 9 Jan 2026).

The basic tension that motivates TS-GRPO is a granularity mismatch. GRPO applies off-policy correction at the token level even though supervision arrives only at the sequence level. The paper describing DHPO states that this creates a “misalignment between the granularity of its off-policy correction and its supervision,” and that variance grows with response length, causing frequent clipping that can constrain learning and induce early collapse. GSPO addresses that mismatch by using a single sequence-level importance ratio shared across all tokens in a response. This better matches sequence-level rewards and is more stable, but it sacrifices token-wise credit assignment, which the paper describes as “particularly detrimental in reasoning tasks, where only a critical subset of tokens determines the final outcome” (Min et al., 9 Jan 2026).

TS-GRPO is therefore best understood as an attempt to preserve both local and global structure: token-wise correction for fine-grained reasoning credit, and sequence-wise correction for stability under sparse, verifier-defined rewards.

2. Core objective and mathematical structure

The common advantage used by GRPO, GSPO, and DHPO is the group-relative normalized reward

$$A_i = \frac{r_i - \operatorname{mean}(r_1, r_2, \dots, r_G)} {\operatorname{std}(r_1, r_2, \dots, r_G)}.$$

The same $A_i$ is uniformly assigned to all tokens of response $o_i$ (Min et al., 9 Jan 2026).

GRPO uses the token-level importance ratio

$$r_{i,t}(\theta)= \frac{\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})} {\pi_{\theta_{\text{old}}}(o_{i,t}\mid q,o_{i,<t})},$$

whereas GSPO defines a length-normalized sequence ratio

$$s_i(\theta)=\left(\frac{\pi_{\theta}(o_i\mid q)}{\pi_{\theta_{\text{old}}}(o_i\mid q)}\right)^{\frac{1}{|o_i|}},$$

together with a token-wise adjustment

$$s_{i,t}(\theta)=\text{sg}[s_i(\theta)]\cdot \frac{\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})} {\text{sg}[\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})]}.$$

DHPO, and thus TS-GRPO in this direct mapping, defines a mixed importance ratio

$G$0

with $G$1 controlling interpolation between the token-level and sequence-level branches. The clipped surrogate objective is

$G$2

where $G$3 is the clipped mixed ratio (Min et al., 9 Jan 2026).

Conceptually, this objective makes TS-GRPO a GRPO-style framework whose importance weighting is no longer purely token-local. Instead, it is explicitly hybridized across token and sequence granularities.

3. Mixing mechanisms, clipping, and optimization behavior

Two mixing mechanisms are described. The first is averaged mixing, which sets $G$4 and is presented as a hyperparameter-free interpolation. The second is entropy-guided mixing, which computes token entropy

$G$5

then maps it through min-max normalization with stop-gradient,

$G$6

Higher entropy places more weight on the token-level ratio $G$7, while lower entropy places more weight on the sequence-level ratio $G$8 (Min et al., 9 Jan 2026).

A central stabilization device is branch-specific clipping. Instead of mixing first and clipping afterward, DHPO clips the token-level and sequence-level branches separately and then combines them:

$G$9

The stated rationale is that this “prevents an outlier value in one branch from unduly influencing the combined update” and provides “independent control over the trust regions for local (token-level) and global (sequence-level) corrections” (Min et al., 9 Jan 2026).

The gradient analysis given for the unclipped objective shows that the mixed ratio still collapses to the standard score-function form,

$o_i$0

which is used to motivate the claim that DHPO “unifies GRPO and GSPO within a single gradient framework” (Min et al., 9 Jan 2026). A plausible implication is that TS-GRPO preserves ordinary policy-gradient implementation structure while changing only the importance-weighting geometry.

4. Training pipeline and empirical profile

The reported implementation uses verl together with vLLM on the SimpleRL dataset of 8,192 examples. For each query, the behavior policy samples groups at temperature $o_i$1; the experiments use prompt batch size $o_i$2, generate $o_i$3 responses per prompt, and use nucleus top-$o_i$4. Verifiable rewards are computed by a rule-based checker, advantages are normalized within each group, and no explicit KL term is added to the reward or objective. Additional settings include actor learning rate $o_i$5, weight decay $o_i$6, warmup $o_i$7 steps, input length $o_i$8, training response cap $o_i$9, evaluation max length $r_i$0, dynamic batching for log-probs, gradient checkpointing, padding removal, and no optimizer or parameter offloading. The reported compute budget is $r_i$1 nodes $r_i$2 $r_i$3 NVIDIA H100, for $r_i$4 GPUs total (Min et al., 9 Jan 2026).

The experiments cover seven mathematical reasoning benchmarks: AIME 2024 (Avg@32), AIME 2025 (Avg@32), AMC 2023 (Avg@4), OlympiadBench, MATH-500, Minerva Math, and GSM8K. The evaluated models are Qwen3-1.7B-Base, Qwen3-4B-Base, and Qwen3-30B-A3B-Base (Min et al., 9 Jan 2026).

On Qwen3-30B-A3B-Base, the entropy-guided variant DHPO-E improves AIME24 from $r_i$5 under GRPO to $r_i$6, and AIME25 from $r_i$7 to $r_i$8; averaged across the seven benchmarks, it exceeds GRPO by about $r_i$9 and GSPO by $$A_i = \frac{r_i - \operatorname{mean}(r_1, r_2, \dots, r_G)} {\operatorname{std}(r_1, r_2, \dots, r_G)}.$$0. On Qwen3-1.7B-Base, DHPO-E raises AIME24 from $$A_i = \frac{r_i - \operatorname{mean}(r_1, r_2, \dots, r_G)} {\operatorname{std}(r_1, r_2, \dots, r_G)}.$$1 to $$A_i = \frac{r_i - \operatorname{mean}(r_1, r_2, \dots, r_G)} {\operatorname{std}(r_1, r_2, \dots, r_G)}.$$2, AMC23 from $$A_i = \frac{r_i - \operatorname{mean}(r_1, r_2, \dots, r_G)} {\operatorname{std}(r_1, r_2, \dots, r_G)}.$$3 to $$A_i = \frac{r_i - \operatorname{mean}(r_1, r_2, \dots, r_G)} {\operatorname{std}(r_1, r_2, \dots, r_G)}.$$4, and OlympiadBench from $$A_i = \frac{r_i - \operatorname{mean}(r_1, r_2, \dots, r_G)} {\operatorname{std}(r_1, r_2, \dots, r_G)}.$$5 to $$A_i = \frac{r_i - \operatorname{mean}(r_1, r_2, \dots, r_G)} {\operatorname{std}(r_1, r_2, \dots, r_G)}.$$6, with an approximately $$A_i = \frac{r_i - \operatorname{mean}(r_1, r_2, \dots, r_G)} {\operatorname{std}(r_1, r_2, \dots, r_G)}.$$7 average gain over GRPO. Across nearly all benchmarks and sizes, both DHPO-A and DHPO-E outperform GRPO and GSPO consistently (Min et al., 9 Jan 2026).

The ablation on clipping strategy is notable because it isolates the trust-region design rather than the mixing rule. Branch-Specific Clip is reported to maintain consistently higher entropy in later training stages, preserving exploration and reducing premature determinism. Accuracy curves across the seven benchmarks show smoother progress with smaller oscillations than Unified Clip, while reaching similar final accuracy (Min et al., 9 Jan 2026).

5. Terminology and related formulations

The label “Token-Sequence GRPO” is not used uniformly across the literature. In the DHPO paper, the term is mapped directly onto a dynamic hybrid of token-level GRPO-style ratios and sequence-level GSPO-style ratios, making DHPO the clearest direct realization of TS-GRPO in the provided corpus (Min et al., 9 Jan 2026).

Adjacent work uses closely related constructions under different names. “$$A_i = \frac{r_i - \operatorname{mean}(r_1, r_2, \dots, r_G)} {\operatorname{std}(r_1, r_2, \dots, r_G)}.$$8-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferences” does not mention TS-GRPO, but introduces a sequence-level weight $$A_i = \frac{r_i - \operatorname{mean}(r_1, r_2, \dots, r_G)} {\operatorname{std}(r_1, r_2, \dots, r_G)}.$$9 that is learned and then applied uniformly to all tokens in a response; the paper explicitly notes that this can instantiate behaviors akin to a token-sequence weighting scheme (Wang et al., 8 Oct 2025). “On the Theory and Practice of GRPO: A Trajectory-Corrected Approach with Fast Convergence” interprets a token-sequence view as a trajectory-level treatment and maps it to TIC-GRPO, which replaces token-level ratios with a single trajectory-level probability ratio (Pang et al., 4 Aug 2025). “Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective” proposes CTPO, where the relevant TS-GRPO-style upgrade is the cumulative prefix ratio $A_i$0 together with position-adaptive clipping in log space (Zhang et al., 8 May 2026).

A different line of work uses sequence-level likelihood rather than direct token-sequence interpolation. Both TEPO papers define a token-level framework in which group-level rewards are linked to tokens through a sequence-level likelihood or Markov-likelihood ratio, and both explicitly describe TEPO as a realization of TS-GRPO (Lin et al., 10 Oct 2025, Lin et al., 14 Apr 2026). Another variant, “Token Hidden Reward,” defines TS-GRPO through per-token THR-based reweighting that steers exploration and exploitation by amplifying positive- or negative-THR tokens under the GRPO objective (Deng et al., 4 Oct 2025).

This suggests that “TS-GRPO” functions less as a single canonical algorithm than as a descriptive umbrella for GRPO-family methods that couple token-level updates with sequence-level structure.

6. Limitations and open directions

The DHPO study explicitly notes that evaluation is restricted to the Qwen3 family—Qwen3-1.7B, Qwen3-4B, and Qwen3-30B-A3B—with shared tokenizer and pretraining, so generality to other model families and tokenization schemes is not yet validated. It also states that baseline coverage is focused rather than exhaustive, and that broader comparisons across RLVR algorithms and entropy or clipping controls could reveal additional interactions (Min et al., 9 Jan 2026).

The stated future directions are correspondingly structural. They include validating DHPO across diverse architectures, tokenizers, and scaling behaviors; exploring richer variance-reduction techniques, alternative entropy controls, and learning schedules for $A_i$1; investigating length-fair sequence clipping extensions; integrating the method with KL-regularized training regimes or reward shaping; and studying robustness to verifier noise (Min et al., 9 Jan 2026).

Within that agenda, TS-GRPO remains defined by a recurring design principle rather than by a single immutable implementation: retain group-relative, critic-free RLVR training, but replace a purely token-level or purely sequence-level correction with a mechanism that explicitly couples both. In DHPO, that coupling is realized through mixed importance ratios and branch-specific trust regions; in related formulations, it appears through learnable sequence weights, cumulative prefix correction, sequence-likelihood aggregation, or token-wise reweighting. The common objective is finer credit assignment without surrendering the stability required by sparse, verifier-defined sequence rewards.