Group Sub-sequence Policy Optimization (GSsPO)

• GSsPO is a reinforcement learning method that optimizes multi-turn agentic workflows by aligning gradient updates with semantically coherent Think-Action cycles.
• It recalibrates optimization granularity to sub-sequences, preserving logical structure and enhancing credit assignment compared to token- or full-sequence methods.
• Empirical results show GSsPO achieves smoother convergence and superior performance across QA benchmarks, making it effective for complex decision-making tasks.

Group Sub-sequence Policy Optimization (GSsPO) is a structure-aware reinforcement learning (RL) algorithm designed to optimize policies for multi-turn agentic workflows by aligning gradient updates with semantically meaningful decision units—namely, sub-sequences corresponding to atomic Think-Action cycles. In contrast to conventional token-level or full-sequence reinforcement learning, GSsPO recalibrates optimization granularity to sub-sequences, preserving the logical and causal structure of agentic interaction and improving credit assignment in complex reasoning tasks (Kong et al., 1 Feb 2026).

1. Motivation and Background

The emergence of LLM-based agentic workflows has shown significant advances in addressing multi-step reasoning and tool-augmented tasks. Traditional workflow synthesis paradigms typically adopt a one-shot, open-loop, code-centric approach, treating the decision process as monolithic program generation followed by execution. This paradigm—the “Static Execution Trap”—precludes conditioning on intermediate observations and ties optimization to either token-level (e.g., GRPO) or entire-sequence-level (e.g., GSPO) units. Token-level RL, by updating each output token independently, fragments the Think-Action semantic structure and disrupts inter-step dependencies. Sequence-level RL, while maintaining holistic semantic integrity, conflates credit assignment across many steps, often obscuring which segments contribute to successful outcomes. GSsPO specifically addresses this granularity mismatch by targeting sub-sequences that directly map to reasoning–action cycles, thereby preserving semantic coherence and enabling robust credit propagation (Kong et al., 1 Feb 2026).

2. Formalization and Core Concepts

Let $x \in \mathcal{D}$ denote the input prompt. At each turn $k$, the agent maintains a state $s_k$ capturing the history of prior Think-Action segments and tool outputs. The agent emits a “think” segment $y^{(k)}_{\text{think}}$ (rationale) and then an “action” segment $y^{(k)}_{\text{act}}$ (tool call or answer), forming an atomic sub-sequence. The policy $\pi_\theta$ is factorized autoregressively: $$\pi_\theta(y|x) = \prod_{t=1}^T \pi_\theta(y_t \mid y_{<t}, x).$$ A sampled response $y_i$ can be decomposed as a sequence of sub-sequences $$\mathcal{S}_i = \{s: s \text{ is one Think-Action turn in } y_i\}$$. A group of $G$ such trajectories is $k$0. Sub-sequences are token-contiguous, and each has length $k$1.

3. Algorithmic Formulation

GSsPO’s core objective targets group-averaged, sub-sequence–normalized, clipped policy gradients in the spirit of PPO, but at the sub-sequence level: $k$2 where $k$3 is the geometric mean importance ratio for the sub-sequence, and $k$4 is the standardized advantage for trajectory $k$5, normalized within its group. The per-token policy gradient is averaged over tokens within $k$6, then over sub-sequences and group members: $k$7 with

$k$8

This averaging ensures sub-sequences are not penalized for verbosity and credit is assigned at the correct semantic granularity (Kong et al., 1 Feb 2026).

4. Algorithmic Workflow and Implementation

The standard GSsPO loop proceeds as follows:

1. Sample a batch of prompts $k$9.
2. For each $s_k$0, sample $s_k$1 trajectories under $s_k$2.
3. Parse sub-sequences $s_k$3 and compute per-trajectory rewards $s_k$4, where $s_k$5.
4. Group-wise standardization yields $s_k$6.
5. For each sub-sequence, compute $s_k$7, surrogate loss $s_k$8, and the averaged gradient $s_k$9.
6. Accumulate gradients, perform optimizer step, and update $y^{(k)}_{\text{think}}$0.

Parsing sub-sequences is operationalized via unambiguous tags demarcating Think and Action segments (“> ”...“<think>”, “<tool>”...“<tool>” markers). A separate reference policy imposes a KL penalty for regularization (Kong et al., 1 Feb 2026).

5. Theoretical Significance

By aligning updates with atomic reasoning units, GSsPO provides finer-grained credit assignment than sequence-level RL, while avoiding the instability of token-level methods. The structure-aware surrogate maintains PPO-style monotonic improvement guarantees under bounded importance ratios and adequate exploration. Empirically, GSsPO demonstrates faster and more stable convergence compared with both token-level (e.g., GRPO) and sequence-level (e.g., GSPO) baselines (as shown in reward and convergence plots), and ablation studies validate the superiority of the sub-sequence granularity. The algorithm is generalizable to any multi-turn agentic process that can be factored into semantically coherent, atomic cycles—such as conversational agents, plan–execute–observe planning systems, or interactive theorem proving (Kong et al., 1 Feb 2026).

6. Empirical Validation and Applications

Workflow-R1 with GSsPO has been validated on seven QA benchmarks: NaturalQuestions, TriviaQA, PopQA (general QA), and HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle (multi-hop QA). Against direct inference, chain-of-thought, self-consistency, MedPrompt, agentic workflows MaAS/AFlow, and search-augmented RLHF baselines, Workflow-R1+GSsPO achieves superior performance (average exact-match gain ∼0.03–0.05). Workflow-R1-Search further raises performance by composing in search-based operators, demonstrating the extensibility of GSsPO-based optimization. Ablations confirm GSsPO $y^{(k)}_{\text{think}}$1 GSPO $y^{(k)}_{\text{think}}$2 GRPO across standard and search-augmented settings. GSsPO also yields smoother convergence and mitigates reward-plateauing. A plausible implication is that atomic sub-sequence optimization is optimal for agentic workflows demarcated by natural reasoning–action boundaries (Kong et al., 1 Feb 2026).

7. Comparison to Related Methods

GSsPO is situated between token-level (GRPO) and group-level sequence RL (GSPO). Agent-GSPO (Fan et al., 26 Oct 2025) leverages the GSPO framework for communication-efficient multi-agent systems but operates at the level of entire sequences, trading token-level operations for memory efficiency and enabling optimization for token economy via communication-aware rewards. In contrast, GSsPO (as implemented in Workflow-R1) uses sub-sequence–level clipping and normalization to align with the Think-Action cycle, yielding improved credit propagation and semantic alignment in agentic decision-making. Both GSsPO and GSPO employ group-wise advantage normalization, clipped surrogate objectives, and length-normalized importance ratios, but differ fundamentally in the atomic unit of optimization. GSsPO is thus distinct in its semantic granularity and is empirically validated as a more effective structure-aware RL solution in multi-turn workflow construction (Kong et al., 1 Feb 2026, Fan et al., 26 Oct 2025).

---

In summary, Group Sub-sequence Policy Optimization provides a principled, empirically validated approach for aligning RL optimization with the atomic semantics of agentic workflows, overcoming the granularity and credit assignment challenges innate to token- and sequence-level RL. Its formalization enables robust, sample-efficient learning and generalization to a broad spectrum of multi-turn decision-making tasks (Kong et al., 1 Feb 2026).