GraphGPO: Graph-based Group Policy Optimization

• GraphGPO is a reinforcement learning framework that uses explicit graph representations of trajectories to enhance step-level credit assignment.
• It aggregates group data to reduce variance during policy updates, showing significant success-rate improvements over traditional methods.
• GraphGPO is versatile, supporting varied formulations from state-transition graphs to semantic DAGs, applicable in multi-agent and reasoning tasks.

Searching arXiv for the primary paper and closely related graph-based group policy optimization formulations. Graph-based Group Policy Optimization (GraphGPO) designates a family of reinforcement-learning methods that combine group-based policy optimization with explicit graph structure over trajectories, states, transitions, or reasoning states. In long-horizon agentic reinforcement learning, the term is used most directly for methods that replace independent linear trajectories with a global state-transition graph and then compute graph-derived, step-level credit assignment; a representative instantiation is Group-Graph Policy Optimization (G2PO) for multi-turn LLM agents (Wang et al., 22 Jun 2026). Closely related formulations use the same graph-centric principle to estimate distance-to-goal on unified rollout graphs for agentic RL (Cheng et al., 26 May 2026), to merge semantically equivalent reasoning states in a directed acyclic graph for reasoning models (Zhan et al., 17 Jun 2026), and to optimize communication topologies by sampling groups of graphs in multi-agent systems (Cang et al., 3 Mar 2026). Across these formulations, the central claim is that group-based RL becomes more data-efficient and more faithful in credit assignment when the shared structure across rollouts is represented explicitly rather than treated as isolated sequences.

1. Historical setting and motivation

GraphGPO emerged from the limitations of group-based reinforcement learning methods such as GRPO when they were extended from single-output reasoning tasks to long-horizon, sparse-reward agentic settings. In WebShop, ALFWorld, and AppWorld, a LLM acts as the policy, receives an observation $o_t$, emits an action $a_t$, and often receives only a terminal reward $R$ after tens of interaction steps. Under this regime, trajectory-level training suffers from context explosion and coarse credit assignment, while step-level local grouping still inherits high-variance value estimates and myopic, state-local comparisons (Wang et al., 22 Jun 2026).

The core objection raised by the graph-based literature is that agent interaction is not naturally a set of independent lines. In WebShop and ALFWorld, many trajectories revisit the same pages, rooms, or configurations; in reasoning models, different branches often reach semantically equivalent intermediate states; in multi-agent topology learning, many candidate communication graphs differ only in a few critical edges. Treating these data as isolated samples discards shared structure and inflates variance. GraphGPO therefore starts from a different ontological commitment: trajectories are projections of a latent graph of reachable states and transitions, and policy optimization should exploit that graph directly (Cheng et al., 26 May 2026).

This motivation recurs beyond agentic RL. GraphPO for reasoning models identifies two specific deficiencies in standard RLVR: independently sampled responses often contain similar intermediate reasoning steps, and sparse final-answer rewards make it difficult to identify useful steps. Tree-based methods share prefixes, but still expand branches independently and ignore semantically equivalent later states, so they cannot share suffix information or reduce variance by pooling over equivalent nodes (Zhan et al., 17 Jun 2026).

2. Graph representations of experience

In G2PO, the basic construction begins with a group of $N$ trajectories for the same task,

$$\tau_i=(o_1^i,a_1^i,\dots,o_T^i,a_T^i), \quad i=1,\dots,N.$$

All intermediate observations are collected and clustered by exact string equality into state groups $\mathcal{G}_k$, each represented by a canonical observation $\bar{o}_k$. These groups become the nodes of a global state-transition graph, and directed edges correspond to observed action-induced transitions $(\mathcal{G}_s,a,\mathcal{G}_t)$ across any trajectory. Exact equality is used because the target environments are deterministic simulators or stable web/API environments, and all occurrences of the same observation across different trajectories are merged into one node (Wang et al., 22 Jun 2026).

A second GraphGPO formulation builds a unified state-transition graph $\mathcal{G}=(\mathcal{S},\mathcal{E})$ from rollout trajectories, but defines nodes through deterministic state signatures rather than only raw observation equality. In ALFWorld, for example, the signature may include textual observation, current location, and items held. The same work also describes an embedding-similarity fallback for noisy settings: in a noisy WebShop variant, states can be treated as identical when embedding similarity exceeds $95\%$, which preserves graph coherence under random ad insertion (Cheng et al., 26 May 2026).

GraphPO for reasoning models generalizes the graph construction further. Nodes are semantic states summarized from the full reasoning path, edges are reasoning segments, and semantically equivalent nodes are merged into equivalence classes when cosine similarity between state embeddings exceeds a threshold such as $a_t$0. Merging is virtual rather than destructive: nodes preserve their own local histories, but share suffixes through graph-level successor sets, which reallocates computation away from redundant expansions and toward novel semantic states (Zhan et al., 17 Jun 2026).

These constructions imply different notions of “state equality.” In agentic environment control, equality may be literal observation identity or deterministic signature identity. In reasoning, it is semantic equivalence under summarization and embedding. A plausible implication is that GraphGPO is best understood not as a single state representation scheme, but as a policy-optimization pattern that becomes graph-based whenever multiple rollouts can be collapsed onto shared nodes without losing the credit-assignment structure.

3. Credit assignment on the graph

The defining feature of GraphGPO is that value or advantage estimation is performed on graph objects rather than on isolated trajectories. In G2PO, each step inherits a per-step return

$a_t$1

and the value of a state group is the average over all of its occurrences: $a_t$2 This group-aggregation state-value estimator reduces variance from $a_t$3 for a single-trajectory estimate to $a_t$4 in the sparse-reward, $a_t$5 analysis. The same framework defines a node-centric advantage by comparing next-state values reachable from the same source node, and an edge-centric advantage by globally standardizing Temporal Difference errors

$a_t$6

across all edges in the graph. The final step-level signal combines episode-level, node-centric, and edge-centric terms: $a_t$7 When $a_t$8, the method degenerates to GRPO; empirically, performance peaks around $a_t$9 (Wang et al., 22 Jun 2026).

The distance-to-goal variant of GraphGPO uses a different nonparametric value surrogate. Given the unified graph, it defines

$R$0

then assigns each edge a graph-based reward

$R$1

Advantages are computed by normalizing these edge rewards within the set of outgoing transitions from the same state, and then combined with the standard episode-level group advantage. Under deterministic dynamics, the method proves that graph-based advantage is monotone with respect to progress toward the goal and that its conditional variance is no greater than trajectory-level feedback conditioned on the same $R$2 tuple (Cheng et al., 26 May 2026).

GraphPO introduces a third graph-based credit mechanism for reasoning models. It pools terminal correctness statistics over semantic equivalence classes to obtain node scores $R$3, then defines a step reward $R$4, where $R$5 is a novelty gate based on semantic similarity. Correctness advantage compares outgoing edges from a semantic state, while efficiency advantage compares incoming paths that reach the same equivalence class and favors shorter successful paths. The dual-group advantage is the sum of correctness and efficiency components, and is applied uniformly to all tokens on an edge segment (Zhan et al., 17 Jun 2026).

4. Optimization objectives and algorithmic workflow

Despite the graph restructuring, the policy-update machinery usually remains PPO-like. G2PO uses a GRPO/PPO-style clipped surrogate at the step level, with importance ratios $R$6, per-step KL regularization to a reference policy, and graph-derived advantages $R$7. The graph changes the advantage estimator rather than the underlying REINFORCE/PPO gradient form. One training iteration samples a task, initializes $R$8 environments, rolls out trajectories, constructs the graph, computes state-group values and TD errors, standardizes node- and edge-level signals, and updates the policy. All graph-side computations are reported as CPU-side and lightweight, adding roughly $R$9 overhead to training time (Wang et al., 22 Jun 2026).

The distance-based GraphGPO follows an analogous loop: collect grouped rollouts, deduplicate states into a directed graph, compute shortest-path-style distance to a goal state, assign graph-based rewards to edges, normalize them within source-state groups, combine them with episode-level group advantages, and optimize a PPO-like clipped objective with KL regularization. Reported overhead is similarly small relative to rollout and policy-update time: graph construction is about $N$0 seconds and distance/advantage computation about $N$1 seconds, compared with rollout around $N$2 seconds and policy update around $N$3 seconds (Cheng et al., 26 May 2026).

GraphPO’s workflow is more elaborate because graph construction and budget allocation are part of data collection itself. Each iteration samples a prompt, expands reasoning segments layer by layer, summarizes paths into semantic states, merges non-causal nodes by union-find if similarity exceeds $N$4, reallocates expansion budget by halving the next-layer budget of already discovered equivalence-class states, obtains final verifier rewards, computes node scores and dual-group advantages, and then applies a DAPO-style PPO objective over edge tokens. This produces a graph-native RLVR loop rather than merely a graph-based postprocessing of linear rollouts (Zhan et al., 17 Jun 2026).

5. Empirical behavior and benchmark performance

In long-horizon LLM-agent benchmarks, graph-based group policy optimization is reported to improve both success rate and training efficiency. On WebShop and ALFWorld with Qwen2.5-1.5B, G2PO raises WebShop success from about $N$5 under GRPO to about $N$6, and ALFWorld overall success from about $N$7 to about $N$8. On AppWorld with a 14B model, it reaches $N$9 success versus $$\tau_i=(o_1^i,a_1^i,\dots,o_T^i,a_T^i), \quad i=1,\dots,N.$$0 for GRPO and $$\tau_i=(o_1^i,a_1^i,\dots,o_T^i,a_T^i), \quad i=1,\dots,N.$$1 for GiGPO. The paper highlights success-rate improvements of up to $$\tau_i=(o_1^i,a_1^i,\dots,o_T^i,a_T^i), \quad i=1,\dots,N.$$2 over GRPO and reports fewer environment steps to solve tasks than GRPO and often fewer than GiGPO (Wang et al., 22 Jun 2026).

The distance-to-goal GraphGPO reports further gains across ALFWorld, WebShop, and Sokoban. With Qwen2.5-1.5B on ALFWorld, GraphGPO reaches $$\tau_i=(o_1^i,a_1^i,\dots,o_T^i,a_T^i), \quad i=1,\dots,N.$$3 overall success, compared with $$\tau_i=(o_1^i,a_1^i,\dots,o_T^i,a_T^i), \quad i=1,\dots,N.$$4 for GRPO and $$\tau_i=(o_1^i,a_1^i,\dots,o_T^i,a_T^i), \quad i=1,\dots,N.$$5 for GiGPO. On WebShop with the same base model it reaches $$\tau_i=(o_1^i,a_1^i,\dots,o_T^i,a_T^i), \quad i=1,\dots,N.$$6, versus $$\tau_i=(o_1^i,a_1^i,\dots,o_T^i,a_T^i), \quad i=1,\dots,N.$$7 for GRPO and $$\tau_i=(o_1^i,a_1^i,\dots,o_T^i,a_T^i), \quad i=1,\dots,N.$$8 for GiGPO. In Sokoban with Qwen2.5-VL-3B, it achieves $$\tau_i=(o_1^i,a_1^i,\dots,o_T^i,a_T^i), \quad i=1,\dots,N.$$9, compared with $\mathcal{G}_k$0 for GRPO and $\mathcal{G}_k$1 for GiGPO. Training curves show earlier convergence than both baselines, and ablations indicate that removing episode-level advantages hurts but does not erase the advantage of graph-based step credit (Cheng et al., 26 May 2026).

GraphPO extends the empirical picture beyond environment control into reasoning and search. On Qwen2.5-7B-Math, the reported average accuracy over AIME24, AIME25, MATH500, GPQA, and LiveCodeBench is $\mathcal{G}_k$2, compared with $\mathcal{G}_k$3 for PROS and $\mathcal{G}_k$4 for TreePO. On agentic deep-search tasks with a ReAct agent and Qwen2.5-7B, the composite metric rises to $\mathcal{G}_k$5, compared with $\mathcal{G}_k$6 for TREE-GRPO and $\mathcal{G}_k$7 for DAPO. The paper attributes part of this gain to retained exploration, shorter final responses, and lower advantage-estimation variance under semantic pooling (Zhan et al., 17 Jun 2026).

6. Related variants and conceptual boundaries

GraphGPO is not a single architecture. The graph can enter the learning system as a rollout graph, a semantic DAG, a communication topology, a topology-aware encoder, or a graph-valued output target.

Formulation	What the graph represents	Optimization role
G2PO (Wang et al., 22 Jun 2026)	Global state-transition graph over grouped trajectories	Group-aggregated values, node- and edge-centric advantages
GraphGPO (Cheng et al., 26 May 2026)	Unified state-transition graph with goal distances	Distance-to-goal rewards and edge-level graph advantages
GraphPO (Zhan et al., 17 Jun 2026)	DAG of semantic reasoning states	Equivalence-class pooling, correctness and efficiency advantages
Graph-GRPO (Cang et al., 3 Mar 2026)	Communication graph in multi-agent systems	Edge-level group-relative topology optimization
GPPO (Ngo et al., 1 Sep 2025)	Environment graph encoded by a GNN	Topology-aware state encoder plus action masking
GEPO (Yuan et al., 30 Oct 2025)	Persistent state-transition graph	Centrality-based intrinsic rewards, graph-enhanced advantages, dynamic discount
GRAPH-GRPO-LEX (Dechtiar et al., 10 Nov 2025)	Contract semantic graph output by an LLM	Graph-aware reward shaping under GRPO

A common misconception is that graph-based group policy optimization necessarily means using a graph neural network as the policy backbone. That is true of GPPO in O-RAN resource management, where a GINEConv encoder produces a topology-aware graph embedding for PPO with action masking (Ngo et al., 1 Sep 2025). It is not true of the main agentic GraphGPO formulations, where the graph is primarily a data structure for aggregation, value surrogates, and credit assignment rather than the parametric policy model itself (Wang et al., 22 Jun 2026).

Another important distinction concerns the meaning of “group.” In GRPO, a group is simply a set of sampled outputs for the same prompt. In GraphPO, groups are induced by semantic equivalence classes and correctness or efficiency comparison sets. In Graph-GRPO for multi-agent topology learning, a group is a set of sampled communication graphs for the same query, and edge advantages are computed from relative success rates within that group: $\mathcal{G}_k$8 This shows that graph-based group policy optimization can operate over action edges directly, not only over state-transition graphs (Cang et al., 3 Mar 2026).

7. Limitations, failure modes, and open directions

The main limitations recur around state abstraction, stochasticity, and scaling. G2PO assumes that exact observation equality is a good proxy for equality of underlying state, which is appropriate for deterministic simulators, stable web environments, and consistent API outputs, but becomes fragile in highly stochastic or partially observable settings. The same paper explicitly leaves approximate matching or embedding-based similarity for future work, and notes that scalability to high-dimensional multimodal observations where exact equality is rare requires further study (Wang et al., 22 Jun 2026).

The distance-based GraphGPO shares related issues. Its theoretical monotonicity and conditional variance results assume deterministic dynamics, while the empirical graph is built from a finite set of sampled trajectories and therefore induces approximation bias in $\mathcal{G}_k$9. State abstraction quality is critical: over-fragmentation yields sparse graphs and weak value propagation, while over-aggressive clustering may merge semantically distinct states and distort distances. The authors identify graph scale and state deduplication as the main open systems problems (Cheng et al., 26 May 2026).

Reasoning-oriented graph methods add a different failure mode: semantic over-merging versus under-merging. In GraphPO, low $\bar{o}_k$0 harms performance by collapsing genuinely distinct reasoning states, while $\bar{o}_k$1 degenerates the graph back to tree behavior. Pooling weight $\bar{o}_k$2 exhibits a similar bias-variance trade-off: $\bar{o}_k$3 loses variance reduction, while $\bar{o}_k$4 increases bias from semantically non-identical states. This suggests that graph-based group optimization inherits a representation problem in addition to a policy-optimization problem (Zhan et al., 17 Jun 2026).

More generally, the recent literature suggests three directions. First, graph construction may move from exact matching to learned or similarity-based state abstraction. Second, graph-based policy optimization may be integrated more tightly with search procedures, tree sampling, or asynchronous rollout regimes. Third, structural signals need not be limited to values or distances: GEPO shows that centrality can be injected as intrinsic reward, graph-enhanced advantage, and dynamic discount factor, with absolute success-rate gains of $\bar{o}_k$5, $\bar{o}_k$6, and $\bar{o}_k$7 over competitive baselines on ALFWorld, WebShop, and Workbench, respectively (Yuan et al., 30 Oct 2025). Taken together, these developments indicate that GraphGPO is becoming a broader design pattern for critic-free or lightweight-critic RL in environments where shared graph structure can be mined from experience and converted into denser, lower-variance learning signals.