Graph-Guided Policy Optimization

• Graph-Guided Policy Optimization is a novel class of methods that integrates graph structures into policy learning, providing an inductive bias through detailed state, edge, and planning representations.
• It leverages various graph constructions—ranging from local GNN message passing to global state-transition graphs—to produce refined optimization signals and improve decision-making quality.
• Applications include multi-agent robotics, autonomous driving, and POMDP planning, where graph-based guidance boosts performance, safety measures, and efficiency in complex environments.

Graph-Guided Policy Optimization denotes a class of methods in which graph structure directly shapes policy learning, policy evaluation, or search guidance. Across the literature, the graph may encode local neighborhoods in a graph neural network, an interaction graph over agents, an exploration graph of poses and frontiers, a global state-transition graph built from trajectories, an action-centric belief graph for POMDPs, or a directed acyclic graph of semantic reasoning states. The optimization component is correspondingly diverse: deep Q-learning, REINFORCE, A2C, PPO, WCSAC, trust region–navigated clipping, eager policy gradients for diffusion, and supervised learning from expert planning have all been coupled to graph-structured representations or graph-derived learning signals (Lai et al., 2020, Wang et al., 22 Jun 2026, Mangannavar et al., 15 Oct 2025, Zhan et al., 17 Jun 2026).

1. Scope and defining idea

A common formulation is that the graph provides the inductive bias by which the policy sees the environment and receives credit. In "Policy-GNN" (Lai et al., 2020), graph-structured node states are inputs to a meta-policy that decides, per node, the number of message-passing iterations. In G2PO, linear interaction trajectories are transformed into a global state-transition graph so that identical observations across trajectories can share value estimates and TD statistics (Wang et al., 22 Jun 2026). In GEPO, a dynamic directed graph is constructed from agent experience and graph-theoretic centrality is used to define structured intrinsic rewards, a graph-enhanced advantage function, and a dynamic discount factor (Yuan et al., 30 Oct 2025). In GammaZero, belief states are transformed into action-centric graphs and a learned graph-to-policy/value mapping guides belief-space Monte Carlo tree search (Mangannavar et al., 15 Oct 2025). In GraphPO, rollouts are represented as a directed acyclic graph with reasoning steps as edges and semantic states as nodes, so semantically equivalent reasoning paths can be merged (Zhan et al., 17 Jun 2026).

This literature therefore uses graph guidance in at least three technically distinct senses. First, the graph can be the state representation on which the policy is conditioned, as in graph-based exploration, multi-robot control, and autonomous driving (Khan et al., 2019, Chen et al., 2020, Chowdhury et al., 2023). Second, the graph can be the substrate for optimization signals such as pooled value estimates, edge-centric advantages, centrality-based rewards, or safety costs (Wang et al., 22 Jun 2026, Yuan et al., 30 Oct 2025, Yoo et al., 2022). Third, the graph can guide search or planning rather than direct action sampling, as in action-centric belief graphs for MCTS and classical graph planners paired with learned low-level policies (Mangannavar et al., 15 Oct 2025, Beeching et al., 2021).

2. Graph constructions and state abstractions

The most elementary construction is the attributed graph $G=(V,E)$ with adjacency and node features. In the formulation used by Policy-GNN, $V=\{v_1,\dots,v_n\}$ is the node set, $E \subseteq V \times V$ the edge set, $A \in \mathbb{R}^{n \times n}$ the adjacency matrix, and $X \in \mathbb{R}^{n \times m}$ the node-attribute matrix. Standard message passing is written as

$$h_v^{(l+1)} = \sigma\!\left(W^{(l)} \cdot \mathrm{AGG}\big(\{h_u^{(l)} : u \in N(v)\cup\{v\}\}\big)\right),$$

and the specific GCN instantiation used there is

$$h_v^{k} = \sigma\!\left( \sum_{u \in \{v\} \cup N_1(v)} \widetilde{A}_{uv} W_k h_u^{k-1} \right),$$

with $h_u^0 = X_u$ (Lai et al., 2020).

In robotics and autonomous systems, the graph typically reflects interaction geometry or exploration structure rather than a fixed relational dataset. Graph Policy Gradients represent robots as nodes, construct edges by spatial proximity or communication links, and use node features consisting of relative poses to nearest goals, nearest robots, and nearest obstacles (Khan et al., 2019). The exploration papers construct online graphs whose nodes include robot poses, landmarks, and frontier candidates, with node features that inject uncertainty via A-optimality, relative geometry, occupancy, and node identity (Chen et al., 2020, Chen et al., 2021). GP3Net builds a dynamic spatio-temporal graph over traffic participants, with node histories, edge features such as relative position and relative velocity, and temporal encoding via LSTMs with neighbor aggregation; predicted futures are then rasterized into occupancy maps that condition a PPO policy (Chowdhury et al., 2023). GIN uses a spatiotemporal interaction graph over surrounding vehicles, multi-hop distance-weighted adjacencies, and graph convolution followed by temporal convolution and GRU summarization to produce a shared latent social context for prediction and control (Yoo et al., 2022). The cluttered-exploration system of (Calzolari et al., 16 Apr 2025) uses a dynamically reconstructed graph with an agent node, up to eight neighboring navigation cells, and frontier nodes connected to nearest neighbors.

A different family of methods turns trajectories or beliefs into graphs. G2PO defines a global state-transition graph whose nodes are equivalence classes of observations aggregated across trajectories and whose edges are action-conditioned transitions between nodes (Wang et al., 22 Jun 2026). GEPO also maintains an online directed state-transition graph, but uses Sentence-BERT-based state abstraction and centrality scores rather than message passing (Yuan et al., 30 Oct 2025). GammaZero maps a particle belief to an action-centric heterogeneous graph containing object nodes, location nodes, predicate instance nodes, action nodes, and a global node, with features derived from $b$, $T$, and $V=\{v_1,\dots,v_n\}$0 (Mangannavar et al., 15 Oct 2025). GraphPO represents reasoning rollouts as a DAG and merges non-causal nodes when cosine similarity between semantic-state embeddings exceeds a threshold $V=\{v_1,\dots,v_n\}$1, so equivalent reasoning states can share suffixes and pooled outcome statistics (Zhan et al., 17 Jun 2026).

These designs all encode the same principle: the graph is not merely an auxiliary visualization. It is the structure over which locality, symmetry, equivalence, uncertainty, and reachability are defined.

3. Policy optimization mechanisms

The optimization layer spans both classical RL and search-guidance regimes. Policy-GNN uses a Deep Q-Network to learn a meta-policy $V=\{v_1,\dots,v_n\}$2 over aggregation depth, with replay memory size $V=\{v_1,\dots,v_n\}$3, target network updates, epsilon-greedy exploration, and a reward shaped by validation accuracy relative to a recent baseline (Lai et al., 2020). Graph Policy Gradients use vanilla policy gradients with shared graph-filter parameters and centralized team reward, relying on permutation-equivariant local aggregation for scalability and zero-shot transfer (Khan et al., 2019). The exploration framework of (Chen et al., 2020) evaluates both DQN and A2C over frontier-selection graphs, while the zero-shot exploration system of (Chen et al., 2021) uses Advantage Actor–Critic with separate policy and value g-U-Nets.

PPO-style optimization dominates in several later systems, but with distinct graph-specific roles. GP3Net trains the policy network with PPO on contextual BEV masks, past observations, future occupancy masks, and odometry, using $V=\{v_1,\dots,v_n\}$4, $V=\{v_1,\dots,v_n\}$5, GAE $V=\{v_1,\dots,v_n\}$6, and learning rate $V=\{v_1,\dots,v_n\}$7 (Chowdhury et al., 2023). The cluttered-exploration system of (Calzolari et al., 16 Apr 2025) uses PPO with a GATv2 actor–critic on a dynamically constructed exploration graph, with $V=\{v_1,\dots,v_n\}$8, PPO clip $V=\{v_1,\dots,v_n\}$9, learning rate $E \subseteq V \times V$0, rollouts per update $E \subseteq V \times V$1, mini-batches $E \subseteq V \times V$2, and update epochs per batch $E \subseteq V \times V$3. GraphPO optimizes a PPO-style surrogate over tokens, but every token on an edge shares the same graph-level edge advantage $E \subseteq V \times V$4 derived from correctness and efficiency signals on the rollout DAG (Zhan et al., 17 Jun 2026). G2PO similarly uses PPO/GRPO-style clipped updates, but defines advantages on edges of a global state-transition graph through graph-level standardized TD errors (Wang et al., 22 Jun 2026).

Other systems modify the optimization rule itself to fit graph structure. GIN uses Worst-Case SAC with a CVaR-style risk measure $E \subseteq V \times V$5 built from reward and cost critics, where the cost includes both environment collisions and a prediction-derived auxiliary interaction cost (Yoo et al., 2022). The causal-discovery system of (Liu et al., 2024) introduces trust region–navigated clipping policy optimization, in which clipping is activated only when the per-subaction Bernoulli KL divergence exceeds a threshold $E \subseteq V \times V$6. GDPO formulates discrete graph diffusion as a finite-horizon MDP over reverse denoising steps and replaces the REINFORCE estimator with an eager gradient that uses $E \subseteq V \times V$7 rather than $E \subseteq V \times V$8 (Liu et al., 2024). GammaZero does not perform online RL at deployment time; instead, it learns graph-conditioned policy and value predictors from expert demonstrations and inserts them into MCTS via a PUCT-style selection rule (Mangannavar et al., 15 Oct 2025).

A defining consequence is that graph guidance need not imply a specific optimizer. What is shared is that the policy update, target, or search prior is structurally conditioned by a graph.

4. Credit assignment, safety, and efficiency

Many graph-guided methods were motivated explicitly by deficiencies of trajectory-level credit assignment. G2PO identifies severe reward sparsity and delay in long-horizon agentic RL and addresses them by aggregating identical observations across trajectories into nodes, estimating node values by group aggregation, and defining edge-centric TD errors

$E \subseteq V \times V$9

which are then globally standardized across the entire graph to prioritize critical transitions (Wang et al., 22 Jun 2026). GEPO takes a related but distinct route: centrality scores from a state-transition graph define structured intrinsic rewards, a topology-aware advantage, and a dynamic discount factor $A \in \mathbb{R}^{n \times n}$0 that increases when the agent enters more central states (Yuan et al., 30 Oct 2025).

GraphPO pushes this logic to semantic reasoning DAGs. Node scores pool terminal correctness across equivalent states, step rewards are defined as

$A \in \mathbb{R}^{n \times n}$1

correctness advantages are standardized over outgoing comparison groups, and efficiency advantages prefer shorter paths to the same equivalence class when correctness support is present (Zhan et al., 17 Jun 2026). Graph-GRPO tackles multi-agent topology learning by sampling a group of communication graphs for each query and computing edge-level conditional success rates

$A \in \mathbb{R}^{n \times n}$2

followed by group-relative normalization

$A \in \mathbb{R}^{n \times n}$3

so easy and hard queries both produce near-zero updates when they are non-informative (Cang et al., 3 Mar 2026).

Safety-critical domains add graph-derived constraints or shields. GIN augments sparse collision signals with dense auxiliary costs computed from predicted ego–others polygon and polyline intersections; these costs feed a risk-sensitive WCSAC update and encourage early evasive behavior (Yoo et al., 2022). The cluttered-exploration method of (Calzolari et al., 16 Apr 2025) executes a shielded action $A \in \mathbb{R}^{n \times n}$4 whenever the PPO policy proposes an infeasible move, choosing the closest feasible alternative and assigning a penalty $A \in \mathbb{R}^{n \times n}$5. GP3Net conditions its policy on uncertainty-aware future occupancy maps generated from a spatio-temporal graph and reports that including the prediction module improves safety measures in non-stationary environments (Chowdhury et al., 2023).

Across these systems, graph guidance is increasingly used to shift learning signals from whole trajectories to nodes, edges, equivalence classes, or safety-relevant substructures. This suggests that the main contribution of the graph is often not representational compression alone, but finer-grained control of variance, attribution, and feasibility.

5. Representative systems and reported results

The reported empirical record spans graph learning, robot exploration, autonomous driving, combinatorial optimization, POMDP planning, graph generation, and LLM-based reasoning and agent training.

System	Domain	Reported result
Policy-GNN (Lai et al., 2020)	Node classification	Cora: $A \in \mathbb{R}^{n \times n}$6; Citeseer: $A \in \mathbb{R}^{n \times n}$7; Pubmed: $A \in \mathbb{R}^{n \times n}$8; reported $A \in \mathbb{R}^{n \times n}$9 speedup over naive per-step reconstruction/training
Graph Policy Gradients (Khan et al., 2019)	Unlabeled multi-robot motion planning	GPG’s decentralized execution reaches goals within an $X \in \mathbb{R}^{n \times m}$0-margin of CAPT in time-to-goal, approximately $X \in \mathbb{R}^{n \times m}$1–$X \in \mathbb{R}^{n \times m}$2 seconds across formations F1–F3
Exploration on graphs (Chen et al., 2020)	Autonomous exploration under uncertainty	Average decision-making time is $X \in \mathbb{R}^{n \times m}$3 s per step; A2C+GG-NN exhibits the highest exploration efficiency among learned approaches
GP3Net (Chowdhury et al., 2023)	Autonomous driving	Mean Success Rate and Driving Score improvements are approximately $X \in \mathbb{R}^{n \times m}$4 and $X \in \mathbb{R}^{n \times m}$5; in unseen new weather conditions, GP3Net completes the desired route with fewer traffic infractions
G2PO (Wang et al., 22 Jun 2026)	Long-horizon agentic RL	Success rate improvements of up to $X \in \mathbb{R}^{n \times m}$6 over GRPO on ALFWorld; WebShop success gain of $X \in \mathbb{R}^{n \times m}$7 points at 1.5B
GammaZero (Mangannavar et al., 15 Oct 2025)	POMDP planning	RockSample$X \in \mathbb{R}^{n \times m}$8: $X \in \mathbb{R}^{n \times m}$9 average return; RockSample$$h_v^{(l+1)} = \sigma\!\left(W^{(l)} \cdot \mathrm{AGG}\big(\{h_u^{(l)} : u \in N(v)\cup\{v\}\}\big)\right),$$0: $$h_v^{(l+1)} = \sigma\!\left(W^{(l)} \cdot \mathrm{AGG}\big(\{h_u^{(l)} : u \in N(v)\cup\{v\}\}\big)\right),$$1; zero-shot generalization to problems $$h_v^{(l+1)} = \sigma\!\left(W^{(l)} \cdot \mathrm{AGG}\big(\{h_u^{(l)} : u \in N(v)\cup\{v\}\}\big)\right),$$2–$$h_v^{(l+1)} = \sigma\!\left(W^{(l)} \cdot \mathrm{AGG}\big(\{h_u^{(l)} : u \in N(v)\cup\{v\}\}\big)\right),$$3 larger
GraphPO (Zhan et al., 17 Jun 2026)	RLVR for reasoning models	Average scores of $$h_v^{(l+1)} = \sigma\!\left(W^{(l)} \cdot \mathrm{AGG}\big(\{h_u^{(l)} : u \in N(v)\cup\{v\}\}\big)\right),$$4, $$h_v^{(l+1)} = \sigma\!\left(W^{(l)} \cdot \mathrm{AGG}\big(\{h_u^{(l)} : u \in N(v)\cup\{v\}\}\big)\right),$$5, and $$h_v^{(l+1)} = \sigma\!\left(W^{(l)} \cdot \mathrm{AGG}\big(\{h_u^{(l)} : u \in N(v)\cup\{v\}\}\big)\right),$$6 for Qwen2.5-7B-Math, Qwen3-8B-Base, and DeepSeek-R1-Distill-Qwen-7B under the same token budget
Graph-GRPO (Cang et al., 3 Mar 2026)	Multi-agent topology learning	MMLU $$h_v^{(l+1)} = \sigma\!\left(W^{(l)} \cdot \mathrm{AGG}\big(\{h_u^{(l)} : u \in N(v)\cup\{v\}\}\big)\right),$$7, GSM8K $$h_v^{(l+1)} = \sigma\!\left(W^{(l)} \cdot \mathrm{AGG}\big(\{h_u^{(l)} : u \in N(v)\cup\{v\}\}\big)\right),$$8, HumanEval $$h_v^{(l+1)} = \sigma\!\left(W^{(l)} \cdot \mathrm{AGG}\big(\{h_u^{(l)} : u \in N(v)\cup\{v\}\}\big)\right),$$9, average $$h_v^{k} = \sigma\!\left( \sum_{u \in \{v\} \cup N_1(v)} \widetilde{A}_{uv} W_k h_u^{k-1} \right),$$0, with average gain $$h_v^{k} = \sigma\!\left( \sum_{u \in \{v\} \cup N_1(v)} \widetilde{A}_{uv} W_k h_u^{k-1} \right),$$1 over prior SOTA
Safe cluttered exploration (Calzolari et al., 16 Apr 2025)	Safe exploration	Coverage reaches approximately $$h_v^{k} = \sigma\!\left( \sum_{u \in \{v\} \cup N_1(v)} \widetilde{A}_{uv} W_k h_u^{k-1} \right),$$2 at $$h_v^{k} = \sigma\!\left( \sum_{u \in \{v\} \cup N_1(v)} \widetilde{A}_{uv} W_k h_u^{k-1} \right),$$3 steps and approximately $$h_v^{k} = \sigma\!\left( \sum_{u \in \{v\} \cup N_1(v)} \widetilde{A}_{uv} W_k h_u^{k-1} \right),$$4 at $$h_v^{k} = \sigma\!\left( \sum_{u \in \{v\} \cup N_1(v)} \widetilde{A}_{uv} W_k h_u^{k-1} \right),$$5 steps; mean shield intervention rate is approximately $$h_v^{k} = \sigma\!\left( \sum_{u \in \{v\} \cup N_1(v)} \widetilde{A}_{uv} W_k h_u^{k-1} \right),$$6

The qualitative analyses are equally notable. Policy-GNN reports substantial heterogeneity in optimal aggregation depth, with most nodes assigned two layers but more than $$h_v^{k} = \sigma\!\left( \sum_{u \in \{v\} \cup N_1(v)} \widetilde{A}_{uv} W_k h_u^{k-1} \right),$$7 of Citeseer nodes assigned three layers and approximately $$h_v^{k} = \sigma\!\left( \sum_{u \in \{v\} \cup N_1(v)} \widetilde{A}_{uv} W_k h_u^{k-1} \right),$$8 assigned four layers (Lai et al., 2020). The exploration work of (Chen et al., 2021) reports zero-shot transfer from a single training environment to larger simulated environments and to a real building. GEPO reports absolute success rate gains of $$h_v^{k} = \sigma\!\left( \sum_{u \in \{v\} \cup N_1(v)} \widetilde{A}_{uv} W_k h_u^{k-1} \right),$$9, $h_u^0 = X_u$0, and $h_u^0 = X_u$1 over competitive baselines on ALFWorld, WebShop, and Workbench, respectively (Yuan et al., 30 Oct 2025). GDPO reports average reductions of $h_u^0 = X_u$2 in Deg/Clus/Orb on Planar and average improvement of $h_u^0 = X_u$3 on SBM relative to DiGress, while outperforming DDPO-style baselines on larger graphs (Liu et al., 2024).

These outcomes do not support a single uniform conclusion about all graph-guided methods, because the tasks, optimizers, and graphs differ substantially. They do, however, repeatedly associate graph guidance with lower-variance credit assignment, better zero-shot or cross-scale transfer, and improved efficiency under long-horizon or safety-critical structure.

6. Relation to adjacent methods, limitations, and open directions

Graph-guided policy optimization overlaps with several adjacent traditions, but differs from each in a precise way. Relative to skip connections and JK-Net-style multi-scale aggregation, Policy-GNN customizes effective depth per node rather than deepening uniformly (Lai et al., 2020). Relative to NAS over GNN depth, it emphasizes per-node depth selection rather than a single fixed architecture (Lai et al., 2020). Relative to tree-based reasoning RL, GraphPO shares suffixes across semantically equivalent states rather than only sharing prefixes (Zhan et al., 17 Jun 2026). Relative to novelty-driven exploration such as RND, GEPO uses centrality-guided intrinsic rewards that target high-impact bottlenecks rather than first-visit novelty (Yuan et al., 30 Oct 2025). Relative to PPO-style clipped updates over factorized actions, TRC in causal discovery argues that per-subaction KL-gated clipping better matches the effective trust region when actions decompose into many Bernoulli decisions (Liu et al., 2024).

Several limitations recur. State equivalence can be wrong: G2PO notes imperfect grouping under partial observability, GEPO notes aliasing from noisy text states, GraphPO analyzes false positives as introducing bias of order $h_u^0 = X_u$4, and GammaZero depends on faithful observation and transition models for graph construction (Wang et al., 22 Jun 2026, Yuan et al., 30 Oct 2025, Zhan et al., 17 Jun 2026, Mangannavar et al., 15 Oct 2025). Scalability remains nontrivial: GEPO reports an additional approximately $h_u^0 = X_u$5–$h_u^0 = X_u$6 per-step wall-clock cost from graph construction and centrality recomputation, SDGAT has quadratic attention complexity in the number of variables, and large state-transition or belief graphs can stress memory (Yuan et al., 30 Oct 2025, Liu et al., 2024, Mangannavar et al., 15 Oct 2025). Safety guarantees are often partial: GIN uses dense auxiliary costs and a risk-sensitive objective rather than formal hard constraints, and the cluttered-exploration method relies on a one-step safety shield rather than multi-step safety verification (Yoo et al., 2022, Calzolari et al., 16 Apr 2025). Expert dependence also remains important in some regimes, especially GammaZero, which learns from expert demonstrations on tractable instances (Mangannavar et al., 15 Oct 2025).

The extension space is correspondingly broad. Policy-GNN explicitly points to composite rewards balancing accuracy and efficiency, heterogeneous and temporal graphs, actor-critic or advantage methods, and joint NAS plus per-node depth selection (Lai et al., 2020). GEPO points to learned graph embeddings, hierarchical abstractions, dynamic community detection, and distributed centrality computation (Yuan et al., 30 Oct 2025). GammaZero points to hierarchical or lifted graph representations, stronger attention, self-supervised or end-to-end RL training, and improved uncertainty quantification (Mangannavar et al., 15 Oct 2025). GIN suggests differentiable risk surrogates and stronger uncertainty-aware prediction for real-world deployment (Yoo et al., 2022). More generally, the literature suggests that graph-guided policy optimization is moving toward three convergent goals: richer state abstraction, finer-grained credit assignment, and tighter integration of structural priors with safety or search.