Graph-as-Policy (GaP) Overview

• Graph-as-Policy (GaP) is a framework where decision-making policies are defined directly on graph structures, linking states, transitions, and constraints.
• GaP spans diverse applications—from agentic reinforcement learning to graph diffusion and safe orchestration—leveraging topology for improved credit assignment and decision gating.
• Researchers apply GaP in robotics and POMDP control using modular computation graphs and finite-state controllers to boost control performance and policy interpretability.

Graph-as-Policy (GaP) denotes a family of formulations in which policy representation, execution, or optimization is organized around a graph. Recent work uses this idea in several technically distinct ways: an agentic reinforcement-learning policy operating on a global state-transition graph rather than isolated trajectories; a diffusion model treated as a policy over graph denoising steps; an executable graph-plus-constraint layer that admits or rejects actions; authorization systems in which paths, vertices, and edges are first-class policy objects; fixed-size policy graphs for partially observable control; and directed computation graphs assembled from modular robot skills (Wang et al., 22 Jun 2026, Liu et al., 2024, Vijay et al., 23 Dec 2025, Mohamed et al., 2023, Pajarinen, 2020, Chen et al., 6 Jul 2026). The common feature is that graph structure is not merely an auxiliary data structure or retrieval index: it becomes the substrate on which decisions are defined, scored, constrained, or executed.

1. Conceptual scope and terminological variants

GaP is not a single formalism, and the term is used unevenly across the literature. In long-horizon agentic RL, GaP means that the environment is viewed as a graph $G=(V,E)$ whose nodes encode states or observations and whose directed edges encode transitions induced by actions; the policy can then be written as $\pi_\theta(e\mid u)$, with $e=(u\to v)$ selected from the current node $u$ (Wang et al., 22 Jun 2026). In graph diffusion, the same idea appears as a time-indexed policy over graph states, where reverse denoising transitions $p_\theta(G_{t-1}\mid G_t)$ are actions and the generated terminal graph receives the reward (Liu et al., 2024). In telecom orchestration, the Network Knowledge Graph and SHACL shapes are described as an executable policy layer that deterministically gates probabilistic plans before execution (Vijay et al., 23 Dec 2025). In authorization, GaP means that policy is expressed directly in graph topology and content, so that decisions depend on vertices, edges, and admissible paths rather than only on subject and resource attributes (Mohamed et al., 2023).

A recurring distinction is therefore between graph-structured optimization and graphs as policy primitives. In the former, the graph organizes sampled experiences or generations. In the latter, the graph itself prescribes action admissibility, message routing, or controller transitions. Robotics work makes this explicit by representing the robot policy as a directed computation graph over perception, planning, control, and verification nodes drawn from a Modular Open Robot Skill Library (Chen et al., 6 Jul 2026). Classical POMDP work uses the older but closely related notion of a policy graph or finite-state controller, where nodes encode action choices and edges encode observation-conditioned controller transitions (Pajarinen, 2020).

Setting	Policy object	Representative formulation
Agentic RL	State-transition graph	Node values and edge advantages (Wang et al., 22 Jun 2026)
Graph generation	Reverse diffusion trajectory on graphs	$p_\theta(G_{t-1}\mid G_t)$ as policy (Liu et al., 2024)
Safe orchestration	Knowledge graph plus symbolic constraints	NKG + SHACL as executable policy layer (Vijay et al., 23 Dec 2025)
Authorization	Paths, vertices, and edges	Path-aware XACML4G decisions (Mohamed et al., 2023)
POMDP control	Finite-state controller / policy graph	Fixed-size policy graph improvement (Pajarinen, 2020)
Robotics	Directed computation graph	MORSL-based task graph (Chen et al., 6 Jul 2026)

Related formulations place the policy at finer graph granularity. Policy Message Passing casts graph inference as a stochastic sequential process in which per-edge agents choose one of $K+1$ message functions, including the null action “pass no information” (Deng et al., 2019). GPS defines a deterministic top-$K$ policy over neighbors, so that sampling, message aggregation, node updating, and graph-level readout are all guided by learned correlation scores (Zhang et al., 2021). These works do not uniformly adopt the GaP label, but they instantiate the same move: graph operations become policy-controlled decisions rather than fixed procedures.

2. Formal models

In long-horizon agentic RL, the canonical formalization starts from an MDP $M=(S,A,P,R,\gamma)$, with observations $o$ used as proxies for hidden states. Identical observations across trajectories are grouped into state classes, yielding a graph $\pi_\theta(e\mid u)$0 in which $\pi_\theta(e\mid u)$1 are observation groups and $\pi_\theta(e\mid u)$2 are observed transitions. A step-level policy $\pi_\theta(e\mid u)$3 is then reinterpreted as an edge-conditioned policy $\pi_\theta(e\mid u)$4, while node values $\pi_\theta(e\mid u)$5, edge $\pi_\theta(e\mid u)$6-values, and advantages are defined over the graph rather than over isolated rollouts (Wang et al., 22 Jun 2026).

Graph diffusion adopts a different state space but an analogous control interpretation. A graph sample is $\pi_\theta(e\mid u)$7, where nodes and edges are represented by categorical one-hot variables. The forward process $\pi_\theta(e\mid u)$8 corrupts the graph toward approximate uniform noise, and the reverse model $\pi_\theta(e\mid u)$9 recovers samples from noise. The RL mapping is explicit: the state at time $e=(u\to v)$0 is $e=(u\to v)$1, the action is $e=(u\to v)$2, transitions are deterministic in diffusion time, and the reward is terminal-only, $e=(u\to v)$3 when denoising ends (Liu et al., 2024).

Multi-agent topology learning makes the graph itself the action. For a query $e=(u\to v)$4, the policy samples a directed communication graph $e=(u\to v)$5 on $e=(u\to v)$6 agents, with adjacency $e=(u\to v)$7 and a DAG mask enforcing acyclicity. The edge-factorized policy is

$e=(u\to v)$8

and the objective is

$e=(u\to v)$9

The return is defined by running the multi-agent system under the sampled topology and scoring the final answer (Cang et al., 3 Mar 2026).

At the message-passing level, the same graph-as-policy idea appears as a sequential latent-action model. Policy Message Passing defines the state

$u$0

where $u$1 is per-edge memory and $u$2 is a global shared memory. The action on each edge is a discrete message-function index $u$3, with $u$4 denoting “pass nothing,” and the policy factorizes over edges as $u$5 (Deng et al., 2019). GPS uses a harder deterministic variant: $u$6 so policy is realized as learned top-$u$7 neighbor selection rather than stochastic sampling (Zhang et al., 2021).

3. Credit assignment, optimization, and variance reduction

A central motivation for GaP in learning systems is credit assignment under sparse or delayed rewards. In G2PO, long-horizon trajectories are transformed into a global state-transition graph, and identical observations are aggregated before value estimation. If $u$8 is the terminal reward of trajectory $u$9, then the discounted per-step return is

$p_\theta(G_{t-1}\mid G_t)$0

and grouped node value is

$p_\theta(G_{t-1}\mid G_t)$1

This is followed by an edge-centric TD error

$p_\theta(G_{t-1}\mid G_t)$2

or more generally $p_\theta(G_{t-1}\mid G_t)$3, and by global standardization of TD errors across the entire graph. The combined step-level advantage is

$p_\theta(G_{t-1}\mid G_t)$4

On WebShop, ALFWorld, and AppWorld, this graph-level credit assignment produced success-rate improvements of up to $p_\theta(G_{t-1}\mid G_t)$5 over GRPO; on ALFWorld with Qwen2.5-1.5B, the “All” success rate was $p_\theta(G_{t-1}\mid G_t)$6 for G2PO versus $p_\theta(G_{t-1}\mid G_t)$7 for GRPO, and on WebShop with Qwen2.5-1.5B the success rate was $p_\theta(G_{t-1}\mid G_t)$8 versus $p_\theta(G_{t-1}\mid G_t)$9 (Wang et al., 22 Jun 2026).

GraphPO generalizes the same logic from state-transition graphs to reasoning DAGs. Rollouts are represented as a directed acyclic graph whose nodes are semantic states and whose edges are reasoning steps. Non-causal node pairs are merged into equivalence classes when

$p_\theta(G_{t-1}\mid G_t)$0

with $p_\theta(G_{t-1}\mid G_t)$1 in experiments. Shared suffixes are then exposed through

$p_\theta(G_{t-1}\mid G_t)$2

Node scores are pooled over own descendants and equivalent partners, step reward is

$p_\theta(G_{t-1}\mid G_t)$3

and edge updates use a dual advantage

$p_\theta(G_{t-1}\mid G_t)$4

The paper gives variance bounds showing that equivalence-class pooling contracts advantage-estimation variance and reports consistent gains over chain- and tree-based RLVR baselines at matched budgets; for Qwen2.5-7B-Math, average performance increased to $p_\theta(G_{t-1}\mid G_t)$5, compared with $p_\theta(G_{t-1}\mid G_t)$6 for DAPO and $p_\theta(G_{t-1}\mid G_t)$7–$p_\theta(G_{t-1}\mid G_t)$8 for the tree baselines listed in the study (Zhan et al., 17 Jun 2026).

Graph-GRPO applies group-relative credit directly to graph topology learning. For each query, a group of $p_\theta(G_{t-1}\mid G_t)$9 topologies is sampled, and for each edge $K+1$0 the conditional success rate is estimated as

$K+1$1

Edge-level advantage is then a z-score,

$K+1$2

and the critic-free surrogate is

$K+1$3

This edge-wise normalization suppresses non-informative updates on easy queries and sharpens structural credit assignment on difficult ones. Across MMLU, GSM8K, AQuA, MultiArith, SVAMP, and HumanEval, Graph-GRPO achieved $K+1$4 average accuracy versus $K+1$5 for EIB-LEARNER; replacing edge-level credit with graph-level credit reduced average performance by $K+1$6 points (Cang et al., 3 Mar 2026).

Graph diffusion policy optimization tackles a different source of variance: the mismatch between terminal graph reward and intermediate reverse-diffusion transitions. The standard REINFORCE estimator uses $K+1$7, but GDPO replaces it with $K+1$8, yielding the “eager” estimator

$K+1$9

After reward normalization and clipping to $K$0, the estimator is biased relative to the true REINFORCE gradient but is reported to have much lower variance and better convergence. On Planar graphs, GDPO achieved an $K$1 average reduction in Deg/Clus/Orb versus DiGress, and on ZINC250k it improved Novel Hit Ratio by approximately $K$2 over state-of-the-art methods (Liu et al., 2024).

4. Executable policy layers in governance, safety, and authorization

In governance-oriented work, GaP often means that the graph itself is the admissibility mechanism. G-SPEC defines a typed, directed Network Knowledge Graph $K$3 and a set $K$4 of SHACL shapes. An action plan $K$5 is simulated on hypothetical graph states $K$6, and validation passes iff every step preserves entity existence and shape satisfaction: $K$7 The system measures Safety Violation Rate and Remediation Success Rate and reports, on a simulated $K$8-node $K$9G Core with $M=(S,A,P,R,\gamma)$0 scenarios, $M=(S,A,P,R,\gamma)$1, $M=(S,A,P,R,\gamma)$2, and hallucination detection $M=(S,A,P,R,\gamma)$3, versus $M=(S,A,P,R,\gamma)$4 remediation success for the GPT-4 zero-shot baseline. Validation latency scales empirically as $M=(S,A,P,R,\gamma)$5, with $M=(S,A,P,R,\gamma)$6 ms overhead at the reported operating point (Vijay et al., 23 Dec 2025).

In path-aware authorization, GaP means that policy is expressed directly over a property graph. XACML4G formalizes

$M=(S,A,P,R,\gamma)$7

defines a path set $M=(S,A,P,R,\gamma)$8 between subject and resource, and uses the decision rule

$M=(S,A,P,R,\gamma)$9

otherwise Deny. The policy language adds Meta, Pattern, and PatternCondition elements, supports variable-length path segments via edge attributes such as MinLength and MaxLength, and permits edges to be protected resources. Enforcement is datastore-independent: the policy administration point builds a source-subset graph, the policy information point compiles path constraints into Cypher, and the policy decision point uses the query result as a rule condition (Mohamed et al., 2023).

A closely related graph-native ABAC model represents subjects, objects, actions, attributes, and policies inside the same graph, with policies as authorization subgraphs linked by condition edges $o$0, $o$1, and $o$2. If $o$3 is the set of required conditions of type $o$4, matching is

$o$5

and the final decision is $o$6. The paper states soundness and completeness of the Cypher evaluation pipeline and illustrates the approach in a healthcare use case over Neo4j (Ahmadi et al., 2019).

Graph-GAP extends the executable-graph idea to AI governance auditing. Requirements from UNICEF Guidance on AI and Children 3.0 are decomposed into a four-layer graph of evidence, mechanism, governance, and indicator. The minimal typed schema uses Foundation, Requirement, Risk, Harm, Control, and Metric nodes, with edges $o$7, $o$8, $o$9, and $\pi_\theta(e\mid u)$00. Gap severity is summarized by

$\pi_\theta(e\mid u)$01

with requirement-level priority

$\pi_\theta(e\mid u)$02

The study reports that transparency, accountability, and explainability ($\pi_\theta(e\mid u)$03) have $\pi_\theta(e\mid u)$04 and $\pi_\theta(e\mid u)$05, while privacy and data protection ($\pi_\theta(e\mid u)$06) show $\pi_\theta(e\mid u)$07 and $\pi_\theta(e\mid u)$08. Reliability is quantified with Krippendorff’s $\pi_\theta(e\mid u)$09, weighted $\pi_\theta(e\mid u)$10, ICC, and bootstrap confidence intervals (Meng, 20 Dec 2025).

5. Policy graphs in partially observable control and robotics

The oldest direct precursor to GaP is the policy graph or finite-state controller for POMDPs. A policy graph has controller nodes $\pi_\theta(e\mid u)$11, each with an action distribution $\pi_\theta(e\mid u)$12 and observation-conditioned node transitions $\pi_\theta(e\mid u)$13. Its value function is

$\pi_\theta(e\mid u)$14

and expected return from an initial belief is $\pi_\theta(e\mid u)$15. PGI optimizes a fixed-size layered graph by alternating a forward pass that computes node occupancies $\pi_\theta(e\mid u)$16 and a backward pass that greedily selects the best action and observation-conditioned successor for each controller node. Because policy size is fixed, the report emphasizes predictable per-iteration compute and compact, understandable policies, and it states monotonic improvement for the exact finite-horizon deterministic controller update (Pajarinen, 2020).

Recent robotics work revives this controller-graph perspective in an explicitly agentic form. The GaP harness for Variational Automation represents a robot policy as a directed computation graph $\pi_\theta(e\mid u)$17 assembled from a Modular Open Robot Skill Library containing perception, grasp planning, motion planning, vision utilities, control, and verification nodes. A task class $\pi_\theta(e\mid u)$18 is optimized through

$\pi_\theta(e\mid u)$19

with

$\pi_\theta(e\mid u)$20

A multi-agent harness partitions the language task, synthesizes subgraphs, validates types and exits, rehearses candidate graphs in internal simulation, and applies structural or parametric edits based on failure analysis (Chen et al., 6 Jul 2026).

The empirical program is unusually broad for a GaP system in robotics. Across eight open Variational Automation benchmarks, including grocery fulfillment, grocery packing, popcorn making, USB-C insertion, and crate washing, the method is reported to outperform the listed VLA and TAMP baselines under pose and geometry variation. In simulation, success on Fulfill Grocery Orders remained $\pi_\theta(e\mid u)$21–$\pi_\theta(e\mid u)$22 across object swaps, basket swaps, permutations, and mixed variation, while the VLA baselines dropped as low as $\pi_\theta(e\mid u)$23–$\pi_\theta(e\mid u)$24. In real-world trials, GaP achieved $\pi_\theta(e\mid u)$25 on Fulfill Grocery Orders versus $\pi_\theta(e\mid u)$26 for TipTop, $\pi_\theta(e\mid u)$27 on Pack Grocery Items versus $\pi_\theta(e\mid u)$28, $\pi_\theta(e\mid u)$29 on Make Popcorn versus $\pi_\theta(e\mid u)$30, and $\pi_\theta(e\mid u)$31 overall on cable insertion with $\pi_\theta(e\mid u)$32. On the crate-washing simulation benchmark it reached $\pi_\theta(e\mid u)$33 success rate and $\pi_\theta(e\mid u)$34 successes per hour, close to a hand-engineered graph at $\pi_\theta(e\mid u)$35 and $\pi_\theta(e\mid u)$36 successes per hour (Chen et al., 6 Jul 2026).

This suggests a concrete continuity between classical policy graphs and modern graph-executed robot programs. In both cases, the graph is the controller rather than a post hoc explanation. What changes is the mechanism of construction: PGI improves a fixed-size controller via dynamic programming, whereas the robotics GaP harness synthesizes and refines typed computation graphs by agent-guided search and rehearsal.

6. Limitations, terminology, and research directions

The GaP label is broader than its current standardization. Several papers explicitly state that they do not use the term even though their mechanisms are conceptually equivalent or closely aligned. G2PO says its graph-level policy optimization is conceptually equivalent to GaP even though the term does not appear in the paper; XACML4G and Policy Message Passing likewise describe graph-centric policy or inference mechanisms without adopting the label (Wang et al., 22 Jun 2026, Mohamed et al., 2023, Deng et al., 2019). A practical consequence is that GaP is best understood as a cross-paper design pattern rather than a settled canonical framework.

Method-specific limitations recur. In G2PO, grouping depends on accurate detection of identical observations; strict equality can under-cluster near-duplicates, while over-aggregation can bias value estimates, and graph bookkeeping can grow on very long horizons (Wang et al., 22 Jun 2026). GDPO’s eager gradient is biased relative to unbiased REINFORCE, reward normalization and clipping may suppress informative extremes, and the method does not hard-enforce chemical valence during molecular generation (Liu et al., 2024). GraphPO is sensitive to the similarity threshold $\pi_\theta(e\mid u)$37 and pooling coefficient $\pi_\theta(e\mid u)$38, incurs overhead from summary extraction and embedding, and retains residual bias of order $\pi_\theta(e\mid u)$39 (Zhan et al., 17 Jun 2026). Graph-GRPO has $\pi_\theta(e\mid u)$40 policy scoring through the GAT-based topology network, outputs a static per-query topology rather than a turn-level adaptive one, and yields near-zero updates when all sampled graphs in a group fail (Cang et al., 3 Mar 2026).

Executable-policy systems face different constraints. G-SPEC notes the expressivity limits of SHACL for complex temporal and logical requirements, dependence on synchronization between telemetry and the Network Knowledge Graph, and unresolved issues of policy conflict and evolution (Vijay et al., 23 Dec 2025). Graph-GAP emphasizes source-coverage limits, coder subjectivity despite multi-algorithm aggregation, modest external-signal correlations, and wide confidence intervals when unit counts are small (Meng, 20 Dec 2025). The robotics GaP harness remains below industrial throughput targets such as $\pi_\theta(e\mid u)$41 units per hour, focuses mainly on quasi-static pick-and-place style tasks, and continues to exhibit IK failures, misgrasps from accumulated kinematic error, and perception failures under challenging lighting (Chen et al., 6 Jul 2026).

The forward-looking agenda is correspondingly heterogeneous. Proposed directions include dynamic graph management, pruning and expansion policies, hierarchical graph policies, curriculum learning, and model-based integration for long-horizon agentic RL (Wang et al., 22 Jun 2026); validity-aware constraints and hybrid approaches for graph diffusion (Liu et al., 2024); temporal logic, active reconciliation, probabilistic verification, and multi-agent coordination for governance graphs (Vijay et al., 23 Dec 2025); hierarchical or sparse generators and turn-level adaptive topologies for multi-agent communication policies (Cang et al., 3 Mar 2026); and richer graph grammars, formal verification, learned skill primitives, and broader contact-rich robotic task classes for graph-executed automation (Chen et al., 6 Jul 2026).

Taken together, these lines of work indicate that GaP is less a single algorithm than a unifying computational stance: policy can be attached to graph nodes, graph edges, graph paths, equivalence classes in a DAG, symbolic shapes over knowledge graphs, or typed computation graphs for embodied control. The central technical wager is that once the graph is promoted from data structure to policy substrate, one can obtain either finer credit assignment, stronger pre-execution guarantees, more interpretable control, or some combination of the three.