---
title: Graph-as-Policy (GaP) Overview
url: https://www.emergentmind.com/topics/graph-as-policy-gap
type: topic
---

# Graph-as-Policy (GaP) Overview

Graph-as-Policy (GaP) denotes a family of formulations in which policy representation, execution, or optimization is organized around a graph. Recent work uses this idea in several technically distinct ways: an agentic reinforcement-learning policy operating on a global state-transition graph rather than isolated trajectories; a diffusion model treated as a policy over graph denoising steps; an executable graph-plus-constraint layer that admits or rejects actions; authorization systems in which paths, vertices, and edges are first-class policy objects; fixed-size policy graphs for partially observable control; and directed computation graphs assembled from modular robot skills [2606.22995], [2402.16302], [2512.20275], [2306.12819], [2009.02164], [2607.05369]. The common feature is that graph structure is not merely an auxiliary data structure or retrieval index: it becomes the substrate on which decisions are defined, scored, constrained, or executed.

## 1. Conceptual scope and terminological variants

GaP is not a single formalism, and the term is used unevenly across the literature. In long-horizon agentic RL, GaP means that the environment is viewed as a graph \(G=(V,E)\) whose nodes encode states or observations and whose directed edges encode transitions induced by actions; the policy can then be written as \(\pi_\theta(e\mid u)\), with \(e=(u\to v)\) selected from the current node \(u\) [2606.22995]. In graph diffusion, the same idea appears as a time-indexed policy over graph states, where reverse denoising transitions \(p_\theta(G_{t-1}\mid G_t)\) are actions and the generated terminal graph receives the reward [2402.16302]. In telecom orchestration, the Network Knowledge Graph and SHACL shapes are described as an executable policy layer that deterministically gates probabilistic plans before execution [2512.20275]. In authorization, GaP means that policy is expressed directly in graph topology and content, so that decisions depend on vertices, edges, and admissible paths rather than only on subject and resource attributes [2306.12819].

A recurring distinction is therefore between **graph-structured optimization** and **graphs as policy primitives**. In the former, the graph organizes sampled experiences or generations. In the latter, the graph itself prescribes action admissibility, message routing, or controller transitions. Robotics work makes this explicit by representing the robot policy as a directed computation graph over perception, planning, control, and verification nodes drawn from a Modular Open Robot Skill Library [2607.05369]. Classical POMDP work uses the older but closely related notion of a policy graph or finite-state controller, where nodes encode action choices and edges encode observation-conditioned controller transitions [2009.02164].

| Setting | Policy object | Representative formulation |
|---|---|---|
| Agentic RL | State-transition graph | Node values and edge advantages [2606.22995] |
| Graph generation | Reverse diffusion trajectory on graphs | \(p_\theta(G_{t-1}\mid G_t)\) as policy [2402.16302] |
| Safe orchestration | Knowledge graph plus symbolic constraints | NKG + SHACL as executable policy layer [2512.20275] |
| Authorization | Paths, vertices, and edges | Path-aware XACML4G decisions [2306.12819] |
| POMDP control | Finite-state controller / policy graph | Fixed-size policy graph improvement [2009.02164] |
| Robotics | Directed computation graph | MORSL-based task graph [2607.05369] |

Related formulations place the policy at finer graph granularity. Policy Message Passing casts graph inference as a stochastic sequential process in which per-edge agents choose one of \(K+1\) message functions, including the null action “pass no information” [1909.13196]. GPS defines a deterministic top-\(K\) policy over neighbors, so that sampling, message aggregation, node updating, and graph-level readout are all guided by learned correlation scores [2112.14482]. These works do not uniformly adopt the GaP label, but they instantiate the same move: graph operations become policy-controlled decisions rather than fixed procedures.

## 2. Formal models

In long-horizon agentic RL, the canonical formalization starts from an MDP \(M=(S,A,P,R,\gamma)\), with observations \(o\) used as proxies for hidden states. Identical observations across trajectories are grouped into state classes, yielding a graph \(G=(V,E)\) in which \(V=\{\mathcal{G}_k\}\) are observation groups and \(E=\{(\mathcal{G}_s,a,\mathcal{G}_t)\}\) are observed transitions. A step-level policy \(\pi_\theta(a\mid o)\) is then reinterpreted as an edge-conditioned policy \(\pi_\theta(e\mid u)\), while node values \(V(u)\), edge \(Q\)-values, and advantages are defined over the graph rather than over isolated rollouts [2606.22995].

Graph diffusion adopts a different state space but an analogous control interpretation. A graph sample is \(G=(X,E)\), where nodes and edges are represented by categorical one-hot variables. The forward process \(q(G_{1:T}\mid G_0)=\prod_{t=1}^T q(G_t\mid G_{t-1})\) corrupts the graph toward approximate uniform noise, and the reverse model \(p_\theta(G_{0:T})=p(G_T)\prod_{t=1}^T p_\theta(G_{t-1}\mid G_t)\) recovers samples from noise. The RL mapping is explicit: the state at time \(t\) is \(s_t=(G_{T-t},T-t)\), the action is \(a_t=G_{T-t-1}\), transitions are deterministic in diffusion time, and the reward is terminal-only, \(r(s_t,a_t)=r(G_0)\) when denoising ends [2402.16302].

Multi-agent topology learning makes the graph itself the action. For a query \(s\), the policy samples a directed communication graph \(G\) on \(N\) agents, with adjacency \(A\in\{0,1\}^{N\times N}\) and a DAG mask enforcing acyclicity. The edge-factorized policy is
\[
\pi_\theta(G\mid s)=\prod_{i=1}^N\prod_{j<i}\mathrm{Bern}(A_{ij};p_{ij}(s)),
\]
and the objective is
\[
J(\theta)=\mathbb{E}_{s\sim D,\,G\sim \pi_\theta(\cdot\mid s)}[r(s,G)].
\]
The return is defined by running the multi-agent system under the sampled topology and scoring the final answer [2603.02701].

At the message-passing level, the same graph-as-policy idea appears as a sequential latent-action model. Policy Message Passing defines the state
\[
s_t\equiv \big(v_t,e_t,u_t,\{h_t^{ij}\}_{(i,j)\in E},h_t^u\big),
\]
where \(h_t^{ij}\) is per-edge memory and \(h_t^u\) is a global shared memory. The action on each edge is a discrete message-function index \(z_t^{ij}\in\{0,\dots,K\}\), with \(z=0\) denoting “pass nothing,” and the policy factorizes over edges as \(\pi_\theta(a_t\mid s_t)=\prod_{(i,j)\in E}\pi_\theta(z_t^{ij}\mid s_t,i,j)\) [1909.13196]. GPS uses a harder deterministic variant:
\[
\pi_K(u\mid v,H^{(l)},X^0;\theta)=1 \;\text{if}\; u\in \mathrm{TopK}_v\{c_{v,u}^{(l)}:u\in N(v)\},
\]
so policy is realized as learned top-\(K\) neighbor selection rather than stochastic sampling [2112.14482].

## 3. Credit assignment, optimization, and variance reduction

A central motivation for GaP in learning systems is credit assignment under sparse or delayed rewards. In G2PO, long-horizon trajectories are transformed into a global state-transition graph, and identical observations are aggregated before value estimation. If \(R_i\) is the terminal reward of trajectory \(\tau_i\), then the discounted per-step return is
\[
v_j^i=\gamma^{T-j+1}R_i,
\]
and grouped node value is
\[
V(\mathcal{G}_k)=\frac{1}{|\mathcal{G}_k|}\sum_{o_j^i\in \mathcal{G}_k} v_j^i.
\]
This is followed by an edge-centric TD error
\[
\delta_j^i=V(\mathcal{G}_{k'})-V(\mathcal{G}_k),
\]
or more generally \(\delta(u,e)=r(u,e)+\gamma V(v)-V(u)\), and by global standardization of TD errors across the entire graph. The combined step-level advantage is
\[
A_j^i=A^{EP}(a_j^i)+w\,[A^{NC}(a_j^i)+A^{EC}(a_j^i)].
\]
On WebShop, ALFWorld, and AppWorld, this graph-level credit assignment produced success-rate improvements of up to \(22.2\%\) over GRPO; on ALFWorld with Qwen2.5-1.5B, the “All” success rate was \(95.0\%(\pm 0.8)\) for G2PO versus \(72.8\%(\pm 3.6)\) for GRPO, and on WebShop with Qwen2.5-1.5B the success rate was \(71.2\%(\pm 2.6)\) versus \(56.8\%(\pm 3.8)\) [2606.22995].

GraphPO generalizes the same logic from state-transition graphs to reasoning DAGs. Rollouts are represented as a directed acyclic graph whose nodes are semantic states and whose edges are reasoning steps. Non-causal node pairs are merged into equivalence classes when
\[
\operatorname{sim}(u,v)=\frac{\mathbf z_u^\top \mathbf z_v}{\|\mathbf z_u\|\,\|\mathbf z_v\|}\ge \kappa,
\]
with \(\kappa=0.92\) in experiments. Shared suffixes are then exposed through
\[
\mathcal C_g(u)=\mathcal C(u)\cup \{c\mid q\in \mathcal Q(u)\setminus\{u\},\,c\in \mathcal C(q)\}.
\]
Node scores are pooled over own descendants and equivalent partners, step reward is
\[
r_{\mathrm{step}}(u,v)=\bigl(S(v)-S(u)\bigr)\bigl(1-\eta(u,v)\bigr),
\]
and edge updates use a dual advantage
\[
A_{\mathrm{dual}}(e)=A_{\mathrm{cor}}(e)+\lambda_{\mathrm{eff}}A_{\mathrm{eff}}(e).
\]
The paper gives variance bounds showing that equivalence-class pooling contracts advantage-estimation variance and reports consistent gains over chain- and tree-based RLVR baselines at matched budgets; for Qwen2.5-7B-Math, average performance increased to \(40.9\), compared with \(34.1\) for DAPO and \(35\)–\(37.7\) for the tree baselines listed in the study [2606.18954].

Graph-GRPO applies group-relative credit directly to graph topology learning. For each query, a group of \(K\) topologies is sampled, and for each edge \(e_{ij}\) the conditional success rate is estimated as
\[
S_{ij}=\frac{\sum_{k=1}^K \mathbf{1}\{e_{ij}\in G_k\}\,r_k}{\sum_{k=1}^K \mathbf{1}\{e_{ij}\in G_k\}+\epsilon}.
\]
Edge-level advantage is then a z-score,
\[
A_{ij}=\frac{S_{ij}-\mu_S}{\sigma_S+\epsilon},
\]
and the critic-free surrogate is
\[
L(\theta)=\frac{1}{|E_{\mathrm{batch}}|}\sum_{(i,j)\in E_{\mathrm{batch}}}\big[-A_{ij}\log \pi_\theta(e_{ij}\mid s)+\beta D_{KL}(\pi_\theta\Vert \pi_{\mathrm{ref}})\big].
\]
This edge-wise normalization suppresses non-informative updates on easy queries and sharpens structural credit assignment on difficult ones. Across MMLU, GSM8K, AQuA, MultiArith, SVAMP, and HumanEval, Graph-GRPO achieved \(92.45\%\) average accuracy versus \(91.38\%\) for EIB-LEARNER; replacing edge-level credit with graph-level credit reduced average performance by \(1.82\) points [2603.02701].

Graph diffusion policy optimization tackles a different source of variance: the mismatch between terminal graph reward and intermediate reverse-diffusion transitions. The standard REINFORCE estimator uses \(\nabla_\theta\log p_\theta(G_{t-1}\mid G_t)\), but GDPO replaces it with \(\nabla_\theta\log p_\theta(G_0\mid G_t)\), yielding the “eager” estimator
\[
g(\theta)=\frac{1}{K}\sum_{k=1}^K \frac{T}{|\mathcal T_k|}\sum_{t\in \mathcal T_k} r(G_0^{(k)})\,\nabla_\theta \log p_\theta(G_0^{(k)}\mid G_t^{(k)}).
\]
After reward normalization and clipping to \([-5,5]\), the estimator is biased relative to the true REINFORCE gradient but is reported to have much lower variance and better convergence. On Planar graphs, GDPO achieved an \(81.97\%\) average reduction in Deg/Clus/Orb versus DiGress, and on ZINC250k it improved Novel Hit Ratio by approximately \(5.72\%\) over state-of-the-art methods [2402.16302].

## 4. Executable policy layers in governance, safety, and authorization

In governance-oriented work, GaP often means that the graph itself is the admissibility mechanism. G-SPEC defines a typed, directed Network Knowledge Graph \(G=(V,E,\tau)\) and a set \(S\) of SHACL shapes. An action plan \(A=\{a_1,\dots,a_n\}\) is simulated on hypothetical graph states \(G'_i\), and validation passes iff every step preserves entity existence and shape satisfaction:
\[
V(G,S,A)=\mathrm{pass}\;\text{if}\;\forall i:\; (targets(a_i)\subseteq V(G'_i))\wedge (G'_i\models S).
\]
The system measures Safety Violation Rate and Remediation Success Rate and reports, on a simulated \(450\)-node \(5\)G Core with \(500\) scenarios, \(SVR=0.0\%\), \(RSR=94.1\%(\pm 1.2\%)\), and hallucination detection \(0.2\%(\pm 0.1\%)\), versus \(82.4\%(\pm 2.1\%)\) remediation success for the GPT-4 zero-shot baseline. Validation latency scales empirically as \(O(k^{1.2})\), with \(142\pm18\) ms overhead at the reported operating point [2512.20275].

In path-aware authorization, GaP means that policy is expressed directly over a property graph. XACML4G formalizes
\[
G=(V,E,L_V,L_E,A_V,A_E),
\]
defines a path set \(\mathrm{Paths}_G(s,r)\) between subject and resource, and uses the decision rule
\[
D(s,r,a,ctx)=\mathrm{Permit}\;\text{if}\;\exists p\in \mathrm{Paths}_G(s,r):\Phi(p,a,ctx),
\]
otherwise Deny. The policy language adds **Meta**, **Pattern**, and **PatternCondition** elements, supports variable-length path segments via edge attributes such as MinLength and MaxLength, and permits edges to be protected resources. Enforcement is datastore-independent: the policy administration point builds a source-subset graph, the policy information point compiles path constraints into Cypher, and the policy decision point uses the query result as a rule condition [2306.12819].

A closely related graph-native ABAC model represents subjects, objects, actions, attributes, and policies inside the same graph, with policies as authorization subgraphs linked by condition edges \(\mathrm{SUB\_CON}\), \(\mathrm{ACT\_CON}\), and \(\mathrm{OBJ\_CON}\). If \(Req(pol,con)\) is the set of required conditions of type \(con\), matching is
\[
\phi_{pol}(u,r,a):=\bigwedge_{con\in CT}\forall c\in Req(pol,con):R_N(x_{con},c),
\]
and the final decision is \(D(u,r,a)=Comb(u,r,a,Match(u,r,a))\). The paper states soundness and completeness of the Cypher evaluation pipeline and illustrates the approach in a healthcare use case over Neo4j [1909.09904].

Graph-GAP extends the executable-graph idea to AI governance auditing. Requirements from UNICEF Guidance on AI and Children 3.0 are decomposed into a four-layer graph of evidence, mechanism, governance, and indicator. The minimal typed schema uses Foundation, Requirement, Risk, Harm, Control, and Metric nodes, with edges \(supports\), \(leads\_to\), \(mitigates\), and \(measures\). Gap severity is summarized by
\[
GapScore(u)=0.25E(u)+0.25M(u)+0.25G(u)+0.25K(u),
\]
with requirement-level priority
\[
Priority(r)=GapScore(r)\times\left(1-\frac{Readiness_{P80}(r)}{5}\right).
\]
The study reports that transparency, accountability, and explainability (\(R5\)) have \(GapScore=4.50\) and \(Readiness_{P80}=3.0\), while privacy and data protection (\(R3\)) show \(GapScore=4.15\) and \(Readiness_{P80}=1.0\). Reliability is quantified with Krippendorff’s \(\alpha\), weighted \(\kappa\), ICC, and bootstrap confidence intervals [2601.04216].

## 5. Policy graphs in partially observable control and robotics

The oldest direct precursor to GaP is the policy graph or finite-state controller for POMDPs. A policy graph has controller nodes \(n\in N\), each with an action distribution \(\pi(a\mid n)\) and observation-conditioned node transitions \(\tau(n'\mid n,o)\). Its value function is
\[
V(s,n)=\sum_a \pi(a\mid n)\Big[R(s,a)+\gamma\sum_{s',o,n'}T(s'\mid s,a)Z(o\mid s',a)\tau(n'\mid n,o)V(s',n')\Big],
\]
and expected return from an initial belief is \(J(b_0,n_0)=\sum_s b_0(s)V(s,n_0)\). PGI optimizes a fixed-size layered graph by alternating a forward pass that computes node occupancies \(b_{t,q}(s)\) and a backward pass that greedily selects the best action and observation-conditioned successor for each controller node. Because policy size is fixed, the report emphasizes predictable per-iteration compute and compact, understandable policies, and it states monotonic improvement for the exact finite-horizon deterministic controller update [2009.02164].

Recent robotics work revives this controller-graph perspective in an explicitly agentic form. The GaP harness for Variational Automation represents a robot policy as a directed computation graph \(\mathcal G=(V,E)\) assembled from a Modular Open Robot Skill Library containing perception, grasp planning, motion planning, vision utilities, control, and verification nodes. A task class \(\mathcal T=\langle \mathcal J,\mathcal E,\mathcal R,\mathcal O,\mathcal X,\mathcal B,\mathcal J\rangle\) is optimized through
\[
\mathcal G^*=\arg\max_{\mathcal G}\mathbb E_{x\sim \mathcal B}[\mathcal J(\pi_{\mathcal G}(a\mid \mathcal I))],
\]
with
\[
\mathcal J=w_s\cdot \mathbf 1(\mathrm{success})+w_t\cdot \Phi.
\]
A multi-agent harness partitions the language task, synthesizes subgraphs, validates types and exits, rehearses candidate graphs in internal simulation, and applies structural or parametric edits based on failure analysis [2607.05369].

The empirical program is unusually broad for a GaP system in robotics. Across eight open Variational Automation benchmarks, including grocery fulfillment, grocery packing, popcorn making, USB-C insertion, and crate washing, the method is reported to outperform the listed VLA and TAMP baselines under pose and geometry variation. In simulation, success on Fulfill Grocery Orders remained \(0.93\)–\(0.97\) across object swaps, basket swaps, permutations, and mixed variation, while the VLA baselines dropped as low as \(0.10\)–\(0.43\). In real-world trials, GaP achieved \(25/25\) on Fulfill Grocery Orders versus \(8/25\) for TipTop, \(28/30\) on Pack Grocery Items versus \(10/30\), \(18/20\) on Make Popcorn versus \(0/20\), and \(121/130\) overall on cable insertion with \(SR=0.93\). On the crate-washing simulation benchmark it reached \(0.95\) success rate and \(18\) successes per hour, close to a hand-engineered graph at \(0.99\) and \(19\) successes per hour [2607.05369].

This suggests a concrete continuity between classical policy graphs and modern graph-executed robot programs. In both cases, the graph is the controller rather than a post hoc explanation. What changes is the mechanism of construction: PGI improves a fixed-size controller via dynamic programming, whereas the robotics GaP harness synthesizes and refines typed computation graphs by agent-guided search and rehearsal.

## 6. Limitations, terminology, and research directions

The GaP label is broader than its current standardization. Several papers explicitly state that they do not use the term even though their mechanisms are conceptually equivalent or closely aligned. G2PO says its graph-level policy optimization is conceptually equivalent to GaP even though the term does not appear in the paper; XACML4G and Policy Message Passing likewise describe graph-centric policy or inference mechanisms without adopting the label [2606.22995], [2306.12819], [1909.13196]. A practical consequence is that GaP is best understood as a cross-paper design pattern rather than a settled canonical framework.

Method-specific limitations recur. In G2PO, grouping depends on accurate detection of identical observations; strict equality can under-cluster near-duplicates, while over-aggregation can bias value estimates, and graph bookkeeping can grow on very long horizons [2606.22995]. GDPO’s eager gradient is biased relative to unbiased REINFORCE, reward normalization and clipping may suppress informative extremes, and the method does not hard-enforce chemical valence during molecular generation [2402.16302]. GraphPO is sensitive to the similarity threshold \(\kappa\) and pooling coefficient \(w\), incurs overhead from summary extraction and embedding, and retains residual bias of order \(O(1-\kappa)\) [2606.18954]. Graph-GRPO has \(O(N^2)\) policy scoring through the GAT-based topology network, outputs a static per-query topology rather than a turn-level adaptive one, and yields near-zero updates when all sampled graphs in a group fail [2603.02701].

Executable-policy systems face different constraints. G-SPEC notes the expressivity limits of SHACL for complex temporal and logical requirements, dependence on synchronization between telemetry and the Network Knowledge Graph, and unresolved issues of policy conflict and evolution [2512.20275]. Graph-GAP emphasizes source-coverage limits, coder subjectivity despite multi-algorithm aggregation, modest external-signal correlations, and wide confidence intervals when unit counts are small [2601.04216]. The robotics GaP harness remains below industrial throughput targets such as \(500\) units per hour, focuses mainly on quasi-static pick-and-place style tasks, and continues to exhibit IK failures, misgrasps from accumulated kinematic error, and perception failures under challenging lighting [2607.05369].

The forward-looking agenda is correspondingly heterogeneous. Proposed directions include dynamic graph management, pruning and expansion policies, hierarchical graph policies, curriculum learning, and model-based integration for long-horizon agentic RL [2606.22995]; validity-aware constraints and hybrid approaches for graph diffusion [2402.16302]; temporal logic, active reconciliation, probabilistic verification, and multi-agent coordination for governance graphs [2512.20275]; hierarchical or sparse generators and turn-level adaptive topologies for multi-agent communication policies [2603.02701]; and richer graph grammars, formal verification, learned skill primitives, and broader contact-rich robotic task classes for graph-executed automation [2607.05369].

Taken together, these lines of work indicate that GaP is less a single algorithm than a unifying computational stance: policy can be attached to graph nodes, graph edges, graph paths, equivalence classes in a DAG, symbolic shapes over knowledge graphs, or typed computation graphs for embodied control. The central technical wager is that once the graph is promoted from data structure to policy substrate, one can obtain either finer credit assignment, stronger pre-execution guarantees, more interpretable control, or some combination of the three.

Source: https://www.emergentmind.com/topics/graph-as-policy-gap