Graph Diffusion Policy Optimization (GDPO)

• Graph Diffusion-based Policy Optimization is a framework that reinterprets reverse diffusion as a finite-horizon MDP to optimize graph structures based on terminal rewards.
• It introduces an eager policy gradient estimator that replaces per-step log probabilities with a final-sample score to reduce variance and stabilize training.
• GDPO adapts to various domains—including molecular design, neural architecture search, and UAV networks—by customizing graph representations and reward formulations.

Searching arXiv for the cited GDPO and related papers to ground the article. Graph Diffusion-based Policy Optimization (GDPO), introduced as “Graph Diffusion Policy Optimization,” is a reinforcement-learning framework for optimizing graph diffusion models against arbitrary, including non-differentiable, objectives by treating the reverse denoising chain as a stochastic policy over graph trajectories (Liu et al., 2024). In its original formulation, GDPO targets generic graph generation and molecular graph optimization, where the final reward may derive from black-box signals such as docking score or validity-related metrics rather than a differentiable training loss. Closely related later developments include Directed Graph Policy Optimization (DGPO), which extends the same RL-steered diffusion paradigm from undirected graphs to directed acyclic graphs (DAGs) for neural architecture search by incorporating topological node ordering and positional encoding (Liuliakov et al., 22 Feb 2026), and a UAV-network formulation of Graph Diffusion-based Policy Optimization that combines diffusion-based topology generation with a Stackelberg Game incentive mechanism for covert communication (Tang et al., 8 Aug 2025). Across these variants, the unifying premise is that pretrained graph diffusion models capture a structural prior, while policy-gradient fine-tuning steers sampling toward high-reward graph instances.

1. Foundational formulation

GDPO begins from the observation that, in many graph-generation settings, the operative target is not likelihood but reward. The original framework emphasizes applications such as drug design, where the relevant objective may be a non-differentiable black-box signal, for example binding affinity from a simulator or synthetic accessibility, and notes that maximum-likelihood training of graph diffusion probabilistic models cannot directly ingest such rewards (Liu et al., 2024). Predictor-guided alternatives are possible, but the framework explicitly identifies the risk that differentiable property predictors misestimate complex signals and thereby induce suboptimal or unstable control.

The central construction is to reinterpret reverse diffusion as a finite-horizon Markov decision process. In the original GDPO formulation, the state at step $t$ is

$s_t = (G_{T-t},\,T-t),$

the action is the choice of the next denoised graph,

$a_t = G_{T-t-1},$

and the policy is the reverse kernel itself,

$$\pi_\theta(a_t\mid s_t)\equiv p_\theta(G_{T-t-1}\mid G_{T-t}).$$

Transitions are deterministic, and reward is zero at intermediate steps and applied only at the terminal graph $G_0$. Under this construction, the expected return of the induced MDP is equal to the expected reward of the generated graph:

$$J_{\mathrm{RL}}(\theta)=\mathbb{E}_{\tau\sim p_\theta}\!\Bigl[\sum_t r(s_t,a_t)\Bigr] =\mathbb{E}_{G_0\sim p_\theta(G_0)}[r(G_0)].$$

This MDP view persists in later domain-specific adaptations. DGPO applies the same principle to neural architectures represented as DAGs, with the objective

$$L(\theta)=\mathbb{E}_{G\sim p_\theta}[R(G)]\to\max_\theta,$$

while the UAV-topology variant defines a topology-construction MDP over UAV positions, ground-user locations, and UAV–UAV links, again optimizing the expected final-topology reward (Liuliakov et al., 22 Feb 2026, Tang et al., 8 Aug 2025).

2. Diffusion-policy mechanics and gradient estimators

The canonical policy-gradient expression used by these methods is the score-function gradient

$$\nabla_\theta L(\theta)=\mathbb{E}_{G\sim p_\theta}\bigl[R(G)\,\nabla_\theta\log p_\theta(G)\bigr],$$

or, in trajectory form,

$$\nabla_\theta J_{\mathrm{RL}}(\theta) =\mathbb{E}_{\tau\sim p_\theta}\Bigl[r(G_0)\sum_{t=1}^T \nabla_\theta \log p_\theta(G_{t-1}\mid G_t)\Bigr].$$

The original GDPO work reports that a naïve Monte Carlo REINFORCE estimator of this form has extremely high variance for discrete graph diffusion and exhibits empirical failure in a toy experiment (Liu et al., 2024).

To address this, GDPO introduces an eager policy gradient estimator. Rather than using $\nabla_\theta \log p_\theta(G_{t-1}\mid G_t)$ directly, it replaces that term with $s_t = (G_{T-t},\,T-t),$0 so that learning acts on the probability of recovering the final sample, the object on which reward is actually computed. The estimator is

$s_t = (G_{T-t},\,T-t),$1

Reward normalization is applied by subtracting the batch mean and dividing by the empirical standard deviation, with clipping to $s_t = (G_{T-t},\,T-t),$2; the implementation summary also specifies gradient-norm clipping to at most $s_t = (G_{T-t},\,T-t),$3 (Liu et al., 2024).

DGPO retains the same RL logic but instantiates it as a reward-weighted denoising loss with clipped advantages:

$s_t = (G_{T-t},\,T-t),$4

Here the clipped advantage functions as a baseline-based variance-reduction device, and the timestep subset $s_t = (G_{T-t},\,T-t),$5 reduces trajectory-level variance (Liuliakov et al., 22 Feb 2026).

The UAV-topology formulation likewise adopts an eager policy gradient estimator over a random subset of timesteps,

$s_t = (G_{T-t},\,T-t),$6

with reward normalization by mean and standard deviation before the parameter update $s_t = (G_{T-t},\,T-t),$7 (Tang et al., 8 Aug 2025).

A common misconception is that GDPO is simply standard REINFORCE applied to diffusion. The original paper makes a sharper claim: its defining algorithmic move is the eager replacement of the usual reverse-kernel score with a final-sample score, yielding a biased but lower-variance surrogate, and the lack of a full convergence proof is explicitly acknowledged (Liu et al., 2024).

3. Architectural instantiations and graph representations

The original GDPO implementation uses the same graph-transformer backbone as DiGress and pretrains it by maximum likelihood before RL fine-tuning. During fine-tuning, it typically fixes most layers, such as message-passing GNN layers, and updates only attention or readout layers. Typical hyperparameters are domain-dependent: for general graphs, $s_t = (G_{T-t},\,T-t),$8, $s_t = (G_{T-t},\,T-t),$9, $a_t = G_{T-t-1},$0 on SBM or $a_t = G_{T-t-1},$1 on Planar, with $a_t = G_{T-t-1},$2 and $a_t = G_{T-t-1},$3; for molecules, $a_t = G_{T-t-1},$4, $a_t = G_{T-t-1},$5, $a_t = G_{T-t-1},$6, $a_t = G_{T-t-1},$7, and $a_t = G_{T-t-1},$8 (Liu et al., 2024).

DGPO modifies the graph representation itself because directed acyclic graphs are not adequately modeled by undirected diffusion assumptions. A DAG is first topologically ordered so that its adjacency matrix is strictly upper-triangular, and each node feature $a_t = G_{T-t-1},$9 is augmented with a positional encoding $$\pi_\theta(a_t\mid s_t)\equiv p_\theta(G_{T-t-1}\mid G_{T-t}).$$0 reflecting this order. The underlying diffusion backbone remains DiGress-style discrete graph diffusion, but the directional semantics are preserved by ordering and positional information. The forward chain applies categorical noise for $$\pi_\theta(a_t\mid s_t)\equiv p_\theta(G_{T-t-1}\mid G_{T-t}).$$1 steps with a cosine schedule, the reverse chain is an 8-layer graph Transformer, and final adjacency is projected to the upper-triangular part to guarantee acyclicity. The reported fine-tuning regime uses a two-phase procedure: standard cross-entropy pretraining on either 7% or 100% of the NAS search space, followed by RL fine-tuning for 60 epochs with batch size $$\pi_\theta(a_t\mid s_t)\equiv p_\theta(G_{T-t-1}\mid G_{T-t}).$$2, learning rate $$\pi_\theta(a_t\mid s_t)\equiv p_\theta(G_{T-t-1}\mid G_{T-t}).$$3 on NAS-Bench-101 or $$\pi_\theta(a_t\mid s_t)\equiv p_\theta(G_{T-t-1}\mid G_{T-t}).$$4 on NAS-Bench-201, 75% layer freezing, AdamW, and approximately 2,000 benchmark queries (Liuliakov et al., 22 Feb 2026).

The UAV-network variant adopts a different denoising parameterization. Its state contains UAV positions, altitudes, powers, ground-user locations, and the current set of UAV–UAV links. The diffusion operator is expressed through the symmetric normalized Laplacian

$$\pi_\theta(a_t\mid s_t)\equiv p_\theta(G_{T-t-1}\mid G_{T-t}).$$5

and a heat kernel

$$\pi_\theta(a_t\mid s_t)\equiv p_\theta(G_{T-t-1}\mid G_{T-t}).$$6

with node-feature propagation

$$\pi_\theta(a_t\mid s_t)\equiv p_\theta(G_{T-t-1}\mid G_{T-t}).$$7

In practice, the denoiser is a 3-layer GCN or GIN with hidden dimension 128, ReLU, layer norm, and dropout $$\pi_\theta(a_t\mid s_t)\equiv p_\theta(G_{T-t-1}\mid G_{T-t}).$$8; the reported settings are $$\pi_\theta(a_t\mid s_t)\equiv p_\theta(G_{T-t-1}\mid G_{T-t}).$$9, $G_0$0, $G_0$1, AdamW with learning rate $G_0$2, weight decay $G_0$3, and gradient-clip norm $G_0$4 (Tang et al., 8 Aug 2025).

These instantiations indicate that GDPO is not tied to a single graph architecture. A plausible implication is that the essential invariants are the trajectory-level diffusion policy, a terminal reward, and a low-variance policy-gradient surrogate, while the graph encoder-decoder can be adapted to undirected graphs, DAGs, or communication-network topologies.

4. Reward design across application domains

A defining feature of GDPO is the use of domain-specific terminal rewards rather than training-time log-likelihood alone. In the original framework for general graph generation, the reward is

$G_0$5

where each metric term lies in $G_0$6, graph-statistic distances are converted by $G_0$7, and validity is binary (Liu et al., 2024). For molecular graph optimization, the reward is

$G_0$8

with $G_0$9 clipped to $$J_{\mathrm{RL}}(\theta)=\mathbb{E}_{\tau\sim p_\theta}\!\Bigl[\sum_t r(s_t,a_t)\Bigr] =\mathbb{E}_{G_0\sim p_\theta(G_0)}[r(G_0)].$$0, $$J_{\mathrm{RL}}(\theta)=\mathbb{E}_{\tau\sim p_\theta}\!\Bigl[\sum_t r(s_t,a_t)\Bigr] =\mathbb{E}_{G_0\sim p_\theta(G_0)}[r(G_0)].$$1, $$J_{\mathrm{RL}}(\theta)=\mathbb{E}_{\tau\sim p_\theta}\!\Bigl[\sum_t r(s_t,a_t)\Bigr] =\mathbb{E}_{G_0\sim p_\theta(G_0)}[r(G_0)].$$2 if $$J_{\mathrm{RL}}(\theta)=\mathbb{E}_{\tau\sim p_\theta}\!\Bigl[\sum_t r(s_t,a_t)\Bigr] =\mathbb{E}_{G_0\sim p_\theta(G_0)}[r(G_0)].$$3 else $$J_{\mathrm{RL}}(\theta)=\mathbb{E}_{\tau\sim p_\theta}\!\Bigl[\sum_t r(s_t,a_t)\Bigr] =\mathbb{E}_{G_0\sim p_\theta(G_0)}[r(G_0)].$$4, and $$J_{\mathrm{RL}}(\theta)=\mathbb{E}_{\tau\sim p_\theta}\!\Bigl[\sum_t r(s_t,a_t)\Bigr] =\mathbb{E}_{G_0\sim p_\theta(G_0)}[r(G_0)].$$5. Evaluation then uses novel, unique molecules only, with metrics such as Hit Ratio (%) and $$J_{\mathrm{RL}}(\theta)=\mathbb{E}_{\tau\sim p_\theta}\!\Bigl[\sum_t r(s_t,a_t)\Bigr] =\mathbb{E}_{G_0\sim p_\theta(G_0)}[r(G_0)].$$6 (Liu et al., 2024).

In DGPO for neural architecture search, the reward is normalized validation accuracy. For NAS-Bench-101 it is validation accuracy on CIFAR-10, and for NAS-Bench-201 it is validation accuracy on CIFAR-10, CIFAR-100, or ImageNet-16-120, depending on the task. The summary also specifies inverse control with $$J_{\mathrm{RL}}(\theta)=\mathbb{E}_{\tau\sim p_\theta}\!\Bigl[\sum_t r(s_t,a_t)\Bigr] =\mathbb{E}_{G_0\sim p_\theta(G_0)}[r(G_0)].$$7 and a multi-objective setting

$$J_{\mathrm{RL}}(\theta)=\mathbb{E}_{\tau\sim p_\theta}\!\Bigl[\sum_t r(s_t,a_t)\Bigr] =\mathbb{E}_{G_0\sim p_\theta(G_0)}[r(G_0)].$$8

The normalized accuracy reward is

$$J_{\mathrm{RL}}(\theta)=\mathbb{E}_{\tau\sim p_\theta}\!\Bigl[\sum_t r(s_t,a_t)\Bigr] =\mathbb{E}_{G_0\sim p_\theta(G_0)}[r(G_0)].$$9

clipped to $$L(\theta)=\mathbb{E}_{G\sim p_\theta}[R(G)]\to\max_\theta,$$0. Optional parameter-count or FLOPs terms may be normalized similarly and added to $$L(\theta)=\mathbb{E}_{G\sim p_\theta}[R(G)]\to\max_\theta,$$1 (Liuliakov et al., 22 Feb 2026).

In the UAV-network setting, the topology reward is explicitly multi-objective:

$$L(\theta)=\mathbb{E}_{G\sim p_\theta}[R(G)]\to\max_\theta,$$2

where

$$L(\theta)=\mathbb{E}_{G\sim p_\theta}[R(G)]\to\max_\theta,$$3

$$L(\theta)=\mathbb{E}_{G\sim p_\theta}[R(G)]\to\max_\theta,$$4

$$L(\theta)=\mathbb{E}_{G\sim p_\theta}[R(G)]\to\max_\theta,$$5

with positive weights $$L(\theta)=\mathbb{E}_{G\sim p_\theta}[R(G)]\to\max_\theta,$$6 and reported values $$L(\theta)=\mathbb{E}_{G\sim p_\theta}[R(G)]\to\max_\theta,$$7, $$L(\theta)=\mathbb{E}_{G\sim p_\theta}[R(G)]\to\max_\theta,$$8, $$L(\theta)=\mathbb{E}_{G\sim p_\theta}[R(G)]\to\max_\theta,$$9, $$\nabla_\theta L(\theta)=\mathbb{E}_{G\sim p_\theta}\bigl[R(G)\,\nabla_\theta\log p_\theta(G)\bigr],$$0 (Tang et al., 8 Aug 2025).

Taken together, these reward constructions show that GDPO is less a single objective than a general optimization wrapper around graph diffusion, provided that the domain can supply a scalar reward on the final graph.

5. Empirical results and reported capabilities

The original GDPO paper reports results on both general graph generation and molecular graph optimization. On Planar graphs, GDPO drives Deg/Clus/Orb near $$\nabla_\theta L(\theta)=\mathbb{E}_{G\sim p_\theta}\bigl[R(G)\,\nabla_\theta\log p_\theta(G)\bigr],$$1 with an $$\nabla_\theta L(\theta)=\mathbb{E}_{G\sim p_\theta}\bigl[R(G)\,\nabla_\theta\log p_\theta(G)\bigr],$$2 average decrease from the best baseline DiGress and achieves $$\nabla_\theta L(\theta)=\mathbb{E}_{G\sim p_\theta}\bigl[R(G)\,\nabla_\theta\log p_\theta(G)\bigr],$$3; on SBM it reports a similar $$\nabla_\theta L(\theta)=\mathbb{E}_{G\sim p_\theta}\bigl[R(G)\,\nabla_\theta\log p_\theta(G)\bigr],$$4 average decrease in distribution metrics with $$\nabla_\theta L(\theta)=\mathbb{E}_{G\sim p_\theta}\bigl[R(G)\,\nabla_\theta\log p_\theta(G)\bigr],$$5. The same study states that DDPO fails catastrophically, especially for larger graphs. In molecular optimization, GDPO achieves approximately $$\nabla_\theta L(\theta)=\mathbb{E}_{G\sim p_\theta}\bigl[R(G)\,\nabla_\theta\log p_\theta(G)\bigr],$$6 Hit Ratio on ZINC250k versus approximately $$\nabla_\theta L(\theta)=\mathbb{E}_{G\sim p_\theta}\bigl[R(G)\,\nabla_\theta\log p_\theta(G)\bigr],$$7 for the next best method, with $$\nabla_\theta L(\theta)=\mathbb{E}_{G\sim p_\theta}\bigl[R(G)\,\nabla_\theta\log p_\theta(G)\bigr],$$8 versus $$\nabla_\theta L(\theta)=\mathbb{E}_{G\sim p_\theta}\bigl[R(G)\,\nabla_\theta\log p_\theta(G)\bigr],$$9; on MOSES it reports approximately $$\nabla_\theta J_{\mathrm{RL}}(\theta) =\mathbb{E}_{\tau\sim p_\theta}\Bigl[r(G_0)\sum_{t=1}^T \nabla_\theta \log p_\theta(G_{t-1}\mid G_t)\Bigr].$$0 Hit Ratio and $$\nabla_\theta J_{\mathrm{RL}}(\theta) =\mathbb{E}_{\tau\sim p_\theta}\Bigl[r(G_0)\sum_{t=1}^T \nabla_\theta \log p_\theta(G_{t-1}\mid G_t)\Bigr].$$1, with an average $$\nabla_\theta J_{\mathrm{RL}}(\theta) =\mathbb{E}_{\tau\sim p_\theta}\Bigl[r(G_0)\sum_{t=1}^T \nabla_\theta \log p_\theta(G_{t-1}\mid G_t)\Bigr].$$2 over MOOD on 4/5 targets and average $$\nabla_\theta J_{\mathrm{RL}}(\theta) =\mathbb{E}_{\tau\sim p_\theta}\Bigl[r(G_0)\sum_{t=1}^T \nabla_\theta \log p_\theta(G_{t-1}\mid G_t)\Bigr].$$3 change in docking score for the top 5% (Liu et al., 2024).

DGPO evaluates the directed extension on NAS-Bench-101 and NAS-Bench-201. The reported maximum validation accuracies are:

Setting	Reported DGPO result	Comparator note
NB101	$$\nabla_\theta J_{\mathrm{RL}}(\theta) =\mathbb{E}_{\tau\sim p_\theta}\Bigl[r(G_0)\sum_{t=1}^T \nabla_\theta \log p_\theta(G_{t-1}\mid G_t)\Bigr].$$4	Competitive
NB201 (CIFAR-10)	$$\nabla_\theta J_{\mathrm{RL}}(\theta) =\mathbb{E}_{\tau\sim p_\theta}\Bigl[r(G_0)\sum_{t=1}^T \nabla_\theta \log p_\theta(G_{t-1}\mid G_t)\Bigr].$$5	Benchmark optimum
NB201 (CIFAR-100)	$$\nabla_\theta J_{\mathrm{RL}}(\theta) =\mathbb{E}_{\tau\sim p_\theta}\Bigl[r(G_0)\sum_{t=1}^T \nabla_\theta \log p_\theta(G_{t-1}\mid G_t)\Bigr].$$6	Benchmark optimum
NB201 (ImageNet-16-120)	$$\nabla_\theta J_{\mathrm{RL}}(\theta) =\mathbb{E}_{\tau\sim p_\theta}\Bigl[r(G_0)\sum_{t=1}^T \nabla_\theta \log p_\theta(G_{t-1}\mid G_t)\Bigr].$$7	Benchmark optimum

The same report states that DGPO matches the benchmark optimum on all three NAS-Bench-201 tasks and is competitive on NAS-Bench-101. Its ablations compare filtered versus full pretraining: on NB101, $$\nabla_\theta J_{\mathrm{RL}}(\theta) =\mathbb{E}_{\tau\sim p_\theta}\Bigl[r(G_0)\sum_{t=1}^T \nabla_\theta \log p_\theta(G_{t-1}\mid G_t)\Bigr].$$8 versus $$\nabla_\theta J_{\mathrm{RL}}(\theta) =\mathbb{E}_{\tau\sim p_\theta}\Bigl[r(G_0)\sum_{t=1}^T \nabla_\theta \log p_\theta(G_{t-1}\mid G_t)\Bigr].$$9, a $\nabla_\theta \log p_\theta(G_{t-1}\mid G_t)$0 percentage-point difference; on NB201, $\nabla_\theta \log p_\theta(G_{t-1}\mid G_t)$1 versus $\nabla_\theta \log p_\theta(G_{t-1}\mid G_t)$2, effectively $\nabla_\theta \log p_\theta(G_{t-1}\mid G_t)$3 percentage-point difference. After RL fine-tuning, filtered models generate more than $\nabla_\theta \log p_\theta(G_{t-1}\mid G_t)$4 of samples above the pretraining threshold $\nabla_\theta \log p_\theta(G_{t-1}\mid G_t)$5 ($\nabla_\theta \log p_\theta(G_{t-1}\mid G_t)$6 on NB101 and $\nabla_\theta \log p_\theta(G_{t-1}\mid G_t)$7 on NB201), reaching up to $\nabla_\theta \log p_\theta(G_{t-1}\mid G_t)$8 percentage points beyond the pretraining ceiling. Bidirectional control drives mean accuracy to $\nabla_\theta \log p_\theta(G_{t-1}\mid G_t)$9–$s_t = (G_{T-t},\,T-t),$00 when maximizing and down to random-chance levels of $s_t = (G_{T-t},\,T-t),$01–$s_t = (G_{T-t},\,T-t),$02 when minimizing. In a multi-objective setting, one joint run matches the Pareto front of three single-task runs with hypervolume $s_t = (G_{T-t},\,T-t),$03 (Liuliakov et al., 22 Feb 2026).

The UAV-network formulation compares GDPO with PPO and DDPO in a $s_t = (G_{T-t},\,T-t),$04 scenario with $s_t = (G_{T-t},\,T-t),$05 UAVs and $s_t = (G_{T-t},\,T-t),$06 ground users. The reported summary is:

Method	Avg. Reward $s_t = (G_{T-t},\,T-t),$07	Connectivity / Avg. Edges / Throughput
PPO	$s_t = (G_{T-t},\,T-t),$08	$s_t = (G_{T-t},\,T-t),$09, $s_t = (G_{T-t},\,T-t),$10, $s_t = (G_{T-t},\,T-t),$11 bits/s
DDPO	$s_t = (G_{T-t},\,T-t),$12	$s_t = (G_{T-t},\,T-t),$13, $s_t = (G_{T-t},\,T-t),$14, $s_t = (G_{T-t},\,T-t),$15 bits/s
GDPO	$s_t = (G_{T-t},\,T-t),$16	$s_t = (G_{T-t},\,T-t),$17, $s_t = (G_{T-t},\,T-t),$18, $s_t = (G_{T-t},\,T-t),$19 bits/s

The accompanying description states that GDPO stabilizes quickly with low variance, systematically prunes redundant links, and that the integrated SG+GDPO approach outperforms throughput-only or cost-only schemes in Alice’s utility (Tang et al., 8 Aug 2025).

6. Interpretation, limitations, and future directions

Several interpretations recur across the literature. DGPO attributes its ability to exceed the pretraining ceiling to compositional generalization: pretraining learns local structural motifs, such as operation combinations and connectivity patterns, and RL fine-tuning recombines them under reward guidance to discover high-accuracy architectures not seen in training (Liuliakov et al., 22 Feb 2026). This suggests that the role of diffusion pretraining is not merely density estimation but acquisition of a reusable structural grammar. The UAV-network paper similarly characterizes the denoising process as diffusion-based exploration, moving from high-entropy, over-connected graphs toward low-entropy, high-reward topologies, thereby reducing the risk of local minima associated with standard policy-gradient methods (Tang et al., 8 Aug 2025).

The limitations are also explicit. The original GDPO paper states that the eager estimator is biased and that no theoretical convergence guarantee is provided; analyzing the bias-variance trade-off and establishing convergence is left to future work (Liu et al., 2024). It also notes the high per-iteration cost of sampling $s_t = (G_{T-t},\,T-t),$20 full trajectories of length $s_t = (G_{T-t},\,T-t),$21. Practical mitigations include reward normalization, clipping, fixing lower GNN layers, and trajectory reuse. DGPO adds a domain-specific caution: without explicit handling of directionality, standard graph diffusion discards the functional semantics encoded by edge direction in DAGs, which is why topological ordering and positional encoding are introduced (Liuliakov et al., 22 Feb 2026). The UAV-network formulation identifies further constraints: computational cost scaling linearly with diffusion steps $s_t = (G_{T-t},\,T-t),$22, the need to tune fixed reward weights for new scenarios, and likely difficulty scaling directly to networks with hundreds of nodes without hierarchical or block-wise diffusion (Tang et al., 8 Aug 2025).

The proposed extensions are correspondingly diverse. The original GDPO work points to learned value baselines, control variates, more efficient sampling, and staged handling of novelty reward in molecular optimization (Liu et al., 2024). DGPO proposes transfer to other DAG domains such as circuit design, workflows, and causal networks, as well as incorporation of parameter count or FLOPs into multi-objective reward and scaling to larger search spaces such as NAS-Bench-301 using offline or off-policy RL methods (Liuliakov et al., 22 Feb 2026). The UAV-network study suggests attention-based denoisers such as GAT, adaptive reward-weight learning via meta-optimization, and joint link-and-position diffusion for continuous UAV repositioning (Tang et al., 8 Aug 2025).

In aggregate, GDPO denotes a family of RL-steered graph diffusion methods whose common structure is stable: pretrain a graph diffusion model to learn domain structure, reinterpret reverse denoising as a policy, and fine-tune it with a reward-normalized policy-gradient estimator keyed to final graph quality. What varies by application is the graph representation, the denoising architecture, and the reward semantics.