Graph-Augmented PPO for O-RAN Optimization

• GPPO is a reinforcement learning framework that augments PPO with graph-based feature extraction and action masking for topology-aware resource management in O-RAN.
• It jointly optimizes functional split selection and virtualized baseband unit placement, integrating dynamic network demands into a single decision process.
• Empirical results demonstrate up to 18% lower deployment cost and 25% higher reward compared to standard PPO approaches, ensuring consistent feasibility.

Searching arXiv for the named GPPO framework and closely related PPO/graph-augmented papers to ground the article in current preprints. Graph-Augmented Proximal Policy Optimization (GPPO) is a reinforcement learning framework for Open Radio Access Network (O-RAN) resource management that augments Proximal Policy Optimization (PPO) with Graph Neural Networks (GNNs) for topology-aware feature extraction and with action masking for combinatorial decision spaces. In the formulation reported for O-RAN, GPPO jointly optimizes functional split selection and virtualized unit placement under dynamic demands and complex topologies, rather than treating these decisions separately. Its reported empirical profile is defined by scalable joint optimization, topology-aware policy learning, and “up to 18% lower deployment cost and 25% higher reward in generalization tests, while maintaining perfect reliability” (Ngo et al., 1 Sep 2025).

1. Position within proximal policy optimization

PPO is a family of policy gradient methods that alternates between sampling data through interaction with the environment and optimizing a surrogate objective function using stochastic gradient ascent. Its defining feature is that, unlike standard policy gradient methods that perform one gradient update per data sample, PPO uses an objective that enables multiple epochs of minibatch updates. The clipped surrogate objective is

$$L^{CLIP}(\theta) = \mathbb{E}_t \left[ \min\left( r_t(\theta)\hat{A}_t,\ \text{clip}\left(r_t(\theta), 1-\epsilon, 1+\epsilon \right)\hat{A}_t \right) \right],$$

with clipping used to prevent overly large destructive updates; in the original presentation, PPO was described as having some of the benefits of TRPO while being simpler to implement, more general, and empirically better in sample complexity (Schulman et al., 2017).

GPPO preserves this PPO substrate but changes the representation and feasible-action handling around it. In the O-RAN setting, the augmentation is not a replacement for PPO’s clipped policy-ratio mechanism; rather, it inserts a graph-based feature extractor upstream of the policy and value networks and constrains exploration through action masking. This suggests that GPPO is best understood as a domain-specialized PPO architecture for graph-structured infrastructure optimization rather than as a fundamentally different policy-gradient family.

2. O-RAN formulation and decision structure

The GPPO formulation targets the joint optimization of functional split selection and virtualized baseband unit placement in O-RAN. The problem is cast as a Markov Decision Process in which the state encodes current network topology, link capacities, server resources, and per-base-station demands and SLA requirements. The action space is compositional: for every RH, the agent selects a functional split, assigns a vDU to an Edge Server, and assigns a vCU to a Regional Cloud (Ngo et al., 1 Sep 2025).

For $N$ RHs, the action vector is described as

$$\mathbf{a}_t = [ a^{\text{split}}_{1,t}, \ldots, a^{\text{split}}_{N,t}, a^{\text{DU}}_{1,t}, \ldots, a^{\text{DU}}_{N,t}, a^{\text{CU}}_{1,t}, \ldots, a^{\text{CU}}_{N,t} ].$$

This structure induces a large, combinatorial action space. The transition dynamics reflect selected placements and splits, and the reward incentivizes feasible, cost-efficient deployments. The underlying motivation is that existing solutions often address split and placement independently or do not scale to large and real-world scenarios. GPPO’s defining claim is therefore joint optimization under practical network constraints rather than isolated subproblem optimization.

3. Graph augmentation and policy architecture

The graph augmentation in GPPO models the O-RAN substrate directly as a graph. Nodes represent RHs, ESs, and RCs. RH nodes carry demand and latency requirement; ES and RC nodes carry available compute. Each node also includes positional “node order” to break GNN permutation invariance. Edges represent physical links and are annotated with delay and bandwidth (Ngo et al., 1 Sep 2025).

The node and edge feature vectors are given as

$$\mathbf{h}_i = [r_i, o_i, \lambda_i, \Delta_i], \qquad \mathbf{e}_{ij} = [b_{ij}, \delta_{ij}].$$

The feature extractor is specified as a two-layer GINEConv. Its layerwise update is

$$x'_i = \mathrm{MLP}\left(x_i + \sum_{j \in \mathcal{N}(i)} \mathrm{ReLU}(x_j + e_{ij})\right).$$

After message passing, node embeddings are mean-pooled into a global graph-level representation, which is then fed to policy and value networks. The factual significance of this design is that topology, node attributes, and edge attributes are encoded jointly before action selection. A plausible implication is that GPPO’s empirical gains depend not only on PPO’s stable optimization but also on the fact that the policy operates over a topology-aware latent state rather than over a flattened MLP input.

4. Constraints, masking, and optimization objective

GPPO couples graph encoding with explicit action masking. Actions that violate hard connectivity constraints, such as assigning a vDU or vCU to a node that is not physically reachable, are masked by zeroing logits for forbidden actions before the policy softmax. The stated benefits are accelerated learning, avoidance of infeasible exploration, and preservation of valid policy gradients (Ngo et al., 1 Sep 2025).

The optimization target is the minimization of total cost subject to assignment, connectivity, resource capacity, latency, and bandwidth constraints. The cost is decomposed into computing resource cost $J_{\rm C}$, reconfiguration cost $J_{\rm R}$, routing cost $J_{\rm L}$, and SLA violation cost $J_{\rm SLA}$, with the overall objective

$$\min \big[ J_{\rm C} + J_{\rm R} + J_{\rm L} + J_{\rm SLA} \big].$$

Training remains PPO-based. At each time step, the GNN encodes the current graph; the policy and value networks compute action distributions and value estimates; invalid options are masked; and PPO updates the policy with clipped probability ratios. The reward is defined so that infeasible actions receive a strong negative reward and can trigger early episode termination for consecutive invalid actions, while feasible actions receive

$N$0

where $N$1 is the total relaxed cost. This establishes a direct connection between lower deployment cost and higher reinforcement signal.

5. Experimental evidence on scalability and generalization

The reported experimental comparison includes DDPG, vanilla PPO, MPPO, and GPPO. The baselines are differentiated by whether they use MLP encoders, PPO, and action masking; GPPO is the PPO + action masking + GNN variant. The paper reports results on both small-scale and large-scale O-RAN scenarios, together with a generalization evaluation across five different topologies (Ngo et al., 1 Sep 2025).

Setting	GPPO result	Comparator pattern
Small-scale network (8 RH)	Deployment cost $N$2; mean reward $N$3; success rate 100%	MPPO: $N$4, $N$5; PPO: $N$6, $N$7; DDPG: $N$8
Large-scale network (64 RH)	Deployment cost $N$9; mean reward $$\mathbf{a}_t = [ a^{\text{split}}_{1,t}, \ldots, a^{\text{split}}_{N,t}, a^{\text{DU}}_{1,t}, \ldots, a^{\text{DU}}_{N,t}, a^{\text{CU}}_{1,t}, \ldots, a^{\text{CU}}_{N,t} ].$$0; success rate 100%	MPPO: cost $$\mathbf{a}_t = [ a^{\text{split}}_{1,t}, \ldots, a^{\text{split}}_{N,t}, a^{\text{DU}}_{1,t}, \ldots, a^{\text{DU}}_{N,t}, a^{\text{CU}}_{1,t}, \ldots, a^{\text{CU}}_{N,t} ].$$1, reward $$\mathbf{a}_t = [ a^{\text{split}}_{1,t}, \ldots, a^{\text{split}}_{N,t}, a^{\text{DU}}_{1,t}, \ldots, a^{\text{DU}}_{N,t}, a^{\text{CU}}_{1,t}, \ldots, a^{\text{CU}}_{N,t} ].$$2, success rate 50%; PPO/DDPG: no feasible solutions, success rate 0%
Generalization on 5 topologies	Mean cost $$\mathbf{a}_t = [ a^{\text{split}}_{1,t}, \ldots, a^{\text{split}}_{N,t}, a^{\text{DU}}_{1,t}, \ldots, a^{\text{DU}}_{N,t}, a^{\text{CU}}_{1,t}, \ldots, a^{\text{CU}}_{N,t} ].$$3; mean reward $$\mathbf{a}_t = [ a^{\text{split}}_{1,t}, \ldots, a^{\text{split}}_{N,t}, a^{\text{DU}}_{1,t}, \ldots, a^{\text{DU}}_{N,t}, a^{\text{CU}}_{1,t}, \ldots, a^{\text{CU}}_{N,t} ].$$4	MPPO: mean cost $$\mathbf{a}_t = [ a^{\text{split}}_{1,t}, \ldots, a^{\text{split}}_{N,t}, a^{\text{DU}}_{1,t}, \ldots, a^{\text{DU}}_{N,t}, a^{\text{CU}}_{1,t}, \ldots, a^{\text{CU}}_{N,t} ].$$5, mean reward $$\mathbf{a}_t = [ a^{\text{split}}_{1,t}, \ldots, a^{\text{split}}_{N,t}, a^{\text{DU}}_{1,t}, \ldots, a^{\text{DU}}_{N,t}, a^{\text{CU}}_{1,t}, \ldots, a^{\text{CU}}_{N,t} ].$$6; reported as 19% cost improvement and 25% reward improvement

The abstract summarizes these findings as “up to 18% lower deployment cost and 25% higher reward in generalization tests, while maintaining perfect reliability.” The detailed results section additionally states that GPPO is the only method that achieves consistent, feasible, and cost-optimal deployments at large scale. This suggests that the interaction between topology-aware representation and masked PPO is especially consequential when feasibility constraints become dominant.

6. Related usages and acronym ambiguity

The term “graph-augmented PPO” is not unique to the O-RAN formulation. In autonomous driving, the “Graph-based Prediction and Planning Policy Network (GP3Net)” uses a spatio-temporal graph to model interactions between traffic participants, predicts future trajectories, converts them into uncertainty-aware future occupancy maps, and then trains the policy network with PPO. Its observation is

$$\mathbf{a}_t = [ a^{\text{split}}_{1,t}, \ldots, a^{\text{split}}_{N,t}, a^{\text{DU}}_{1,t}, \ldots, a^{\text{DU}}_{N,t}, a^{\text{CU}}_{1,t}, \ldots, a^{\text{CU}}_{N,t} ].$$7

where $$\mathbf{a}_t = [ a^{\text{split}}_{1,t}, \ldots, a^{\text{split}}_{N,t}, a^{\text{DU}}_{1,t}, \ldots, a^{\text{DU}}_{N,t}, a^{\text{CU}}_{1,t}, \ldots, a^{\text{CU}}_{N,t} ].$$8 is BEV context, $$\mathbf{a}_t = [ a^{\text{split}}_{1,t}, \ldots, a^{\text{split}}_{N,t}, a^{\text{DU}}_{1,t}, \ldots, a^{\text{DU}}_{N,t}, a^{\text{CU}}_{1,t}, \ldots, a^{\text{CU}}_{N,t} ].$$9 is stacked BEV masks of recent traffic-participant locations, $$\mathbf{h}_i = [r_i, o_i, \lambda_i, \Delta_i], \qquad \mathbf{e}_{ij} = [b_{ij}, \delta_{ij}].$$0 is predicted future occupancy maps, and $$\mathbf{h}_i = [r_i, o_i, \lambda_i, \Delta_i], \qquad \mathbf{e}_{ij} = [b_{ij}, \delta_{ij}].$$1 is odometry; the results emphasize that including the prediction module improves safety in non-stationary environments (Chowdhury et al., 2023). This is a different application pattern from O-RAN GPPO, but it shows that “graph augmentation” can mean adding graph-derived predictive structure to the observation before PPO optimization.

A second nearby line of work adapts PPO to graph-structured generative sampling rather than to graph-encoded control states. “Proximal Policy Optimization for Amortized Discrete Sampling” derives and applies PPO to GFlowNets on graph-based discrete domains, introducing Ent-PPO with clipped importance-weighted soft advantages and an analytic KL penalty. There, the graph is the discrete construction space itself, not an infrastructure topology encoded by a GNN for downstream control (Zykova-Myzina et al., 14 Jun 2026). The methodological overlap is PPO, but the object of optimization differs.

There is also a terminological ambiguity in the acronym “GPPO.” “Gaussian Process Policy Optimization” uses a Gaussian process over policy parameters together with an expected improvement bonus added to the PPO clipped loss. That method augments PPO with Bayesian parameter-space exploration rather than with graph neural feature extraction (Rao et al., 2020). In the literature, therefore, “GPPO” may denote at least two distinct augmentations of PPO: graph-augmented topology-aware PPO for structured environments, and Gaussian-process-augmented PPO for exploration in parameter space.