Percentile-Based Delay-Aware DRL

• The paper demonstrates that PDA-DRL integrates percentile-based delay constraints directly into the DRL reward structure, achieving 15–30% reduction in worst-case delays for Wi-Fi MAPC.
• It formulates network scheduling as a finite MDP and employs actor-critic architectures with advanced reward shaping, ensuring stable policy training under non-stationary conditions.
• Benchmark evaluations in RAN slicing show that PDA-DRL outperforms mean-delay methods by reducing mean delay by 38% while minimizing resource over-allocation and meeting strict QoS targets.

Percentile-based Delay-Aware Deep Reinforcement Learning (PDA-DRL) is a family of deep reinforcement learning (DRL) methodologies that embed percentile-based delay constraints directly into the learning objective and reward structure. PDA-DRL frameworks are designed to minimize high-percentile (tail) latency in communication networks, explicitly addressing Quality of Service (QoS) requirements such as bounded probabilistic delay and tail delay constraints. These methods are primarily realized for resource scheduling tasks in modern wireless and networked systems, including Wi-Fi multi-access point coordination (MAPC) for IEEE 802.11bn networks and radio access network (RAN) slicing in Open RAN (O-RAN) architectures (Nunez et al., 25 Jul 2025, Tehrani et al., 24 Jul 2025).

1. Formal Problem Formulation and Markov Decision Process (MDP) Framework

PDA-DRL methods formulate sequential resource allocation and scheduling as finite Markov Decision Processes (MDPs).

For network scheduling (e.g., Wi-Fi MAPC (Nunez et al., 25 Jul 2025)):

• The state $\mathcal{S}_t$ is a high-dimensional vector comprising normalized per-flow metrics such as head-of-line (HOL) queue delay $\delta_i(t)$, queue occupancy $\varrho_i(t)$, and channel state information $h_i$ for each station.
• The action space $\mathcal{A}$ represents feasible scheduling decisions, e.g., selection of spatial reuse (SR) groups of AP-station pairs.
• Transition dynamics $P$ encode stochastic queue evolution driven by arrivals, packet transmission, and link conditions.
• The reward $R$ is structured to target percentile-based delay objectives.
• The discount factor $\gamma$ determines temporal prioritization in reward propagation.

In RAN slicing (O-RAN) (Tehrani et al., 24 Jul 2025), the system state aggregates statistics over recent history (QoS satisfaction fraction, moments of delay and capacity, prior allocations); actions adjust the number of physical resource blocks (PRBs) allocated to competing MVNOs per slicing slot.

The central optimization objective is the minimization of network-wide high-percentile delay, formalized as minimizing the maximum per-flow empirical $q$-percentile over all flows (e.g., $99$th percentile), subject to resource efficiency and compliance with per-operator QoS guarantees.

2. Percentile-Based Delay Metric and Reward Structuring

PDA-DRL departs from traditional average-delay objectives by targeting explicitly the network-wide or operator-specific empirical percentile-delay:

$\delta_i(t)$0

where $\delta_i(t)$1 is the empirical $\delta_i(t)$2-th percentile delay for flow or user $\delta_i(t)$3. For RAN slicing, the specific probabilistic QoS requirement is

$\delta_i(t)$4

for all users $\delta_i(t)$5 and packets $\delta_i(t)$6 associated with MVNO $\delta_i(t)$7.

Reward construction is driven by solutions to the Lagrangian dual of percentile-constrained optimization. In RAN slicing, the reward per slot incorporates the difference $\delta_i(t)$8 between empirical satisfaction probability and QoS threshold, balanced by resource consumption: $\delta_i(t)$9 and is clipped to a fixed interval for stability (Tehrani et al., 24 Jul 2025).

In Wi-Fi MAPC (Nunez et al., 25 Jul 2025), the stepwise reward

$\varrho_i(t)$0

combines an immediate shaping term to encourage clearing the oldest HOL packets (promoting fairness and short-term progress) with a long-term penalty term that decays with increasing worst-case delay, thus biasing policy search toward tail-delay minimization.

3. Policy Architecture and Training Methodology

PDA-DRL implementations employ deep actor-critic architectures, predominantly with Proximal Policy Optimization (PPO) and, in some O-RAN deployments, DQN (Nunez et al., 25 Jul 2025, Tehrani et al., 24 Jul 2025).

Feature extraction generally involves fully connected layers mapping vectorized system states to policy logits (actor head) and value estimates (critic head). For example, a typical architecture comprises two hidden layers (e.g., 64 or 128 units per layer) with nonlinear activations (tanh or ReLU), followed by domain-specific heads.

Training is performed via on-policy updates:

• Rollouts are collected in parallel environments (e.g., Gymnasium-based custom 802.11bn simulators);
• Advantages are estimated using Generalized Advantage Estimation (GAE);
• Mini-batch gradient-based updates are applied to optimize policy and value components, with appropriate regularization (entropy bonus, value function error).

Reward shaping, action masking (to filter infeasible actions), and periodic evaluation with early stopping are integral to maintain stable and efficient learning, especially under highly non-stationary traffic and channel conditions.

4. Quantitative Performance and Comparative Results

Evaluations in both Wi-Fi MAPC and O-RAN RAN slicing demonstrate that PDA-DRL policies consistently outperform heuristic and baseline DRL approaches that optimize for average delay.

In IEEE 802.11bn MAPC scenarios (Nunez et al., 25 Jul 2025):

• PDA-DRL reduces network-wide worst-case 99th-percentile delays by 15–30% compared to the best heuristic (mean 72.98–88.12 ms vs. 103.85 ms for TAT baseline).
• Across 100 random deployments, PDA-DRL sustains lower median and interquartile range of tail delays.
• Scalability is maintained across STAs from 8 to 20; the approach avoids delay blowup observed with non-learning baselines.

In RAN slicing on O-RAN (Tehrani et al., 24 Jul 2025):

• PDA-DRL achieves the target satisfaction probability (e.g., $\varrho_i(t)$1 for $\varrho_i(t)$2) with minimal PRB usage (mean 56.2 PRBs).
• Compared to mean-delay DRL (MD-DRL), PDA-DRL reduces mean delay by 38% (4.86 ms to 3.01 ms) and standard deviation by 33%.
• Heuristic and fixed-allocation approaches either fail to meet QoS or over-allocate resources.

Experimental validations on testbeds such as Colosseum confirm effectiveness in live srsRAN-based slicing environments.

5. Personalization and Federated Model Aggregation

A salient extension of PDA-DRL in distributed, multi-tenant architectures is reward-based personalization for federated learning across heterogeneous agents. In O-RAN, each MVNO trains its DRL xApp locally. Periodic aggregation is performed in a non-real-time RIC, but naive methods such as federated averaging or feature/weight-similarity aggregation underperform due to cross-environment heterogeneity.

Reward-based personalization aggregates peer policies weighted by their empirical reward when tested in the local environment: $\varrho_i(t)$3 where $\varrho_i(t)$4 is the average reward W_j achieves in environment $\varrho_i(t)$5. This method outperforms all conventional aggregation schemes, maintaining or improving local QoS adherence and resource efficiency (Tehrani et al., 24 Jul 2025).

6. Implementation Considerations for Real-World Systems

Practical deployment of PDA-DRL includes several implementation mechanisms:

• In Wi-Fi MAPC, the DRL-driven scheduler can be hosted on a central controller or a “Sharing AP,” interfacing with the distributed APs via MAPC-compatible information frames. Feature vectors are assembled from queue and channel state reported in standard MAPC control messages. Inference is real-time capable, with decision latency $\varrho_i(t)$6ms on quantized networks.
• Action masking ensures protocol compliance and robust fallback (e.g., to TAT or OP) in case of infeasibility.
• For O-RAN, DRL xApps run in the near-RT RIC, with non-RT RIC orchestrating model aggregation via the A1 interface (Tehrani et al., 24 Jul 2025).
• Online adaptation is feasible via periodic retraining and model hot-swapping; compliance with existing standards requires no PHY/MAC modification (Nunez et al., 25 Jul 2025).

7. Broader Methodological Insights and Extensions

Reward shaping is critical to ensure stable learning under percentile QoS: in scenarios where no packets meet the delay constraint, unshaped rewards yield vanishing gradients and policy stagnation. Law-of-Large Numbers-derived rewards enable reliable convergence, and tuning of shaping parameters ($\varrho_i(t)$7) governs the exploration-exploitation tradeoff.

A plausible implication is that the percentile-centric methodology underlying PDA-DRL generalizes to other QoS metrics, such as throughput or jitter—by recasting the binary reward indicator and percentile thresholds accordingly. The digital twin paradigm is recommended for offline hyperparameter sweeps and environment emulation.

• "Deep Reinforcement Learning-Based Scheduling for Wi-Fi Multi-Access Point Coordination" (Nunez et al., 25 Jul 2025)
• "Percentile-Based Deep Reinforcement Learning and Reward Based Personalization For Delay Aware RAN Slicing in O-RAN" (Tehrani et al., 24 Jul 2025)