---
title: Percentile-Based Delay-Aware DRL
url: https://www.emergentmind.com/topics/percentile-based-delay-aware-deep-reinforcement-learning-pda-drl
type: topic
---

# Percentile-Based Delay-Aware DRL

Percentile-based Delay-Aware Deep Reinforcement Learning (PDA-DRL) is a family of deep reinforcement learning (DRL) methodologies that embed percentile-based delay constraints directly into the learning objective and reward structure. PDA-DRL frameworks are designed to minimize high-percentile (tail) latency in communication networks, explicitly addressing Quality of Service (QoS) requirements such as bounded probabilistic delay and tail delay constraints. These methods are primarily realized for resource scheduling tasks in modern wireless and networked systems, including Wi-Fi multi-access point coordination (MAPC) for IEEE 802.11bn networks and radio access network (RAN) slicing in Open RAN (O-RAN) architectures [2507.19377] [2507.18111].

## 1. Formal Problem Formulation and Markov Decision Process (MDP) Framework

PDA-DRL methods formulate sequential resource allocation and scheduling as finite Markov Decision Processes (MDPs). 

For network scheduling (e.g., Wi-Fi MAPC [2507.19377]):
- The state $\mathcal{S}_t$ is a high-dimensional vector comprising normalized per-flow metrics such as head-of-line (HOL) queue delay $\delta_i(t)$, queue occupancy $\varrho_i(t)$, and channel state information $h_i$ for each station.
- The action space $\mathcal{A}$ represents feasible scheduling decisions, e.g., selection of spatial reuse (SR) groups of AP-station pairs.
- Transition dynamics $P$ encode stochastic queue evolution driven by arrivals, packet transmission, and link conditions.
- The reward $R$ is structured to target percentile-based delay objectives.
- The discount factor $\gamma$ determines temporal prioritization in reward propagation.

In RAN slicing (O-RAN) [2507.18111], the system state aggregates statistics over recent history (QoS satisfaction fraction, moments of delay and capacity, prior allocations); actions adjust the number of physical resource blocks (PRBs) allocated to competing MVNOs per slicing slot.

The central optimization objective is the minimization of network-wide high-percentile delay, formalized as minimizing the maximum per-flow empirical $q$-percentile over all flows (e.g., $99$th percentile), subject to resource efficiency and compliance with per-operator QoS guarantees.

## 2. Percentile-Based Delay Metric and Reward Structuring

PDA-DRL departs from traditional average-delay objectives by targeting explicitly the network-wide or operator-specific empirical percentile-delay:

\[
D_{q}^{\rm net} = \max_{i} P_q(d_i)
\]
where $P_q(d_i)$ is the empirical $q$-th percentile delay for flow or user $i$. For RAN slicing, the specific probabilistic QoS requirement is
\[
\Pr(d_{j^v_k,q} > D^v_{max}) < \epsilon_v
\]
for all users $j$ and packets $q$ associated with MVNO $v$.

Reward construction is driven by solutions to the Lagrangian dual of percentile-constrained optimization. In RAN slicing, the reward per slot incorporates the difference $\Delta$ between empirical satisfaction probability and QoS threshold, balanced by resource consumption:
\[
r = \begin{cases}
-\Delta \gamma_p + \exp(\zeta_p \Delta + \nu_p) (N^v_{T_S})^2, & \Delta \ge 0 \\
\Delta \gamma_n + \exp(\zeta_n \Delta + \nu_n) N^v_{T_S}, & \Delta < 0 \\
\end{cases}
\]
and is clipped to a fixed interval for stability [2507.18111].

In Wi-Fi MAPC [2507.19377], the stepwise reward
\[
r_t = r_t^{\rm sh} + r_t^{\rm lg}
\]
combines an immediate shaping term to encourage clearing the oldest HOL packets (promoting fairness and short-term progress) with a long-term penalty term that decays with increasing worst-case delay, thus biasing policy search toward tail-delay minimization.

## 3. Policy Architecture and Training Methodology

PDA-DRL implementations employ deep actor-critic architectures, predominantly with Proximal Policy Optimization (PPO) and, in some O-RAN deployments, DQN [2507.19377][2507.18111]. 

Feature extraction generally involves fully connected layers mapping vectorized system states to policy logits (actor head) and value estimates (critic head). For example, a typical architecture comprises two hidden layers (e.g., 64 or 128 units per layer) with nonlinear activations (tanh or ReLU), followed by domain-specific heads.

Training is performed via on-policy updates:
- Rollouts are collected in parallel environments (e.g., Gymnasium-based custom 802.11bn simulators);
- Advantages are estimated using Generalized Advantage Estimation (GAE);
- Mini-batch gradient-based updates are applied to optimize policy and value components, with appropriate regularization (entropy bonus, value function error).

Reward shaping, action masking (to filter infeasible actions), and periodic evaluation with early stopping are integral to maintain stable and efficient learning, especially under highly non-stationary traffic and channel conditions.

## 4. Quantitative Performance and Comparative Results

Evaluations in both Wi-Fi MAPC and O-RAN RAN slicing demonstrate that PDA-DRL policies consistently outperform heuristic and baseline DRL approaches that optimize for average delay.

In IEEE 802.11bn MAPC scenarios [2507.19377]:
- PDA-DRL reduces network-wide worst-case 99th-percentile delays by 15–30% compared to the best heuristic (mean 72.98–88.12 ms vs. 103.85 ms for TAT baseline).
- Across 100 random deployments, PDA-DRL sustains lower median and interquartile range of tail delays.
- Scalability is maintained across STAs from 8 to 20; the approach avoids delay blowup observed with non-learning baselines.

In RAN slicing on O-RAN [2507.18111]:
- PDA-DRL achieves the target satisfaction probability (e.g., $\Pr(d \le 5\,\mathrm{ms}) = 0.89$ for $\epsilon=0.1$) with minimal PRB usage (mean 56.2 PRBs).
- Compared to mean-delay DRL (MD-DRL), PDA-DRL reduces mean delay by 38% (4.86 ms to 3.01 ms) and standard deviation by 33%.
- Heuristic and fixed-allocation approaches either fail to meet QoS or over-allocate resources.

Experimental validations on testbeds such as Colosseum confirm effectiveness in live srsRAN-based slicing environments.

## 5. Personalization and Federated Model Aggregation

A salient extension of PDA-DRL in distributed, multi-tenant architectures is reward-based personalization for federated learning across heterogeneous agents. In O-RAN, each MVNO trains its DRL xApp locally. Periodic aggregation is performed in a non-real-time RIC, but naive methods such as federated averaging or feature/weight-similarity aggregation underperform due to cross-environment heterogeneity.

Reward-based personalization aggregates peer policies weighted by their empirical reward when tested in the local environment:
\[
\alpha_{i,j} = \frac{\exp(\beta \hat{R}_{i,j}^T)}{\sum_k \exp(\beta \hat{R}_{i,k}^T)}
\]
where $\hat{R}_{i,j}^T$ is the average reward W_j achieves in environment $i$. This method outperforms all conventional aggregation schemes, maintaining or improving local QoS adherence and resource efficiency [2507.18111].

## 6. Implementation Considerations for Real-World Systems

Practical deployment of PDA-DRL includes several implementation mechanisms:
- In Wi-Fi MAPC, the DRL-driven scheduler can be hosted on a central controller or a “Sharing AP,” interfacing with the distributed APs via MAPC-compatible information frames. Feature vectors are assembled from queue and channel state reported in standard MAPC control messages. Inference is real-time capable, with decision latency $<1\,$ms on quantized networks.
- Action masking ensures protocol compliance and robust fallback (e.g., to TAT or OP) in case of infeasibility.
- For O-RAN, DRL xApps run in the near-RT RIC, with non-RT RIC orchestrating model aggregation via the A1 interface [2507.18111].
- Online adaptation is feasible via periodic retraining and model hot-swapping; compliance with existing standards requires no PHY/MAC modification [2507.19377].

## 7. Broader Methodological Insights and Extensions

Reward shaping is critical to ensure stable learning under percentile QoS: in scenarios where no packets meet the delay constraint, unshaped rewards yield vanishing gradients and policy stagnation. Law-of-Large Numbers-derived rewards enable reliable convergence, and tuning of shaping parameters ($\zeta, \nu, \gamma$) governs the exploration-exploitation tradeoff.

A plausible implication is that the percentile-centric methodology underlying PDA-DRL generalizes to other QoS metrics, such as throughput or jitter—by recasting the binary reward indicator and percentile thresholds accordingly. The digital twin paradigm is recommended for offline hyperparameter sweeps and environment emulation.

## References

- "Deep Reinforcement Learning-Based Scheduling for Wi-Fi Multi-Access Point Coordination" [2507.19377]
- "Percentile-Based Deep Reinforcement Learning and Reward Based Personalization For Delay Aware RAN Slicing in O-RAN" [2507.18111]

Source: https://www.emergentmind.com/topics/percentile-based-delay-aware-deep-reinforcement-learning-pda-drl