Double Deep Q-Network Scheduling Framework

• The DDQN scheduling framework is a reinforcement learning method that leverages a double Q-network to stabilize value estimates and reduce overestimation bias.
• It customizes state, action, and reward design to optimize system objectives in domains like wireless spectrum allocation, industrial sequencing, and operating-system scheduling.
• It employs techniques such as experience replay, periodic target updates, and architectural enrichments (e.g., dueling or weighted variants) to enhance convergence and performance.

A Double Deep Q-Network (DDQN) scheduling framework is a class of reinforcement-learning schedulers that formulates sequential allocation or sequencing as a Markov decision process and learns a state–action value function with the Double DQN target, which decouples next-action selection from next-action evaluation to reduce overestimation bias. In the literature, this framework appears in wireless spectrum allocation, offloading, handover control, industrial job sequencing, packet routing, and operating-system scheduling, but its core structure remains stable: a compact state representation, a discrete action set aligned with real scheduling knobs, a reward or cost tied to system objectives, an online Q-network, a target Q-network, and replay-based training (Chou et al., 8 Sep 2025, Anh et al., 2018, Li et al., 2020).

1. Core definition and distinguishing characteristics

At its most general, a DDQN scheduling framework treats scheduling as sequential decision-making over states $s_t$, actions $a_t$, transitions, and rewards. The scheduler observes a system snapshot, selects one discrete scheduling action, receives a reward or cost, and updates an approximate action-value function $Q(s,a)$. The defining distinction from vanilla DQN is the Double DQN target: $$y_t^{\text{DDQN}} = r_t + \gamma Q_{\theta^-}\Bigl(s_{t+1}, \arg\max_{a'} Q_\theta(s_{t+1}, a')\Bigr),$$ with online parameters $\theta$ and target parameters $\theta^-$. This separation is explicitly used to address the Q-value overestimation associated with single-network max operators (Chou et al., 8 Sep 2025, Anh et al., 2018, Sun et al., 31 Mar 2025).

Across domains, the framework is characterized by a discrete operational action space rather than continuous control. In NR sidelink scheduling, the action is one of five mode–band choices, including CC over 28 GHz or 26 GHz, licensed sidelink over 28 GHz or 26 GHz, and unlicensed 5 GHz sidelink with listen-before-talk (Chou et al., 8 Sep 2025). In RF-powered backscatter cognitive radio, each action is a full frame-level allocation tuple $$(\mu,\alpha_1,\dots,\alpha_N,\beta_1,\dots,\beta_N)$$ subject to slot-budget constraints (Anh et al., 2018). In operating-system scheduling, actions correspond to concrete scheduling decisions such as selecting a ready task, adjusting task priority, or changing CPU time-slice allocation (Sun et al., 31 Mar 2025). The common pattern is that DDQN is most natural when scheduling knobs can be discretized into a manageable set.

A second distinguishing feature is objective shaping. Some frameworks directly maximize throughput or completion reward; others minimize blocking, delay, energy, or switching cost through scalarized rewards or constrained formulations. The NR sidelink framework is posed as a constrained MDP minimizing long-term blocking probability under a throughput constraint, but trained with an instantaneous rate-based reward equal to achieved capacity on success and $0$ on blocking (Chou et al., 8 Sep 2025). This suggests that DDQN scheduling is often implemented with surrogate rewards that encode several system-level objectives in a single scalar signal.

2. State, action, and reward design

The quality of a DDQN scheduler is largely determined by how state, action, and reward are engineered. The literature repeatedly converges on a design in which state exposes queueing, resource availability, channel or service quality, and a small amount of history.

A detailed example is the NR sidelink scheduler, whose state is a 7-dimensional real vector: $$s_t = \bigl[q_t,\ \rho_{\text{CC-28G},t},\ \rho_{\text{CC-26G},t},\ \rho_{\text{SL-L-28G},t},\ \rho_{\text{SL-L-26G},t},\ \rho_{\text{SL-U-5G},t},\ \eta_t\bigr],$$ where $q_t$ is normalized queue occupancy, $a_t$0 are residual resource ratios for each transmission option, and $a_t$1 is Wi‑Fi idle probability. The action space is

$a_t$2

mapping to the five mode–band options, and the reward is

$a_t$3

This design explicitly exposes queue state, per-band load, and coexistence conditions to the agent while making blocking an implicit penalty through zero reward (Chou et al., 8 Sep 2025).

Other domains follow the same template with domain-specific variables. In dueling DDQN handover for LEO satellite networks, the state includes achievable rates to candidate satellites, the visible candidate set, satellite load, and the previously associated satellite; the action is the selected serving satellite; and the reward scalarizes throughput, blocking, and switching cost with adaptive weights $a_t$4 (Chou et al., 4 May 2026). In multi-server vehicular MEC, the per-server DDQN state includes vectors of task sizes and priorities for received offloading requests plus the remaining computing-resource ratio, while the action is a binary accept/reject vector over candidate offloading tasks, and reward combines immediate processed priority and expected benefit of accepted tasks (Wang et al., 2024). In federated IoT offloading, each agent’s state includes queue length, channel gain, task size, CPU demand, energy budget, and CPU budget; the DDQN action is local versus edge versus cloud offloading; and the immediate cost is obtained after solving a convex optimization for local CPU frequency or transmit power (Zarandi et al., 2021).

The same design principle extends beyond explicitly DDQN-implemented papers. The wireless scheduling formulation for distributed networked control derives a countable state from age-like variables that summarize time since successful uplink and downlink transmissions, then couples action choice to a per-step quadratic control-and-communication cost (Pang et al., 2021). This suggests a broader principle: effective DDQN schedulers do not require raw system state if a finite sufficient summary can be constructed.

3. Learning dynamics, stabilization mechanisms, and training pipelines

Training pipelines are remarkably consistent across DDQN scheduling frameworks. They initialize online and target Q-networks, collect transitions into replay memory, sample mini-batches, compute DDQN targets, minimize a Bellman-style loss, and periodically copy online parameters into the target network. In the NR sidelink framework, the online and target networks are initialized, a replay buffer $a_t$5 is filled, actions are selected by $a_t$6-greedy exploration, the loss

$a_t$7

is minimized, and $a_t$8 is applied every $a_t$9 steps (Chou et al., 8 Sep 2025). In RF-powered backscatter scheduling, the same loop is used with replay memory of 50,000 transitions, mini-batch size 32, target update period $Q(s,a)$0, and a linearly decayed $Q(s,a)$1-greedy policy (Anh et al., 2018).

Experience replay and target networks are nearly universal stabilization devices. Replay decorrelates successive scheduling samples, which is important because workloads, channel realizations, and queue states are highly autocorrelated. Periodic target updates slow the drift of the bootstrapping target. Additional stabilizers are domain dependent. The mmWave IAB framework uses replay, target copies every $Q(s,a)$2 timesteps, and separate hyperparameter settings for link scheduling and resource allocation (Abbasalizadeh et al., 11 Aug 2025). The federated IoT framework updates local DDQN models on each device and then aggregates them with FedAvg at the end of a training round, improving learning speed and scalability (Zarandi et al., 2021).

Several papers modify the standard DDQN loss or architecture rather than the scheduling formulation itself. M$Q(s,a)$3DQN replaces the usual single-batch mean TD loss with a max-mean loss over $Q(s,a)$4 sampled batches and solves a small quadratic program to optimize the worst batch loss, yielding a robust multi-batch update that the authors explicitly combine with DDQN (Zhang et al., 2022). The human-in-the-loop KAN-DDQN framework replaces conventional linear layers with Kolmogorov–Arnold Network layers and augments the TD loss with regularization and a feedback-alignment cost, thereby changing the representation and optimization structure while preserving the DDQN target (Jahin et al., 2024). A plausible implication is that many “DDQN scheduling frameworks” are best understood as a stable value-learning core surrounded by domain-specific architectural or loss-level modifications.

4. Architectural patterns and major variants

The literature shows that “DDQN scheduling framework” is not a single architecture but a family of design patterns.

One pattern is hierarchical or bilevel scheduling. In the bilevel deep reinforcement learning scheduler for industrial flow-shop-like problems, DDQN operates at the upper level to construct an initial global job sequence, while a Graph Pointer Network refines partial subsequences at the lower level through a sliding-window mechanism (Li et al., 2020). In hierarchical deep double Q-routing, deep double Q-learning is distributed across group leaders at several cluster levels, each leader choosing local path segments while receiving deferred global feedback about end-to-end performance (Ali et al., 2019). This suggests that DDQN is particularly useful as a high-level coordinator when the full combinatorial scheduling problem is too large for direct optimization.

A second pattern is architectural enrichment of the Q-network. Dueling DDQN decomposes $Q(s,a)$5 into a state value term and an action-advantage term,

$Q(s,a)$6

which is used for adaptive multi-objective LEO handover optimization when many actions have similar values (Chou et al., 4 May 2026). Weighted Double Deep Q-Network further combines two estimators through a weighted double estimator and supplements them with a lenient reward network and scheduled replay strategy to handle stochastic cooperative multiagent environments (Zheng et al., 2018).

A third pattern is decentralized or federated deployment. The mmWave IAB framework separates a greedy DDQN scheduler for access/backhaul activation from a multi-agent DDQN allocator for bandwidth and antenna assignment across slices (Abbasalizadeh et al., 11 Aug 2025). Federated DDQN for IoT networks keeps the double Q-learning logic local to devices but synchronizes learning by aggregating online-network parameters at the end of each round (Zarandi et al., 2021). This suggests that DDQN scheduling is compatible with centralized execution, decentralized execution, and centralized-training/decentralized-execution hybrids.

A fourth pattern is human- or model-guided representation learning. KAN-DDQN integrates human-in-the-loop feature selection into DDQN by training a separate feature-selection module with simulated feedback, while KAN layers expose symbolic and spline-based structure in the Q-network itself (Jahin et al., 2024). In the collaborative-perception scheduler SchedCP, semantic summaries and channel-state information are jointly fed to a DDQN scheduler, and the reward may be label-dependent or label-free depending on whether perceptual labels are available (Liu et al., 12 Feb 2025). These variants show that DDQN scheduling increasingly depends on state abstraction and representational bias, not only on the Bellman update.

5. Representative domains and empirical behavior

The diversity of applications shows that DDQN scheduling is a generic decision framework rather than a domain-specific wireless heuristic.

Domain	Representative framework	Reported outcome
NR sidelink licensed/unlicensed allocation	"Agentic DDQN-Based Scheduling for Licensed and Unlicensed Band Allocation in Sidelink Networks" (Chou et al., 8 Sep 2025)	Up to 87.5% blocking-rate reduction vs threshold-based scheduling
RF-powered backscatter cognitive radio	"Deep Reinforcement Learning for Time Scheduling in RF-Powered Backscatter Cognitive Radio Networks" (Anh et al., 2018)	About 15 packets/frame at $Q(s,a)$7, outperforming random, HTT, and backscatter-only
LEO satellite handover scheduling	"Dueling DDQN-Based Adaptive Multi-Objective Handover Optimization for LEO Satellite Networks" (Chou et al., 4 May 2026)	Up to 10.3% throughput improvement and near-zero blocking
mmWave IAB link scheduling and slicing	"Joint Scheduling and Resource Allocation in mmWave IAB Networks Using Deep RL" (Abbasalizadeh et al., 11 Aug 2025)	99.84 percent scheduling accuracy and 20.90 percent throughput improvement
Operating-system task scheduling	"Dynamic Operating System Scheduling Using Double DQN" (Sun et al., 31 Mar 2025)	250 ms completion time, 3.5 tasks/s throughput, 150 ms response time
Large-scale industrial scheduling	"Bilevel Learning Model Towards Industrial Scheduling" (Li et al., 2020)	Makespan reductions of 27.5%, 28.6%, and 22.1% on the three largest datasets

In the NR sidelink case, the DDQN scheduler operates over a single-server queue in which one packet is scheduled per decision epoch, under coexistence with both cellular communications and Wi‑Fi. At licensed bandwidth $Q(s,a)$8 Mbps, the reported blocking probabilities are $Q(s,a)$9 for DDQN, $$y_t^{\text{DDQN}} = r_t + \gamma Q_{\theta^-}\Bigl(s_{t+1}, \arg\max_{a'} Q_\theta(s_{t+1}, a')\Bigr),$$0 for threshold-based scheduling, and $$y_t^{\text{DDQN}} = r_t + \gamma Q_{\theta^-}\Bigl(s_{t+1}, \arg\max_{a'} Q_\theta(s_{t+1}, a')\Bigr),$$1 for random selection, which corresponds to about $$y_t^{\text{DDQN}} = r_t + \gamma Q_{\theta^-}\Bigl(s_{t+1}, \arg\max_{a'} Q_\theta(s_{t+1}, a')\Bigr),$$2 reduction versus the threshold baseline and about $$y_t^{\text{DDQN}} = r_t + \gamma Q_{\theta^-}\Bigl(s_{t+1}, \arg\max_{a'} Q_\theta(s_{t+1}, a')\Bigr),$$3 versus random selection. The same study reports slightly lower blocking under dynamic per-packet fading than under static-per-episode fading, which is attributed to richer training samples (Chou et al., 8 Sep 2025).

In RF-powered backscatter cognitive radio, DDQN learns how to partition a frame into harvesting, backscatter, and active-transmission slots. The paper reports convergence after roughly 2,000 episodes, a final average throughput of approximately 12 packets per frame, and approximately 15 packets per frame at $$y_t^{\text{DDQN}} = r_t + \gamma Q_{\theta^-}\Bigl(s_{t+1}, \arg\max_{a'} Q_\theta(s_{t+1}, a')\Bigr),$$4, compared with about 10 for random, about 9.3 for HTT, and about 3 for backscatter-only (Anh et al., 2018). In operating-system scheduling, the DDQN-based scheduler reports around 250 ms average task completion time, 3.5 tasks/s throughput, and 150 ms response time, compared with FCFS, SJF, and RR baselines with worse completion, throughput, and response metrics (Sun et al., 31 Mar 2025).

The industrial and IAB examples demonstrate that DDQN scheduling is not limited to choosing a single user or packet. The bilevel industrial scheduler scales to 5,000 jobs per production line and reports substantial makespan reductions with runtime below 200 seconds, while the IAB framework couples greedy DDQN link activation with DDQN resource slicing across 96 dynamic topologies (Li et al., 2020, Abbasalizadeh et al., 11 Aug 2025). This suggests that DDQN scheduling remains applicable when “scheduling” includes sequencing, admission, handover, offloading, or joint resource-allocation decisions, provided that the action space can be discretized and stabilized.

6. Limitations, misconceptions, and current research directions

A common misconception is that DDQN alone solves the scheduling problem once the Bellman target is stabilized. The surveyed papers do not support that view. Performance is strongly conditioned on state abstraction, reward design, action discretization, and domain decomposition. The NR sidelink scheduler assumes a single queue, one packet per decision epoch, simplified coexistence behavior, and no explicit fairness objective (Chou et al., 8 Sep 2025). The vehicular MEC offloading framework does not specify several architectural details needed for implementation and relies on a two-stage decomposition that moves part of the design burden outside the DDQN itself (Wang et al., 2024). The distributed wireless-control formulation shows that even before choosing DDQN, one may need a nontrivial state abstraction based on age variables and explicit action-space reduction to make scheduling learnable (Pang et al., 2021).

A second misconception is that more elaborate DDQN variants are always preferable. The literature shows trade-offs. Dueling DDQN improves discrimination among similar actions in overlapping LEO coverage, but the exact adaptive-weight update law is not fully specified in the summarized text (Chou et al., 4 May 2026). Weighted DDQN introduces a second estimator, a lenient reward network, and scheduled replay, but also adds computational and replay-management overhead (Zheng et al., 2018). M$$y_t^{\text{DDQN}} = r_t + \gamma Q_{\theta^-}\Bigl(s_{t+1}, \arg\max_{a'} Q_\theta(s_{t+1}, a')\Bigr),$$5DQN improves data efficiency by solving a per-update quadratic program over multiple sampled batches, but this introduces extra computational cost that may matter in large-scale schedulers (Zhang et al., 2022). KAN-DDQN improves interpretability and uses fewer hidden neurons than MLP baselines, but depends on spline-grid management, pruning, and auxiliary feature-selection machinery (Jahin et al., 2024).

Research directions in the cited work are therefore mainly about extending the framework rather than replacing it. In NR sidelink, proposed future work includes traffic prediction, adaptive tuning of listen-before-talk parameters, and multi-user multi-queue extensions (Chou et al., 8 Sep 2025). In federated IoT, the natural extension is larger-scale cooperative learning with partial participation and heterogeneous devices (Zarandi et al., 2021). In large-action wireless scheduling and industrial sequencing, hierarchical decomposition, action embedding, and bilevel learning appear repeatedly as practical responses to action-space growth (Pang et al., 2021, Li et al., 2020). In collaborative perception, label-free reward design indicates a broader move toward domain proxies when direct supervision is unavailable (Liu et al., 12 Feb 2025).

Overall, the contemporary DDQN scheduling framework is best understood as a modular architecture: Double Q-learning provides stable value estimation; state abstraction compresses the scheduling context; reward design encodes domain objectives; and auxiliary devices such as hierarchy, federated aggregation, dueling heads, weighted estimators, or interpretable function layers adapt the framework to scale, non-stationarity, and deployment constraints.