Multi-agent Double Deep Q-Network (MA-DDQN)

• MA-DDQN is a family of multi-agent reinforcement learning methods that extend Double DQN updates to reduce overestimation bias in non-stationary, coupled environments.
• It employs strategic action factorization and per-agent Q-learning, enabling decentralized decision making and efficient handling of combinatorial action spaces.
• Empirical results in mission assignment, NOMA power control, and beamforming demonstrate significant performance gains through improved coordination and tailored reward structures.

Searching arXiv for the cited MA-DDQN papers to ground the article in fresh records. arXiv search query: (Nguyen et al., 25 Jul 2025) Multi-agent Double Deep Q-Network Oranits Multi-agent Double Deep Q-Network (MA-DDQN) denotes a family of multi-agent, value-based reinforcement learning methods that extend Double Deep Q-Networks to settings with multiple concurrently acting decision makers whose rewards, transitions, or constraints are coupled. In recent arXiv work, MA-DDQN has been instantiated for mission assignment and task offloading in Open RAN-based intelligent transportation systems, semi-grant-free NOMA power control, mmWave MIMO beamforming, federated IoT offloading, microgrid energy management under communication failures, and opponent-aware policy learning (Nguyen et al., 25 Jul 2025, Fayaz et al., 2021, Wang et al., 2020, Zarandi et al., 2021, Zhou et al., 2021, Tao et al., 2022). Across these instantiations, the defining algorithmic element is the Double DQN update, which decouples action selection from action evaluation to mitigate Q-value overestimation, while the multi-agent layer varies from shared-policy coordination and centralized training with decentralized execution to independent learners, federated aggregation, belief-based coordination, and opponent-modeling mixtures.

1. Conceptual core

At its most general, MA-DDQN combines a per-agent action-value approximation with the Double DQN target. In the Oranits formulation, for agent $k$ and observation $\mathcal{O}_k(s)$, the target is

$$y_k(s) = r_k(s) + \gamma \, Q_{\theta^-}\left( \mathcal{O}_k(s+1), \arg\max_{a' \in \mathcal{A}} Q_{\theta}(\mathcal{O}_k(s+1), a') \right),$$

with loss

$$\mathcal{L}(\theta) = \mathbb{E}\left[ \left( y_k(s) - Q_{\theta}(\mathcal{O}_k(s), a_k(s))\right)^2 \right].$$

This is explicitly introduced to reduce overestimation bias and improve convergence in a non-stationary multi-agent environment (Nguyen et al., 25 Jul 2025).

What is not fixed is the meaning of “multi-agent.” In one line of work, one agent is associated with each vehicle, user, or base station, and each agent learns its own local action-value function while interacting through a common environment (Nguyen et al., 25 Jul 2025, Fayaz et al., 2021, Wang et al., 2020). In another, multiple homogeneous binary-action DDQNs act as a factorized controller over a single environment action, with shared state and shared reward but agent-specific actions (Hafiz et al., 2020). A further variant augments a DDQN with a Mixture-of-Experts gate conditioned on opponent features, so that different DDQN experts specialize to different opponent strategy patterns (Tao et al., 2022). The literature therefore does not support a single canonical MA-DDQN architecture; it supports a recurring double-Q learning principle embedded in several distinct multi-agent control regimes.

2. Problem formulations, observables, and action factorization

The formalization of MA-DDQN depends strongly on the application domain. Oranits formulates mission assignment and offloading as a global optimization problem $\mathscr{P}_1$ that maximizes the number of missions completed before deadline, then approximates this with a multi-agent deep RL formulation in which each vehicle-agent observes

$$\mathcal{O}_k(s) = \{\mathbf{R}(s), \mathbf{V}(s), \mathbf{A}(s), \mathbf{J}(s)\},$$

where road/traffic state, vehicle states, mission assignment status, and mission descriptors encode congestion, offloading availability, precedence constraints, mission routes, and vehicle load (Nguyen et al., 25 Jul 2025). In SGF-NOMA, the resource allocation problem is modeled as a team Markov game whose agent state is the previous-slot rate profile of grant-free users over resource blocks; each action jointly encodes one transmit power level and one resource block, and invalid actions are pruned to reduce complexity (Fayaz et al., 2021). In mmWave beamforming, each BS-agent observes a history of omni-directional rates, optionally augmented with positions, and acts by selecting a beam codebook index (Wang et al., 2020).

The action space is often deliberately factorized. Oranits avoids a centralized combinatorial action over all vehicle-mission assignments by letting each agent choose one mission index at each step, so the joint action is $\mathbf{a}(s)=(a_1(s),\dots,a_{K^*}(s))$ rather than a monolithic assignment tensor (Nguyen et al., 25 Jul 2025). The binary-action-agent framework pushes this further: each agent outputs $a_t^i\in\{0,1\}$, and a deterministic decision structure maps the resulting binary vector to the environment’s native action space (Hafiz et al., 2020). This suggests a recurring design pattern in MA-DDQN: discrete combinatorial control is often decomposed into several low-cardinality per-agent decisions, with environment logic or an aggregation rule enforcing feasibility.

Reward construction is similarly domain-specific. Oranits uses a composite reward that combines mission benefit, remaining budget after offloading, a shared system-profit term, a dependency-aware term, and penalties for reselecting an already assigned mission. Its dependency component,

$$r_k^{\text{dep}(s)} = (\lceil Z/K^* \rceil - s)\left( |\mathcal{M}_{a_k(s)}^{+}| - |\mathcal{M}_{a_k(s)}^{-}| + 1 \right),$$

prioritizes early execution of missions that unlock larger dependency chains (Nguyen et al., 25 Jul 2025). SGF-NOMA uses a team reward equal to the system throughput $\bar C(t+1)$ when all constraints hold and the throughput is not degraded, and $\mathcal{O}_k(s)$0 otherwise (Fayaz et al., 2021). In beamforming, the reward is the normalized local gain

$\mathcal{O}_k(s)$1

which compares beamformed sum-rate with an omni-directional baseline (Wang et al., 2020). In federated IoT offloading, the immediate cost is not directly predicted by the network; it is computed by solving a convex subproblem for local CPU frequency or transmit power after the discrete offloading action is chosen (Zarandi et al., 2021).

3. Architectural patterns and coordination regimes

One major axis of variation is how agents coordinate. Oranits is described as multi-agent value-based RL with parameter sharing and a shared off-policy buffer: all vehicle-agents share a common replay buffer, the same environment and reward structure, and very likely a shared Q-network, even though the notation writes $\mathcal{O}_k(s)$2 (Nguyen et al., 25 Jul 2025). This is close to centralized training with decentralized execution. The shared buffer is intended to aggregate experience across vehicles and reduce the effective non-stationarity induced by simultaneously learning agents.

Other MA-DDQN systems adopt independent learners. In SGF-NOMA, each grant-free user runs its own DDQN with its own replay buffer, while the base station broadcasts global state and a shared team reward; no direct inter-agent signaling is required (Fayaz et al., 2021). In distributed beamforming, each BS maintains its own online and target networks, its own replay buffer, and its own local reward, treating the other BSs as part of the environment despite interference coupling and association feedback (Wang et al., 2020). In the binary-action-agent formulation, each homogeneous agent has its own DDQN and replay buffer, but all agents share the same environment state and reward, and their binary outputs are fused by a decision structure (Hafiz et al., 2020). These implementations show that MA-DDQN often operates without an explicit joint action-value $\mathcal{O}_k(s)$3.

Architectural choices also differ substantially. Oranits uses a fully connected network with SELU and ELU nonlinearities and output dimension $\mathcal{O}_k(s)$4, one Q-value per mission index (Nguyen et al., 25 Jul 2025). The mmWave beamforming implementation uses a three-layer fully connected network whose hidden widths scale with the number of users, and whose output dimension equals the size of the beam codebook (Wang et al., 2020). The microgrid BA-DRL system uses two hidden LSTM layers with 35 units each to model temporal dependencies such as state-of-charge evolution and operating cycles (Zhou et al., 2021). The opponent-modeling system uses DDQN experts and gating networks built from feedforward MLPs with hidden sizes 64 and 128 and ReLU activations, so that the final Q-value is a convex mixture of expert outputs conditioned on opponent features (Tao et al., 2022).

Several papers also introduce higher-level coordination layers on top of DDQN. BA-DRL computes a belief-based correlated equilibrium under communication failures: isolated agents act greedily using their own Q-networks, whereas connected agents combine exchanged Q-values with Bayesian beliefs over problematic agents’ actions (Zhou et al., 2021). Federated DDQN periodically averages online Q-network parameters across selected devices,

$\mathcal{O}_k(s)$5

to accelerate learning while keeping data local (Zarandi et al., 2021). The opponent-modeling DDQN-MoE conditions a softmax gate on opponent-action statistics, then mixes specialized DDQN experts (Tao et al., 2022). These variants preserve the double-Q learning core but materially change the coordination mechanism.

4. Training procedures and execution models

The training regime in MA-DDQN is often as important as the network architecture. Oranits separates offline training at the Non-RT RIC from deployment at the Near-RT RIC. Its reported hyperparameters include discount factor $\mathcal{O}_k(s)$6, learning rate $\mathcal{O}_k(s)$7, replay memory size $\mathcal{O}_k(s)$8, mini-batch size 512, initial $\mathcal{O}_k(s)$9, decay factor $$y_k(s) = r_k(s) + \gamma \, Q_{\theta^-}\left( \mathcal{O}_k(s+1), \arg\max_{a' \in \mathcal{A}} Q_{\theta}(\mathcal{O}_k(s+1), a') \right),$$0, and minimum $$y_k(s) = r_k(s) + \gamma \, Q_{\theta^-}\left( \mathcal{O}_k(s+1), \arg\max_{a' \in \mathcal{A}} Q_{\theta}(\mathcal{O}_k(s+1), a') \right),$$1. Each episode assigns $$y_k(s) = r_k(s) + \gamma \, Q_{\theta^-}\left( \mathcal{O}_k(s+1), \arg\max_{a' \in \mathcal{A}} Q_{\theta}(\mathcal{O}_k(s+1), a') \right),$$2 missions to $$y_k(s) = r_k(s) + \gamma \, Q_{\theta^-}\left( \mathcal{O}_k(s+1), \arg\max_{a' \in \mathcal{A}} Q_{\theta}(\mathcal{O}_k(s+1), a') \right),$$3 vehicles over $$y_k(s) = r_k(s) + \gamma \, Q_{\theta^-}\left( \mathcal{O}_k(s+1), \arg\max_{a' \in \mathcal{A}} Q_{\theta}(\mathcal{O}_k(s+1), a') \right),$$4 steps, then simulates execution to compute delayed mission outcomes and reward shaping before storing transitions (Nguyen et al., 25 Jul 2025).

SGF-NOMA uses smaller per-agent DDQNs with replay memory size $$y_k(s) = r_k(s) + \gamma \, Q_{\theta^-}\left( \mathcal{O}_k(s+1), \arg\max_{a' \in \mathcal{A}} Q_{\theta}(\mathcal{O}_k(s+1), a') \right),$$5, batch size 32, learning rate $$y_k(s) = r_k(s) + \gamma \, Q_{\theta^-}\left( \mathcal{O}_k(s+1), \arg\max_{a' \in \mathcal{A}} Q_{\theta}(\mathcal{O}_k(s+1), a') \right),$$6, discount factor $$y_k(s) = r_k(s) + \gamma \, Q_{\theta^-}\left( \mathcal{O}_k(s+1), \arg\max_{a' \in \mathcal{A}} Q_{\theta}(\mathcal{O}_k(s+1), a') \right),$$7, and hard target updates every $$y_k(s) = r_k(s) + \gamma \, Q_{\theta^-}\left( \mathcal{O}_k(s+1), \arg\max_{a' \in \mathcal{A}} Q_{\theta}(\mathcal{O}_k(s+1), a') \right),$$8 steps. Its reported training structure is 500 episodes with 100 time steps per episode. The paper explicitly states that “DDQN is designed for communication scenarios with a small-size action-state space, while Dueling DDQN is for a large-size case,” and uses the dueling extension when the number of users, RBs, or power levels makes the state-action space large (Fayaz et al., 2021).

In distributed beamforming, each BS trains with its own replay memory, batch size 32, Adam optimizer, $$y_k(s) = r_k(s) + \gamma \, Q_{\theta^-}\left( \mathcal{O}_k(s+1), \arg\max_{a' \in \mathcal{A}} Q_{\theta}(\mathcal{O}_k(s+1), a') \right),$$9, and $$\mathcal{L}(\theta) = \mathbb{E}\left[ \left( y_k(s) - Q_{\theta}(\mathcal{O}_k(s), a_k(s))\right)^2 \right].$$0-greedy exploration between $$\mathcal{L}(\theta) = \mathbb{E}\left[ \left( y_k(s) - Q_{\theta}(\mathcal{O}_k(s), a_k(s))\right)^2 \right].$$1 and $$\mathcal{L}(\theta) = \mathbb{E}\left[ \left( y_k(s) - Q_{\theta}(\mathcal{O}_k(s), a_k(s))\right)^2 \right].$$2 (Wang et al., 2020). In the federated IoT setting, local DDQN training is followed by parameter aggregation at the end of each episode or federated round, and the paper studies how batch size, network depth, and target update frequency affect learning speed (Zarandi et al., 2021). In the microgrid MA-POMDP, each agent stores up to 1200 transitions, samples mini-batches of size 120 every 40 episodes, trains LSTM-based DDQNs with learning rate $$\mathcal{L}(\theta) = \mathbb{E}\left[ \left( y_k(s) - Q_{\theta}(\mathcal{O}_k(s), a_k(s))\right)^2 \right].$$3 and discount factor $$\mathcal{L}(\theta) = \mathbb{E}\left[ \left( y_k(s) - Q_{\theta}(\mathcal{O}_k(s), a_k(s))\right)^2 \right].$$4, and updates the target network after several training iterations (Zhou et al., 2021). Execution can therefore range from fully decentralized online inference to hybrid offline-online workflows in which training, fine-tuning, and deployment are distributed across different network control layers.

5. Representative applications and reported empirical behavior

Across the cited literature, MA-DDQN is reported as effective in heterogeneous distributed decision problems, but the performance gains are tightly coupled to state design, reward shaping, and the surrounding coordination mechanism.

Setting	MA-DDQN instantiation	Reported outcome
Open RAN ITS mission assignment and offloading (Nguyen et al., 25 Jul 2025)	One vehicle-agent per selected vehicle; shared replay; multi-action mission selection	Mission completions improved by 11.0% and overall benefit by 12.5%; modified reward converged around 1200 after ~50,000 epochs versus ~600 for a naïve reward
SGF-NOMA power control (Fayaz et al., 2021)	One grant-free user agent; DDQN or Dueling DDQN over power/RB actions	MA-Dueling DDQN achieved 17.5% throughput gain over existing SGF-NOMA and 22.2% over pure GF-NOMA; MA-DDQN converged within ~300 episodes for moderate action spaces
mmWave MIMO beamforming (Wang et al., 2020)	One BS-agent per BS selecting beam codebook indices	Achieved comparable performance to exhaustive search while operating at much lower complexity
Microgrid energy management under failures (Zhou et al., 2021)	One DDQN per energy agent with belief-based correlated equilibrium	BA-DRL obtained 4.1% and 10.3% higher reward than Nash-DQN and ADMM under 1% communication failure; DDQN yielded 6.7% lower DSM daily cost under PV uncertainty
Federated IoT offloading (Zarandi et al., 2021)	Independent DDQNs with periodic FedAvg over Q-network weights	Larger batch size accelerated convergence: batch size 10 required ~200 iterations, whereas batch size 30 converged around ~40 iterations
5G mmWave sleep mode optimization (Masrur et al., 27 Nov 2025)	One BS-agent per BS with shared global reward for EE/QoS	Achieved up to 0.60 Mbit/Joule EE, 8.5 Mbps 10th-percentile throughput, and met QoS constraints 95% of the time

These results suggest that MA-DDQN is most competitive when three conditions are met simultaneously: the action factorization is aligned with the combinatorial structure of the problem, the reward preserves the actual system objective rather than a weak surrogate, and the coordination mechanism matches the dominant source of coupling. The Oranits ablation on reward shaping is especially indicative: the same MA-DDQN architecture converged to markedly different total system benefit depending on whether the reward included shared profit and dependency-aware terms. The NOMA and mmWave results similarly indicate that Double DQN’s bias reduction is useful, but not sufficient on its own; action-space pruning, dueling decomposition, normalized rewards, and careful state compression materially influence final performance.

6. Limitations, misconceptions, and emerging directions

A recurring misconception is that MA-DDQN necessarily means a centralized network that estimates a joint value over the full joint action space. The surveyed papers do not support that reading. Oranits learns per-agent mission choices with shared parameters and emergent joint action (Nguyen et al., 25 Jul 2025). The SGF-NOMA and beamforming systems rely on independent DDQNs without a joint $$\mathcal{L}(\theta) = \mathbb{E}\left[ \left( y_k(s) - Q_{\theta}(\mathcal{O}_k(s), a_k(s))\right)^2 \right].$$5 (Fayaz et al., 2021, Wang et al., 2020). The binary-action framework goes further by decomposing one environment action into several agent-specific binary sub-actions, each with its own DDQN (Hafiz et al., 2020). MA-DDQN is therefore better understood as a design space than as a fixed algorithm.

The main technical limitations are similarly recurrent. In Oranits, the action space grows with the number of missions $$\mathcal{L}(\theta) = \mathbb{E}\left[ \left( y_k(s) - Q_{\theta}(\mathcal{O}_k(s), a_k(s))\right)^2 \right].$$6, delayed rewards complicate credit assignment, and greedy task offloading is fixed rather than learned (Nguyen et al., 25 Jul 2025). In SGF-NOMA, the authors explicitly distinguish DDQN for small state-action spaces from Dueling DDQN for large ones, implying that naïve MA-DDQN scales poorly when discrete combinatorics expand (Fayaz et al., 2021). In beamforming, independent learners still face non-stationarity because each BS’s reward depends on other BSs’ beams through interference and association (Wang et al., 2020). In the microgrid BA-DRL setting, the current system has three agents and relies on a hand-designed observation model $$\mathcal{L}(\theta) = \mathbb{E}\left[ \left( y_k(s) - Q_{\theta}(\mathcal{O}_k(s), a_k(s))\right)^2 \right].$$7, so scaling belief tracking and correlated-equilibrium computation to larger systems remains an open issue (Zhou et al., 2021). In opponent modeling, the MoE architecture depends on handcrafted opponent features such as recent action frequencies and last action, which may be insufficient in richer strategic environments (Tao et al., 2022).

The future directions named in the surveyed literature point toward a broader convergence between MA-DDQN and structured MARL. Oranits explicitly mentions federated learning and graph neural networks as extensions for privacy-preserving distributed training and better representation of mission dependencies and road networks (Nguyen et al., 25 Jul 2025). The microgrid work points toward CTDE, richer belief models, and actor-critic variants for continuous controls (Zhou et al., 2021). The SGF-NOMA study positions dueling architectures as the natural extension when action-state spaces become large (Fayaz et al., 2021). The opponent-modeling results suggest that explicit modeling of other agents’ policies can improve over plain DDQN when strategic heterogeneity is significant (Tao et al., 2022). A plausible implication is that future MA-DDQN systems will increasingly combine Double Q-learning with graph encoders, federated aggregation, belief updates, or equilibrium layers rather than relying on independent per-agent Q-learning alone.