Cooperative Double Q-Learning Overview

• Cooperative Double Q-Learning is a reinforcement learning approach that uses dual estimators to reduce overestimation bias while promoting coordinated behaviors among agents.
• It incorporates techniques like mean-field averaging, local reward sharing, and reputation mechanisms to achieve scalable cooperation in diverse environments.
• Empirical studies in traffic signal control, public goods games, and continuous domains demonstrate faster learning and improved stability compared to conventional methods.

Cooperative Double Q-Learning (Co-DQL) is a family of reinforcement learning (RL) algorithms that extend classical double Q-learning with explicit mechanisms for promoting cooperation and bias reduction in both multi-agent stochastic games and continuous control MDPs. Co-DQL frameworks have been formally developed in the context of large-scale multi-agent traffic signal control (Wang et al., 2019), reputation-driven public goods games (Xie et al., 31 Mar 2025), and continuous RL with deep function approximation (Kuznetsov, 2023). Distinctive features combine double estimator bias correction, information or policy sharing, and cooperative credit assignment.

1. Foundations and Motivation

Cooperative Double Q-Learning builds on the double Q-learning paradigm, which maintains two independent estimators (Q-functions or critics) to mitigate overestimation bias that arises when a single estimator is used for both maximization and evaluation. In single-agent and decentralized MARL, this bias can impair policy learning, especially in high-variance or partial observability settings.

Co-DQL frameworks incorporate cooperation by introducing additional mechanisms such as mean-field action averaging, localized reward/state sharing, adjusted exploration, or mixture actor policies. The primary motivations are threefold:

• Resolution of overestimation bias: Independence between estimators corrects the upward bias found in classical Q-learning.
• Enhanced cooperative behaviors: Through explicit sharing (rewards, states) or game-theoretic constructs (mean-field, public goods), agents are incentivized towards collective objectives.
• Scalability: Decentralized updates and local interaction modeling permit tractable algorithms even in large-scale agent networks.

2. Algorithmic Components

Co-DQL encompasses several key algorithmic innovations, exemplified by its instantiations in discrete MARL, evolutionary games, and continuous control:

Double Estimator Architecture

Each agent or policy maintains two independent estimators—denoted $Q^A$ and $Q^B$ (or equivalently, two policy-critic pairs in continuous control)—with the following update structure (Wang et al., 2019, Xie et al., 31 Mar 2025, Kuznetsov, 2023):

• At every step, a single estimator is randomly selected for update.
• The target value for the updated estimator uses the other estimator, i.e., selection (argmax or policy output) is performed on one Q and evaluation on the other.
• For tabular and function-approximation settings, this results in:

$$Q^A(s, a) \leftarrow Q^A(s, a) + \alpha [r + \gamma Q^B(s', a^*) - Q^A(s, a)]$$

where $a^* = \arg\max_{a'} Q^A(s', a')$, and conversely for $Q^B$.

Cooperation Mechanisms

Distinct Co-DQL instantiations introduce mechanisms for agent cooperation:

• Mean-Field Approximation: Each agent models neighbor actions by their empirical average $\bar a_k$, reducing joint-action dimensionality in the Q-function. This is formalized as $$Q_k(s_k, a_k, a_{-k}) \approx Q_k(s_k, a_k, \bar a_k)$$ (Wang et al., 2019).
• Local Reward and State Sharing: Agents augment their local experience with the average rewards and states of their immediate neighbors:

$$\hat r_k = r_k + \alpha \sum_{i \in N(k)} r_i, \quad \hat s_k = \langle s_k, \frac{1}{|N(k)|} \sum_{i \in N(k)} s_i \rangle$$

where $N(k)$ denotes the set of neighbors (Wang et al., 2019).

• Reputation-Driven Payoff Integration: In spatial public goods games, reward incorporates both material payoff and reputation, balancing exploitation and social signaling (Xie et al., 31 Mar 2025).

Exploration Strategy

Some Co-DQL variants employ upper-confidence-bound (UCB) exploration over conventional $\epsilon$-greedy, with the selection:

$$a_k = \arg\max_{c \in A_k} \left\{ Q^A_k(s_k, c) + \sqrt{\frac{\ln(R_{s_k})}{R_{s_k, c}}} \right\}$$

enhancing exploration of under-visited actions (Wang et al., 2019).

Continuous Control Adaptation

For continuous state and action spaces, Co-DQL is implemented with dual policy networks (two actor heads) and two critics. Each critic is trained on actions produced by the alternate actor head (mixture policy):

$y_i = r + \gamma Q_{\psi_{i}'}(s', a'_{(i)})$

where $a'_{(i)} = \pi_{\phi_{3-i}}(s')$ (Kuznetsov, 2023). Policy optimization and target value computation both enforce this cross-evaluation protocol.

3. Theoretical Properties

The bias correction property of double Q-learning architecture carries over to Co-DQL, both in single-agent and cooperative settings. Under standard stochastic approximation assumptions—visitation of all state-action pairs, bounded rewards, and GLIE exploration—it can be shown in the tabular case that the Q-functions converge almost surely to the Nash Q-values of the underlying Markov or stochastic game (Wang et al., 2019). In the reputation-driven PGG variant, mean-field analysis of state-transition fractions and stationarity of cooperation levels yields analytic expressions matching empirical simulation (Xie et al., 31 Mar 2025).

The essence of overestimation bias removal is the independence of maximization and evaluation: given uncorrelated approximation errors, the expected Q-update target is unbiased:

$$\mathbb{E}[Q^B(s', \arg\max_a Q^A(s', a))] \approx \max_a \mathbb{E}[Q^A(s', a)]$$

(Xie et al., 31 Mar 2025). Similar logic applies in continuous control Co-DQL, with network weights taking the role of independent estimators (Kuznetsov, 2023).

4. Empirical Evaluation and Applications

Large-Scale Traffic Signal Control

Co-DQL has been applied to multi-agent traffic signal control, modeled as a stochastic game over $N$ intersections, with decentralized (local-only) state observation and shared reward/state information from neighbors. In extensive OpenAI-Gym and real-world SUMO-based simulation studies, Co-DQL achieved superior performance:

Algorithm	Mean vehicle delay (grid)	Mean episode reward (Xi'an)	Avg. vehicle speed (m/s)
Co-DQL	37 t	−930 (±87)	5.35
MA2C	72 t	−1109 (±83)	4.65
IDQL	132 t	−1076 (±194)	—
IQL	149 t	−1161 (±191)	—
DDPG	111 t	−1297 (±141)	—

Measurement units: traffic grid = steps (t), Xi'an = rewards.

Co-DQL produced faster learning and more stable policies with smaller delays and higher vehicle speeds relative to decentralized or model-free deep RL baselines (Wang et al., 2019).

Public Goods Games with Reputation

In spatial public goods scenarios, Co-DQL outperformed single-table Q-learning across cooperation regimes, especially when network reciprocity is weak or social reputation is highly dynamic. Agents utilizing Co-DQL reached higher steady-state cooperation fractions and robustly maintained cooperation even at low-to-moderate synergy factors $r$ (Xie et al., 31 Mar 2025). The learned Q-value matrices aligned with cooperative incentive structures in a way not observed with the single Q-table baseline.

Continuous Reinforcement Learning

In high-dimensional continuous control, Co-DQL matched the final episodic return of top-performing distributional RL algorithms (TQC) without need for hyperparameter tuning. On Walker2d-v3 and Humanoid-v3, Co-DQL achieved competitive sample efficiency and bias reduction, though residual overestimation could remain if neural network errors were not fully decorrelated (Kuznetsov, 2023).

5. Distinctive Mechanisms Across Domains

Co-DQL exhibits domain-adapted mechanisms for cooperative bias-reduced learning:

Domain	Key Cooperation Mechanism	Overestimation Mitigation
Multi-Agent TSC	Mean-field action, reward/state share	Double Q, random update, UCB
Public Goods Game	Reputation feedback, HIORC investment	Double Q, theoretical noise avg.
Continuous Control	Policy mixture, cross-evaluated TD	Cross-policy double Q

In all cases, the algorithmic architecture leverages independent value estimation per agent, localized cooperation, and targeted exploration strategies for robust scalable learning.

6. Limitations and Open Problems

While Co-DQL robustly addresses overestimation bias and cooperative dynamics, several limitations remain:

• In continuous domains, some residual overestimation bias can persist when network approximations are not fully independent. Further decorrelation strategies, such as distinct replay buffers, may provide further improvements (Kuznetsov, 2023).
• Convergence guarantees have only been formally established in finite (tabular) or linear settings, with neural approximators requiring further theoretical development.
• In public goods games, population structure and reputation dynamics can induce non-monotonic and abrupt behavior transitions, which are not always easily controlled by algorithmic parameters (Xie et al., 31 Mar 2025).
• The impact of specific cooperation mechanisms (e.g., UCB vs. $\epsilon$-greedy, type of reward reallocation) on learning stability and scalability remains an open empirical question.

A plausible implication is that extending Co-DQL to other structured multi-agent systems will require careful adaptation of the cooperation and bias correction schemes to the local credit assignment and network interaction structure.

7. Summary and Research Directions

Cooperative Double Q-Learning unifies double estimator bias correction with principled cooperative mechanisms, providing robust, scalable RL frameworks for multi-agent, network-evolutionary, and continuous control settings (Wang et al., 2019, Xie et al., 31 Mar 2025, Kuznetsov, 2023). Future work may address:

• Further distributional and theoretical analyses in the context of deep function approximation,
• Extensions to more general network topologies and asynchronous update strategies,
• Domain-specific cooperation protocols beyond mean-field and reputation, including explicit negotiation or role assignment.

Co-DQL stands as a foundational approach for large-scale decentralized RL where bias mitigation and cooperation are both critical.