Double Actor-Critic Framework

• Double Actor-Critic Framework is defined as a reinforcement learning approach that uses parallel actor–critic pairs to mitigate overestimation bias and improve stability.
• It employs innovative cross-update and ensemble averaging techniques, such as TD-error-driven regularization, to select robust TD target values.
• Empirical evaluations show that variants like TDDR and SDQ-CAL achieve state-of-the-art performance in continuous control tasks on benchmarks like MuJoCo and DeepMind Control Suite.

The Double Actor-Critic (DAC) framework refers to a class of reinforcement learning algorithms that employ two or more parallel actor–critic pairs, exploiting architectural and algorithmic diversity to enhance estimation accuracy, stability, and convergence in challenging decision-making environments. DAC algorithms have been developed in various contexts, including value function estimation, exploration strategies, policy regularization, robust control, and hierarchical RL. The fundamental characteristic is the simultaneous or coordinated use of multiple actors and critics, usually with cross-update or ensemble techniques, to reduce the overestimation bias and learning instability typical of standard single actor–critic methods.

1. Core Architectural Principles

The canonical DAC architecture, as exemplified by the TDDR (TD-Error-Driven Regularization) method, maintains two independent actor networks $\pi_{\phi_1}, \pi_{\phi_2}$ and two critic networks $Q_{\theta_1}, Q_{\theta_2}$, together with corresponding target networks for stabilization. Each actor–critic pair is updated in a cross-update manner: at each gradient step, a single actor–critic pair and its targets are updated. During the Bellman backup, both critics are evaluated on noisy actions generated by both actors, resulting in a four-way action-critique topology. The TD target is selected adaptively by comparing the TD errors associated with both next-state action proposals, choosing the value whose TD error exhibits the smallest absolute magnitude. This TD-error-driven selection provides an implicit regularization without introducing any additional hyperparameters beyond those in baseline double-critic methods such as TD3 (Chen et al., 28 Sep 2024).

In alternative formulations such as SDQ-CAL, both critics are updated simultaneously (as opposed to alternating), using a Conservative Advantage Learning reward shaping to further enlarge the action gap and promote sample efficiency (Li et al., 2022). Other variants focus on ensemble averaging of policy and value outputs (e.g., A2C2/AACHER (Sehgal et al., 2022)) or hierarchical decoupling (e.g., DAC for options (Zhang et al., 2019), layered control (Yang et al., 3 Aug 2024)).

2. Mathematical Formulation and Algorithmic Details

In deterministic continuous-action DAC, the actor and critic objectives are tightly coupled:

• Each actor $\pi_{\phi_i}$ maximizes its paired critic’s value:

$$J(\phi_i) = \mathbb{E}_{s \sim \mathcal{D}} [ Q_{\theta_i}(s, \pi_{\phi_i}(s)) ]$$

with the deterministic policy gradient (DPG) update for $\phi_i$:

$$\nabla_{\phi_i} J(\phi_i) \approx \frac{1}{N} \sum_{k=1}^N \nabla_a Q_{\theta_i}(s_k, a)|_{a=\pi_{\phi_i}(s_k)} \cdot \nabla_{\phi_i} \pi_{\phi_i}(s_k)$$

• Each critic $Q_{\theta_i}$ is trained by minimizing squared TD error with TD-target $y_k$ derived from coordinated multi-actor, multi-critic bootstraps. In TDDR, this proceeds by generating noisy next-state actions for both target actors, evaluating both critics, and computing two sets of TD errors:

$$\delta_j = r + \gamma \psi_j - \min_{i=1,2} Q_{\theta_i'}(s, a)$$

where $\psi_j = \min_{i=1,2} Q_{\theta_i'}(s', a_j')$ for $a_j' = \pi_{\phi_j'}(s') + \epsilon$, $j=1,2$. The TD target is finalized as $y = r + \gamma \psi_{j^*}$, with $j^*$ chosen according to $\arg\min_j |\delta_j|$ (Chen et al., 28 Sep 2024).

• In SDQ-CAL, each critic’s backup uses a cross-evaluated, conservatively reshaped reward. The Bellman backup target for $Q_{\theta_1}$ is

$$y_1 = r_1(s,a) + \gamma Q_{\theta_2'}(s', \pi_{\phi_1}(s'))$$

with

$$r_1(s,a) = r(s,a) + \beta \left( Q_{\min}(s,a) - Q_{\theta_1'}(s, \pi_{\phi_1}(s)) \right)$$

and analogously for $Q_{\theta_2}$ (Li et al., 2022).

• At action selection, double-actor architectures exploit both policy proposals, e.g., sampling actions from both actors and evaluating all critics, then selecting the action with the most promising ensemble value.

3. Statistical Properties and Convergence

Double actor–critic methods aim to correct two principal RL pathologies: overestimation bias in value expansion (Bellman backups) and sensitivity to action noise or critic variance. Double-critic schemes (e.g., TD3) mitigate positive bias by using the minimum of two value estimates, but if only one actor is used, both critics are tied to the same noisy target. By introducing a second actor, the policy proposal space diversifies, leading to a four-fold diversity in target Q-value formation and more effective variance reduction.

The TD-error-driven selection in TDDR adaptively prefers the more consistent (low-TD-error) backup target, providing automatic regularization. Theoretical analysis (Theorem 1 in (Chen et al., 28 Sep 2024)) proves almost sure convergence of both Q-functions to $Q^*$ under standard stochastic approximation assumptions (finite MDP, diminishing stepsizes, full exploration, bounded rewards), via Robbins–Monro techniques and by reducing to existing double Q-learning convergence proofs.

SDQ-CAL demonstrates contraction properties for its conservative Bellman operator when the sum $\beta+\gamma<1$, ensuring unique fixed points and stability (Li et al., 2022). Empirical studies show these architectures are highly robust to hyperparameter mis-specification, especially compared to more heavily regularized or single-actor baselines.

4. Algorithmic Variants and Extended Frameworks

The DAC paradigm encompasses a range of variants:

• Time-Scale Double Actor–Critic: Some works (e.g., (Bhatnagar et al., 2022)) analyze the effect of swapping update time-scales between actor and critic ("actor–critic" vs "critic–actor") and demonstrate that both lead to convergence under suitable step-size schedules. This formal equivalence generalizes the two-timescale stochastic approximation framework, providing theoretical guarantees for either decomposition.
• Dual Actor–Critic via Lagrangian Duality: The “Dual-AC” method frames actor–critic optimization as a Lagrangian saddle-point, solving the Bellman optimality equation via dual ascent on policy and occupancy measures coupled to a minimization in the value function (Dai et al., 2017). This yields a unified objective, multi-step bootstrapped regularization, and unbiased policy gradients.
• Hierarchical and Layered Control: DAC has been adapted to layered optimal control settings, with layered actor–critic schemes coordinating planning (trajectory generation) and tracking (low-level control), often using a learned dual network for reference correction. The convergence of the dual variable to the optimal coordination map is established in LQR (Yang et al., 3 Aug 2024).
• Options and Hierarchical Reinforcement Learning: The DAC architecture has been applied to option discovery and management, framing high-level (master policy) and low-level (intra-option) RL as coupled actor–critic problems on two induced MDPs, with provable efficiency gains in hierarchical transfer (Zhang et al., 2019).
• Ensemble Averaging: Certain implementations use larger ensembles (e.g., A2C2 with two actors and two critics, or A10C10) and average their outputs for both value and policy. This improves stability and success rates over vanilla DDPG/TD3 by smoothing estimation noise (Sehgal et al., 2022).

5. Empirical Evaluation and Benchmark Results

Extensive empirical studies demonstrate that DAC/TDDR and its relatives deliver state-of-the-art or near state-of-the-art performance in continuous control tasks (MuJoCo: Ant-v2, HalfCheetah-v2, Hopper-v2, Walker2d-v2, Reacher-v2, InvertedPendulum-v2, InvertedDoublePendulum-v2; Box2D; DeepMind Control Suite). Key empirical findings include:

• TDDR outperforms DDPG and TD3 (single-actor/single-critic baselines) consistently, achieving higher average returns and lower or comparable variance, without additional hyperparameters (Chen et al., 28 Sep 2024).
• Against more regularized DAC approaches (SD3, DARC, GD3), TDDR maintains competitive or better performance, with lower sensitivity to hyperparameter settings.
• SDQ-CAL achieves the highest returns and superior sample efficiency on challenging control environments, substantially reducing value estimation bias compared to both overestimating (DDPG) and underestimating (TD3/SD3) baselines (Li et al., 2022).
• Ensemble-based DAC variants improve success rates by a factor of 2–3× on the hardest goal-conditioned robotic benchmarks, with diminishing but still positive returns for deeper ensembles (Sehgal et al., 2022).
• Hierarchical DAC for option learning accelerates transfer between related tasks, especially in cases where option reuse is effective (Zhang et al., 2019).

6. Implementation Complexity and Practical Considerations

All mainline DAC implementations require additional network maintenance and compute compared to vanilla actor–critic (six to eight neural networks for TDDR/SDQ-CAL, plus replay buffers). The increase in forward/backward passes is typically linear in the number of actors and critics. TDDR and related methods minimize implementation burden by eliminating additional regularization coefficients or hyperparameters beyond standard ones—noise, learning rate, target-smoothing rate, and batch size. This enhances reproducibility and empirical robustness across a range of environments (Chen et al., 28 Sep 2024). In large-ensemble settings, training time and memory requirements scale linearly with the number of actors/critics, but empirical results suggest most of the stability gains accrue with small ensembles (D=P=2) (Sehgal et al., 2022).

7. Broader Significance and Future Directions

Double actor–critic frameworks represent a general architectural principle for robust, sample-efficient, and bias-reduced reinforcement learning. They provide a unifying underpinning for many recent advances in the field, bridging value function regularization, exploration, hierarchical control, and dual ascent methodologies. The core mechanisms—multiple policy/value proposals, diversity-driven target selection, conservative or TD-error-based regularization, and ensemble averaging—have demonstrated efficacy across diverse RL domains. Potential future extensions include further theoretical analysis of multi-actor/critic convergence in non-stationary or high-dimensional settings, generalization to multi-agent systems, and the development of meta-learnt ensemble strategies. The integration of dual networks for coordination in hierarchical/layered settings represents another promising research axis (Yang et al., 3 Aug 2024).

Key References:

• “Double Actor-Critic with TD Error-Driven Regularization in Reinforcement Learning” (Chen et al., 28 Sep 2024)
• “Simultaneous Double Q-learning with Conservative Advantage Learning for Actor-Critic Methods” (Li et al., 2022)
• “AACHER: Assorted Actor-Critic Deep Reinforcement Learning with Hindsight Experience Replay” (Sehgal et al., 2022)
• “DAC: The Double Actor-Critic Architecture for Learning Options” (Zhang et al., 2019)
• “Coordinating Planning and Tracking in Layered Control Policies via Actor-Critic Learning” (Yang et al., 3 Aug 2024)
• “Boosting the Actor with Dual Critic” (Dai et al., 2017)
• “Actor-Critic or Critic-Actor? A Tale of Two Time Scales” (Bhatnagar et al., 2022)