Actor-Critic Approaches

• Actor-Critic approaches are a class of reinforcement learning methods that combine a differentiable policy (actor) with a value estimator (critic) to balance bias and variance.
• They employ techniques such as temporal-difference learning, off-policy updates, and variance reduction to achieve stable and efficient performance across various domains.
• Advanced variants include entropy-regularized, decoupled, and simulation-based methods that enhance exploration, robustness, and theoretical convergence properties.

Actor-critic approaches constitute a fundamental class of reinforcement learning (RL) algorithms that alternate between estimating value functions (the “critic”) and performing policy improvement (the “actor”). Actor-critic methods maintain both a parameterized policy—enabling direct policy search in continuous or discrete action spaces—and a value function estimator, which is used to reduce variance and bias in policy optimization. The following sections detail the mathematical formulation, algorithmic variants, theoretical and empirical properties, and representative extensions across contemporary actor-critic research.

1. Mathematical Framework and Key Components

Actor-critic methods are defined by the interaction of two principal elements:

• Actor: a differentiable stochastic (or deterministic) policy, typically parameterized by a neural network, denoted π_θ(a|s) where θ are policy parameters and (s,a) denotes the state-action pair.
• Critic: an approximation to the action-value function Q^π(s,a), parameterized as Q_w(s,a) with parameters w, or, in some variants, as a state-value V_w(s).

The policy optimization objective is generally: $J(θ) = \mathbb{E}_{s∼d^π,\,a∼π_θ}[\,Q^π(s,a)\,]$ The policy gradient theorem gives: $$∇_θJ(θ) = \mathbb{E}_{s∼d^π,\,a∼π_θ}[∇_θ\log π_θ(a|s)\;Q^π(s,a)]$$ The critic is updated to approximate Q^π by minimizing a Bellman-consistent objective, for example: $$L(w) = \mathbb{E}_{(s,a,r,s')}\left[\, (r + γV_w(s') - Q_w(s,a))^2\, \right]$$ where V_w(s) = ∑a πθ(a|s) Q_w(s,a) in the discrete-action setting (Allen et al., 2017).

2. Core Algorithmic Variants

2.1 Standard Online and Off-Policy Actor-Critic

Classic actor-critic implementations may be on-policy or off-policy. The critic is trained using temporal-difference (TD) updates or Monte Carlo rollouts. In single-time-scale schemes, both actor and critic are updated with similar step-sizes, shown to converge to a neighborhood of the optimum under linear function approximation (0909.2934). Sample complexity analyses reveal that the estimation error and bias of the critic remain primary bottlenecks in overall learning efficiency (Kumar et al., 2019).

2.2 Low-Variance and Variance-Controlled Estimators

The Mean Actor-Critic (MAC) approach replaces the sampled-action gradient

$∇_θ \log π_θ(a|s) Q_w(s,a)$

with the marginal average over all discrete actions: $∑_{a ∈ A} ∇_θ π_θ(a|s) Q_w(s,a)$ This yields an unbiased gradient estimator with provably reduced variance compared to sample-based actor-critic updates, provided π is non-deterministic (Allen et al., 2017). No action-independent baselines or advantage terms further reduce variance in this sum-over-actions setting.

Variance-adjusted actor-critic methods optimize risk-sensitive objectives of the form

$η(θ) = J^θ(x₀) - μ\,V^θ(x₀)$

where V^θ(x₀) is the variance of cumulative return, and derive corresponding compatible-function approximators for both expected return and return variance (Tamar et al., 2013).

2.3 Off-Policy, Entropy-Regularized, and Decoupled Approaches

Recent works identify the coupling of entropy regularization in both actor and critic as suboptimal in discrete-action off-policy algorithms such as Discrete Soft Actor-Critic (DSAC). Decoupling the critic's entropy coefficient (ζ) from the actor's entropy coefficient (τ) avoids systematic underestimation in Q-learning and enables competitive or superior performance to value-based methods like DQN (Asad et al., 11 Sep 2025). The m-step soft or hard Bellman operator is employed for critic updates, while the actor update uses NPG/SPMA variants with (forward or reverse) KL projections.

2.4 Actor-Critic without Explicit Actor

The ACA (“Actor-Critic without Actor”) framework eliminates the explicit actor network, instead generating actions directly from the gradient field of a noise-conditioned critic trained via a denoising diffusion process. Policy sampling is realized implicitly as a sequence of reverse-diffusion steps, and policy improvement is tightly coupled to the most recent critic estimates. This reduces parameter count and synchronization issues between actor and critic networks, while providing strong multi-modal exploration capabilities (Ki et al., 25 Sep 2025).

2.5 Simulation-Based Optimization and Hyperpolicy

Actor-critic algorithms extend beyond MDPs to black-box simulation-based optimization. In purely simulation-based contexts, the actor encodes a stochastic or deterministic generative model over design variables x, while the critic regressively estimates objective values f(x). For discrete spaces, the optimal policy is shown to be the energy-based softmax distribution over Q-values, while for continuous domains, entropy regularization continues to play a central role in balancing exploration and concentration (Li et al., 2021).

3. Exploration, Regularization, and Extensions

3.1 Explicit Behavior Modeling for Efficient Exploration

Behavior-Guided Actor-Critic (BAC) augments traditional off-policy actor-critic with a behavioral novelty bonus. A policy-conditioned autoencoder is trained to reconstruct state-action pairs, with the reconstruction error serving as an implicit, scalable measure of visitation frequency. This bonus is integrated additively in the Bellman target and critical loss, persistently directing exploration towards less-visited regions in both stochastic and deterministic actor regimes (Fayad et al., 2021).

3.2 Optimistic and Opportunistic Critics

To address systematic value underestimation in twin-critic frameworks (e.g., SAC), “optimistic critics” replace the minimum aggregation in target computation with a mean, max, or median over multiple critics. This reduces over-conservatism and enables smaller-capacity actors to achieve strong performance by maintaining a richer, more diverse replay buffer (Mastikhina et al., 1 Jun 2025, Roy et al., 2020). OPAC expands this idea further, using three critics and dynamically selecting between mean or median aggregation depending on observed Bellman error statistics (Roy et al., 2020).

3.3 PAC-Bayesian and Decision-Aware Critic Objectives

PAC-Bayesian actor-critic approaches use PAC-Bayes generalization bounds on the critic’s Bellman error, incorporating complexity penalties (KL divergence to a prior) and explicit uncertainty terms. This yields lower regret and more stable updates compared to MSE-trained critics, and can be exploited for critic-guided multiple-shooting candidate selection in action sampling (Tasdighi et al., 2023).

“Decision-aware” actor-critic designs propose a joint lower bound on performance involving both actor and critic through a mirror-descent framework. The actor optimizes a surrogate incorporating the critic’s gradient estimates, while the critic is trained to minimize a tailored Bregman divergence measuring decision misalignment rather than solely mean-squared TD error, with provable monotonic improvement guarantees (Vaswani et al., 2023).

3.4 Adversarial and Advisor-Driven Actor-Critic

The Adversarially Guided Actor-Critic (AGAC) introduces an auxiliary adversary policy trained to mimic the actor. The actor is simultaneously trained to maximize reward and differentiate from the adversary, operationalized via an augmented advantage and KL-controlled intrinsic bonus, which provably enhances exploration in sparse-reward and procedurally generated domains (Flet-Berliac et al., 2021).

The Actor-Advisor formulation unifies off-policy critic learning (arbitrary state-of-the-art value estimators) with policy-gradient actors by using “shaped” advice—in the form of a softmax over critic Q-values—to bias the actor in favor of high-reward actions while retaining unbiased learning from Monte Carlo returns. This hybridizes sample efficiency and stability with actor-critic convergence properties, with broad extension to safe RL and policy distillation (Plisnier et al., 2019).

4. Theoretical Properties, Complexity, and Convergence

Convergence and sample complexity analyses in actor-critic schemes reveal:

• In single-time-scale regimes (identical step-sizes for actor and critic), convergence is typically to a neighborhood of the optimal point with bias dependent on critic approximation error (0909.2934).
• Two-time-scale updates (slower actor, faster critic) can provide asymptotic optimality if the critic tracks the value of the current actor sufficiently fast (Kumar et al., 2019).
• The primary learning bottleneck under function approximation is the critic’s estimation bias; increasing critic accuracy (via additional inner-loop steps or multiple ensemble critics) proportionally reduces the final actor suboptimality (Kumar et al., 2019).
• Decision-aware and PAC-Bayesian losses guarantee monotonic policy improvement and bounded regret w.r.t. final critic error (Vaswani et al., 2023, Tasdighi et al., 2023).
• In settings such as offline RL, pessimistic actor-critic with Bellman-closed function approximation classes yields minimax-optimal bounds depending on data coverage and problem dimension (Zanette et al., 2021).

5. Practical Implementations and Domain-Specific Extensions

A multitude of modern actor-critic architectures share canonical implementation ingredients:

• Replay buffers, target-network Polyak averaging for stability.
• Double or triple critics for bias-variance control.
• Entropy or optimism coefficients—either hand-tuned or automatically adapted—to regulate exploration.
• Critic value targets often computed using minimum, mean, or median Q estimates over bootstrapped critics.

Recent extensions include:

• Discrete-action actor-critic using decoupled entropy for DQN-level performance on Atari games (Asad et al., 11 Sep 2025).
• Hybrid model-predictive control architectures using actor-critic policies for warm-starting and trajectory cost evaluation within MPC optimization loops, with explicit performance guarantees relative to RL policy alone (Reiter et al., 2024).
• Zeroth-order compatible policy gradient estimators employing two-point finite differences, resolving incompatibility between critic approximation and DPG requirements in deep off-policy RL (Saglam et al., 2024).
• Quantum circuit–based actor-critic implementations, where variational quantum circuits replace either classical actor or critic, leveraging hybrid quantum-classical models for efficient policy representation on small quantum hardware (Kölle et al., 2024).

6. Summary Table: Representative Actor-Critic Extensions

Approach	Distinguishing Feature	Key Paper
Mean Actor-Critic	Action-marginalized low-variance update	(Allen et al., 2017)
Behavior-Guided AC	Autoencoder novelty-based exploration	(Fayad et al., 2021)
ACA (No explicit actor)	Critic-guided diffusion policy	(Ki et al., 25 Sep 2025)
DSAC (Decoupled Entropy)	Separate actor/critic entropy	(Asad et al., 11 Sep 2025)
Optimistic Critic	Agg_{mean,max} for target computation	(Mastikhina et al., 1 Jun 2025)
OPAC	Triple-critic, opportunistic aggregation	(Roy et al., 2020)
PAC-Bayesian SAC	PAC-Bayes generalization for critic	(Tasdighi et al., 2023)
Adversarially Guided	Actor repelled from adversary imitation	(Flet-Berliac et al., 2021)
Decision-Aware Critic	Surrogate/Bregman dual actor-critic loss	(Vaswani et al., 2023)
Zeroth-Order CPG	Two-point action-gradient for DPG	(Saglam et al., 2024)
Hybrid Quantum AC	VQC-based actor or critic	(Kölle et al., 2024)
AC4MPC	RL-augmented MPC with critique	(Reiter et al., 2024)

7. Outlook and Open Problems

Recent work highlights significant advances in efficiency, exploration, risk-sensitivity, stability, and scalability of actor-critic methods. Unresolved challenges remain:

• Adaptive selection of regularization coefficients (entropy, optimism).
• Alignment and compatibility between non-linear critic approximations and actor update rules.
• Architectures ensuring robustness with highly compact or multimodal policies.
• Theoretical guarantees in non-stationary, partial observability, or hardware-constrained regimes.

Actor-critic approaches will continue to serve as an essential backbone for scalable RL across domains spanning gaming, control, robotics, simulation-based design, and quantum-enhanced learning, with ongoing work continuing to bridge theory and practical performance (Allen et al., 2017, Fayad et al., 2021, Asad et al., 11 Sep 2025, Ki et al., 25 Sep 2025, Saglam et al., 2024).