Actor-Critic Paradigm

• The Actor-Critic Paradigm is a reinforcement learning approach that separates decision-making (actor) and value estimation (critic) to continually improve policy performance.
• It alternates between evaluating current policies and updating actions based on gradient and value-based methods, enabling robust performance in complex environments.
• Recent innovations include multi-critic aggregation, optimism-driven exploration, and adaptive time-scale schemes, which together enhance sample efficiency and convergence.

The actor-critic paradigm in reinforcement learning (RL) designates a class of algorithms characterized by the interplay of two distinct components: the actor, which maintains a parameterized policy for selecting actions, and the critic, which evaluates the policy by approximating value functions. This general approach provides a framework for continuous policy improvement, combining the strengths of policy-gradient and value-based methods. Actor-critic algorithms have achieved state-of-the-art performance in settings ranging from continuous control to large-scale offline RL, but designing stable, efficient, and provably convergent actor-critic methods presents persistent algorithmic and theoretical challenges. Ongoing research addresses issues in exploration, bias-variance trade-offs, function approximation, offline data regimes, multi-critic aggregation, and time-scale scheduling between actor and critic updates.

1. Core Principles and Mathematical Formulation

The essential workflow of an actor-critic algorithm includes two alternating steps: policy evaluation and policy improvement.

• Actor: The actor $\pi_\theta(a|s)$ is a parameterized stochastic (or deterministic) policy (usually a neural network) responsible for selecting actions. The actor is updated with the objective of maximizing expected return under the current critic’s estimates. The canonical policy-gradient update is

$$\nabla_\theta J(\theta) = \mathbb{E}_{s \sim d^{\pi_\theta}, a \sim \pi_\theta}\left[\nabla_\theta \log \pi_\theta(a|s) Q_\phi(s,a)\right],$$

or, in practice, using batches sampled from a replay buffer (Roy et al., 2020, Maei, 2018).

• Critic: The critic $Q_\phi(s, a)$ or $V_\phi(s)$ approximates the policy’s expected return. The critic is updated to minimize the Bellman residual (e.g., mean squared Bellman error, MSBE):

$$J_Q(\phi) = \mathbb{E}_{(s, a, r, s')}\left[ \tfrac{1}{2} (Q_\phi(s,a) - (r + \gamma Q_{\bar\phi}(s', a'))) ^2 \right]$$

where $a'$ is sampled from $\pi_\theta(a'|s')$ and $\bar\phi$ is a slowly updated target network (Roy et al., 2020).

• Alternation and Bilevel Structure: Actor-critic is inherently a bilevel optimization, with the actor aiming to maximize the value function as evaluated by the current critic, while the critic tracks the evolving policy (Yang et al., 2019).

2. Algorithmic Variants and Time-Scale Schemes

A central consideration is the time-scale at which actor and critic updates are performed.

• Two Time-Scale (Standard): Traditionally, the critic is updated on a faster time-scale so it can track the evolving policy, emulating policy iteration. Under mild conditions, such two-time-scale schemes are provably convergent in the tabular and linear case (Bhatnagar et al., 2022, 0909.2934, Maei, 2018).
• Reverse Time-Scale (Critic-Actor): Swapping the time-scales, such that the actor is updated faster, emulates value iteration. This modification is theoretically sound and empirically competitive, especially under nonlinear function approximation (Bhatnagar et al., 2022).
• Single Time-Scale: Simultaneous actor and critic updates on the same time-scale can improve biological plausibility and are provably convergent to a neighborhood of local maxima, with practical benefits in initial learning speed (0909.2934).
• Stackelberg Actor-Critic: A game-theoretic formulation interprets the actor as a leader anticipating the best response from the critic, using total derivatives for actor updates, and demonstrates improved convergence properties (Zheng et al., 2021).

3. Advances in Critic Design and Aggregation

Recent developments target the critical influence of the critic on the overall training process:

• Multi-Critic Aggregation: Algorithms such as OPAC employ three critics and aggregators (e.g., mean-of-min-two, median-of-three) to reduce overestimation bias while controlling variance. This approach yields state-of-the-art performance and robust sample efficiency (Roy et al., 2020).
• Optimism and Exploration: To counteract underexploration, optimistic critics use upper confidence bounds or mean/max aggregation, as in OAC and asymmetric-actor frameworks, enhancing data collection and mitigating critic underestimation, especially when employing small or resource-constrained actors (Ciosek et al., 2019, Mastikhina et al., 1 Jun 2025).
• Functional Critic Modeling: Moving beyond policy-dependent critics, functional critics $\hat Q(\pi, s, a)$ model values across entire policy classes, providing a unified target-based algorithm with theoretically established convergence in off-policy regimes (Bai et al., 26 Sep 2025).
• Regularization and Decision-Awareness: TD-regularized actor-critic methods explicitly penalize inadequate critics in the actor’s update, while decision-aware actor-critic frameworks tightly couple critic and actor losses to guarantee monotonic policy improvement (Parisi et al., 2018, Vaswani et al., 2023).

4. Exploration, Greedification, and Policy Improvement Strategies

Exploration remains a central challenge:

• Standard Exploration: Policy outputs are often sampled from parameterized Gaussians with fixed or adaptive temperature, but such methods are inefficient in certain regimes due to uninformed or isotropic noise (Roy et al., 2020, Ciosek et al., 2019).
• Optimistic/Directed Exploration: OAC constructs upper confidence bounds using critic ensembles and derives exploration policies by maximizing these bounds under KL constraints, yielding principled state-dependent exploration that increases sample efficiency (Ciosek et al., 2019).
• Greedification Operators: Value-Improved Actor-Critic algorithms decouple gradient- and greedification-based improvements. A value-improvement operator greedifies the critic over sampled action proposals, blending stability from smooth actor updates with rapid critic improvement, leading to substantial efficiency gains (Oren et al., 2024).
• Dual and Guide Actor-Critic Approaches: Dual-AC and GAC algorithms restructure the actor update as an explicit optimization—minimax saddle point or Newton-guided in action space—solving jointly for Bellman-consistency and policy improvement (Dai et al., 2017, Tangkaratt et al., 2017).

5. Offline and Off-Policy Actor-Critic: Theory and Practice

Actor-critic methods have been successfully generalized to settings with off-policy data and restrictive data coverage:

• Offline Actor-Critic with Pessimism: Offline RL settings—where no further data can be collected—employ pessimistic critics via second-order cone programs to obtain robust lower bounds on the value of candidate policies, which are then maximized by the actor (e.g., PACLE). This design provides sharp, minimax-optimal suboptimality guarantees under mild coverage conditions (Zanette et al., 2021).
• Convergent Off-Policy Algorithms: Convergent AC variants (e.g., Emphatic Actor-Critic) compute the exact policy gradient in the presence of function approximation and off-policy data by correcting for changes in the stationary state distribution, using follow-on or emphatic weighting (Maei, 2018, Bai et al., 26 Sep 2025).
• Sample Complexity: The convergence rate of actor-critic, agnostic to specific policy evaluation strategies (e.g., TD, GTD, accelerated GTD), is explicitly characterized, with sample complexity determined by the critic’s convergence rate and bias-variance trade-offs (Kumar et al., 2019).

6. Empirical Results and Domain-Specific Innovations

Recent studies benchmark a wide spectrum of actor-critic algorithms across continuous control, navigation, and other RL domains:

Algorithm/Variant	Notable Features	Sample Efficiency/Returns	Environments	Reference
OPAC	Triple-critic, adaptive aggregation	Outperforms SAC/TD3	MuJoCo	(Roy et al., 2020)
VI-AC (TD3/SAC)	Value-improvement greedification inside critic	2× faster than baseline	DM Control Suite	(Oren et al., 2024)
Optimistic Actor-Critic/OAC	Upper confidence-bound-guided exploration	State-of-the-art sample efficiency	MuJoCo	(Ciosek et al., 2019)
Dual Actor-Critic	Saddle-point objective, path regularization	Highest score on hard tasks	MuJoCo	(Dai et al., 2017)
Functional Critic	Policy-functional Q, ensemble, exact off-policy grad	Matches best in class, stable	DM Control, RL Unplugged	(Bai et al., 26 Sep 2025)
Actor-Advisor	Off-policy critic provides sampling advice	Above on-policy AC, robust safety	2D Nav, gridworld	(Plisnier et al., 2019)
GAC	Guide actor via 2nd-order action-space optimization	Top or competitive performance	MuJoCo	(Tangkaratt et al., 2017)

Performance gains are generally realized via improved sample efficiency, reduced return variance, improved robustness to hyperparameter specification, and, in some cases, better asymptotic returns.

7. Theoretical Challenges, Limitations, and Future Directions

Actor-critic methods remain an active area of research due to several open problems:

• Deadly Triad: The combination of function approximation, bootstrapping, and off-policy data can cause divergence. Recent functional and target-based critics present avenues for provably convergent algorithms, even in challenging regimes (Bai et al., 26 Sep 2025).
• Bias-Variance Trade-off: Critic estimation error and update frequency directly influence sample complexity and convergence. Regularization and greedy improvement, while beneficial, can induce instability without careful variance control (Parisi et al., 2018, Oren et al., 2024).
• Structured and Asymmetric Architectures: Asymmetric actor-critic designs with small actors and large critics are promising for real-world deployment but require optimism in the critic to avoid pathological value underestimation and degraded data collection (Mastikhina et al., 1 Jun 2025).
• Unified and Actorless Architectures: Approaches that collapse the actor and critic into a single functional or diffusion-guided value estimator are gaining attention for their alignment, parameter efficiency, and multi-modality (Ki et al., 25 Sep 2025).
• Scaling and Generalization: Extensions to discrete and high-dimensional action spaces, multi-agent settings, and hierarchical or multi-level actor-critic decompositions are active research areas (Oren et al., 2024).

In summary:

The actor-critic paradigm encompasses a diverse set of algorithms unified by actor/critic separation and alternating updates. State-of-the-art instantiations incorporate multi-critic designs, optimism, functional critics, regularization, greedification, and adaptive time-scale scheduling—with rigorous theoretical foundations now available in several important cases (e.g., LQR, function approximation, offline RL). Challenges remain in ensuring stability, scalability, and efficient exploration, but recent work continues to broaden the paradigm’s empirical and theoretical reach (Roy et al., 2020, Bai et al., 26 Sep 2025, Ciosek et al., 2019, Oren et al., 2024, Bhatnagar et al., 2022, Mastikhina et al., 1 Jun 2025, Dai et al., 2017, Vaswani et al., 2023, Yang et al., 2019, Maei, 2018).