Actor-Critic Framework in Reinforcement Learning

• Actor-critic framework is a reinforcement learning approach that combines policy search (actor) with value function estimation (critic) to enable efficient decision making across diverse domains.
• The framework alternates updates between the actor and critic, reducing variance in policy gradients while improving convergence and stability in high-dimensional or sample-constrained environments.
• Extensions such as Stackelberg methods, meta-critic models, and distributional objectives enhance sample efficiency, mitigate overestimation bias, and foster robust exploration in complex tasks.

The actor-critic framework constitutes one of the foundational approaches in modern reinforcement learning (RL), encompassing a wide range of algorithmic methods that couple explicit policy search (the "actor") with value function approximation (the "critic"). At its core, the framework alternates between learning a parameterized policy to select actions and training a value-based model to evaluate state-action pairs, thereby enabling efficient policy improvement across discrete and continuous domains, and facilitating adaptation to large-scale, high-dimensional, or sample-constrained RL tasks.

1. Core Architecture and Algorithmic Principles

In canonical actor-critic methods, the learning system maintains two primary components:

• Actor: The policy $\pi_\theta(a\,|\,s)$, parameterized by $\theta$, which stochastically or deterministically selects actions based on the current state. The actor is trained to maximize the expected return, either directly via the policy gradient theorem or through various surrogate objectives.
• Critic: A value function such as a state-value $V_\psi(s)$ or state-action value (Q-function) $Q_\psi(s,a)$, parameterized by $\psi$, which estimates the expected return (i.e., the cumulative discounted reward) under the actor's current policy. The critic is typically updated by minimizing some form of temporal-difference (TD) error, potentially using off-policy data.

The standard update loop interleaves critic and actor updates, allowing the critic to inform the actor's improvements—either by evaluating the value of chosen actions, providing advantage estimates, or serving as a baseline to reduce variance in policy gradient estimators. This division of labor yields a two-timescale learning process in which the critic rapidly tracks changes in value estimates, while the actor updates more conservatively to ensure stability (Shahrooei et al., 11 Jun 2025, Zheng et al., 2021).

2. Extensions and Theoretical Developments

The actor-critic framework has been generalized and extended in several key directions:

• Game-Theoretic and Bilevel Optimization: The interaction between actor and critic can be formulated as a hierarchical Stackelberg game, where one component acts as the leader and the other as the follower. Stackelberg actor-critic methods replace the actor's standard gradient with a total derivative that anticipates the critic's best response, thereby accounting for the bilevel structure of the optimization and eliminating cycling in the updates. Convergence to local Stackelberg equilibria is guaranteed under mild assumptions, and empirical results indicate acceleration in training compared to naive, simultaneous gradient schemes (Zheng et al., 2021).
• Meta-Critic and Functional Critic Modeling: Meta-critic approaches introduce an additional network that meta-learns a loss function for the actor, enabling rapid adaptation within the learning process and improving sample efficiency when paired with off-policy actor-critic methods such as DDPG, TD3, or SAC. Functional critic modeling, in contrast, treats the critic as a mapping from policies to value functions, which stabilizes off-policy evaluation by generalizing across policy changes and produces unbiased off-policy policy gradients. This resolves both the "moving target" and "deadly triad" issues, leading to provable convergence in the linear setting (Zhou et al., 2020, Bai et al., 26 Sep 2025).
• Distributional and Risk-Sensitive Critic Objectives: Distributional actor-critic methods (such as GMAC) operate by learning value distributions via Cramér distance minimization with multi-step Bellman targets, sometimes parameterized by Gaussian mixture models, addressing distributional instability and improving performance across action types and environments. Risk-sensitive extensions augment the objective to trade off expected return against variance (J−λVar), requiring compatible linear critics for both the first and second moments, and yielding unbiased policy gradients to local optima of variance-adjusted reward (Nam et al., 2021, Tamar et al., 2013).
• Initiative Advisor and Exploration Innovations: Advisor-in-the-loop actor-critic (Ask-AC) introduces explicit mechanisms for querying external advice, using uncertainty-driven ask modules and adaptive selectors to initiate assistance when the agent's value function is unreliable, leading to significantly improved sample efficiency and robustness in non-stationary or safety-critical domains (Liu et al., 2022). Algorithmic innovations such as the virtual actor (VAAC) and Wasserstein barycenter soft actor-critic (WBSAC) enhance exploration efficiency, respectively by leveraging predictive novelty and entropy of imagined actions or by adaptively blending pessimistic and optimistic actors for high-diversity data acquisition (Park et al., 2023, Shahrooei et al., 11 Jun 2025).

3. Formal Frameworks and Algorithmic Realizations

Below is a comparative summary of representative actor-critic instantiations and their theoretical or empirical properties:

Reference	Main Variants & Innovations	Theoretical Guarantees
(Zheng et al., 2021)	Stackelberg actor-critic (bilevel leader-follower, total derivative)	Local Stackelberg equilibrium
(Zhou et al., 2020)	Online meta-critic for off-policy learning, meta-learned actor loss	Standard bilevel/TD stability
(Bai et al., 26 Sep 2025)	Functional critic: $\hat Q(\pi,s,a)$, exact off-policy gradients	Provable linear convergence
(Nam et al., 2021)	GMAC: Distributional Bellman backups, SR($\lambda$), GMM critic	Empirical gains, distributionality
(Shahrooei et al., 11 Jun 2025)	WBSAC: Wasserstein barycenter of pessimistic/optimistic policies	Barycenter entropy lower-bound, empirical SOTA
(Park et al., 2023)	VAAC: Virtual actor with novelty-driven entropy regularization	Empirical exploration gains
(Oren et al., 2024)	Value-improvement operator applied only to critic update (VI-AC)	Generalized Policy Iteration

Implementation details—such as the use of entropy bonuses, KL-divergence projections, mixture policies, or specialized meta-optimization—are chosen according to target domain properties, sample constraints, and required convergence guarantees.

4. Empirical Domains and Application Variants

The flexibility of the actor-critic framework has driven its adoption across a spectrum of challenging domains:

• Continuous Control: Algorithms like SAC, TD3, DDPG, and their value-improved, meta-critic, or dual-critic variants dominate high-dimensional benchmark tasks (e.g., MuJoCo, DeepMind Control Suite), where sample complexity, exploration, and stability are critical (Shahrooei et al., 11 Jun 2025, Zhou et al., 2020, Oren et al., 2024).
• Discrete-Action RL: In off-policy settings, carefully decoupling entropy regularization in actor and critic (as in discrete SAC variants) closes the gap to value-based methods (DQN) and provides robust learning in Atari-scale environments (Asad et al., 11 Sep 2025).
• Advisory RL & Safe Learning: Ask-AC and Actor-Advisor architectures explicitly address safe learning, human-in-the-loop RL, and transfer learning by incorporating initiative advisor queries, deterministic backup policies, and policy mixture mechanisms (Liu et al., 2022, Plisnier et al., 2019).
• Sequence Modeling and Generative Tasks: Actor-critic approaches have been instrumental in language modeling, sequence generation under adversarial or summary-quality critics, and discrete event generation (Goyal et al., 2017, Li et al., 2018).
• Simulation-Based Optimization: By encoding the sampling process as a policy selection problem in degenerate MDPs, the actor-critic paradigm enables efficient optimization over both continuous and discrete design domains (Li et al., 2021).

5. Algorithmic Variants: Stability, Sample Efficiency, and Policy Improvement

Key design tensions and resolution strategies include:

• Gradient-based vs. Greedy Improvement: Classic actor-critic is gradient-based in the actor, which is less greedy and more stable than methods using a hard $\argmax$ (Q-learning). Value-improved actor-critic (VI-AC) introduces a second, possibly non-parametric, greedification operator in the critic update, yielding more aggressive but still stable value boosts (Oren et al., 2024).
• Distributional & Multimodal Policy Support: Recent advances eliminate explicit actor networks entirely, generating actions by sampling from the gradient field of a single noise-level critic, as in ACA, which supports multi-modality and efficient policy improvement with reduced parameter count (Ki et al., 25 Sep 2025).
• Exploration & Regularization: Novel mechanisms such as virtual actors (VAAC), optimistic-actor ensembles (WBSAC), and entropy-fused LLM critics (SAMALM) direct exploration into under-represented or risky regions, improving state coverage and robustness (Park et al., 2023, Shahrooei et al., 11 Jun 2025, Wang et al., 12 Mar 2025).

6. Unified and Dual Objectives

Recent work emphasizes the formal unification of the actor and critic via:

• Saddle-Point Architectures and Duality: Dual Actor-Critic (Dual-AC) optimizes a shared Lagrangian, derived from Bellman’s dual LP, via multi-step bootstrapping and path regularization, reducing actor-critic mismatch and improving stability (Dai et al., 2017).
• Decision-Aware Lower-Bound Optimization: Decision-aware actor-critic algorithms jointly optimize a surrogate lower bound on the policy's performance that tightly couples actor and critic updates through Bregman divergences, supporting monotonic improvement guarantees (Vaswani et al., 2023).

7. Convergence, Scalability, and Open Challenges

The actor-critic framework's scalability and convergence properties rest on time-scale separation between actor and critic, policy and value function class expressivity, and the statistical alignment between the actor’s performance surrogate and the critic’s estimated values. While classic actor-critic under function approximation can be unstable (the "deadly triad"), the use of functional critics, dualized objectives, and value-improved backups yields provable local/global convergence in both linear and deep RL settings (Bai et al., 26 Sep 2025, Zhou et al., 2024, Oren et al., 2024).

Persistent challenges include efficient off-policy gradient estimation, reducing overestimation bias in value-based critics, optimizing exploration in sparse-reward or high-dimensional environments, and integrating non-differentiable advisory or safety constraints without biasing the core policy-gradient updates.

---

References

• Stackelberg Actor-Critic: Game-Theoretic Reinforcement Learning Algorithms (Zheng et al., 2021)
• Online Meta-Critic Learning for Off-Policy Actor-Critic Methods (Zhou et al., 2020)
• Functional Critic Modeling for Provably Convergent Off-Policy Actor-Critic (Bai et al., 26 Sep 2025)
• GMAC: A Distributional Perspective on Actor-Critic Framework (Nam et al., 2021)
• Wasserstein Barycenter Soft Actor-Critic (Shahrooei et al., 11 Jun 2025)
• Virtual Action Actor-Critic Framework for Exploration (Park et al., 2023)
• Value Improved Actor Critic Algorithms (Oren et al., 2024)
• Decision-Aware Actor-Critic with Function Approximation and Theoretical Guarantees (Vaswani et al., 2023)
• Boosting the Actor with Dual Critic (Dai et al., 2017)
• Actor-Critic without Actor (Ki et al., 25 Sep 2025)
• Revisiting Actor-Critic Methods in Discrete Action Off-Policy Reinforcement Learning (Asad et al., 11 Sep 2025)
• Ask-AC: An Initiative Advisor-in-the-Loop Actor-Critic Framework (Liu et al., 2022)
• Variance Adjusted Actor Critic Algorithms (Tamar et al., 2013)
• ACtuAL: Actor-Critic Under Adversarial Learning (Goyal et al., 2017)
• Actor-Critic based Training Framework for Abstractive Summarization (Li et al., 2018)
• An Actor-Critic Method for Simulation-Based Optimization (Li et al., 2021)