Stackelberg Actor-Critic: A Game-Theoretic RL Approach

• Stackelberg Actor-Critic is a game-theoretic reinforcement learning framework that models the actor as a leader and the critic as a follower in a bilevel Stackelberg game.
• It updates the actor by computing the total derivative through the critic’s best response, incorporating mixed second-order information for more accurate policy gradients.
• Empirical studies demonstrate that SAC variants yield accelerated convergence, reduced variance, and improved performance over conventional actor–critic methods in both tabular and continuous control tasks.

Stackelberg Actor-Critic (SAC) methods constitute a game-theoretic framework for reinforcement learning (RL) in which the canonical actor–critic (AC) architecture is recast as a hierarchical, two-player Stackelberg game. Here, the actor ("leader") parameterizes the policy, while the critic ("follower") parameterizes the value function. The actor selects its parameters, anticipating the critic's best response, and updates itself using the total derivative of its objective—explicitly differentiating through the critic’s optimization. This stands in contrast to conventional AC algorithms, which use naive (individual) gradients that ignore this interdependence. Theoretical developments and empirical assessments demonstrate that the Stackelberg formulation results in refined policy gradients, improved learning dynamics, and, under certain cost structures and regularity assumptions, provable local convergence properties (Wen et al., 2021, Zheng et al., 2021).

1. Stackelberg Game Formulation for Actor–Critic

The Stackelberg Actor-Critic formalism treats the actor and critic as players in a bilevel optimization (Stackelberg) game. The actor (leader) has parameters $\theta \in \Theta \subset \mathbb{R}^{d_\theta}$, and the critic (follower) has parameters $q$ or $w \in W \subset \mathbb{R}^{d_w}$, depending on the parameterization scheme.

The protocol is as follows:

1. Actor step: The actor selects $\theta$.
2. Critic best-response: The critic solves its own objective for the given $\theta$: $q^*(\theta) \in \arg\min_{q'} J_q(\theta, q')$ (or for function-approximation: $w^*(\theta) \in \arg\min_{w'} L(\theta, w')$).
3. Actor update: The actor updates $\theta$, explicitly accounting for the dependency $q^*(\theta)$ or $w^*(\theta)$ on $q$0.

The objectives are typically:

• Actor: Surrogate return objective $q$1
• Critic: Bellman-residual objective $q$2 or mean-squared error $q$3 (Wen et al., 2021, Zheng et al., 2021).

This structure contrasts with classic AC setups, where the interaction is not modeled as a game, and actor updates do not account for the reaction of the critic to changes in the policy parameters.

2. Stackelberg Gradient and Total Derivative

The Stackelberg framework dictates that the actor should optimize its parameters using the total derivative (Stackelberg gradient) rather than just the individual gradient. For the general case,

$q$4

or, with critic parameters $q$5,

$q$6

Under on-policy tabular assumptions (Assumption 1), the Stackelberg gradient recovers the true policy gradient of the cumulative reward objective (Theorem 4.2 in (Wen et al., 2021)).

This total derivative requires computation of (or efficient approximation to) mixed second-order derivatives and Hessian-vector products, which can be computed with vector–Jacobian products and automatic differentiation (Zheng et al., 2021).

3. Algorithmic Implementation

Practical Stackelberg AC algorithms replace the exact Hessian inverse with a regularized inverse, often implemented approximately using conjugate-gradient (CG) solvers. The meta-algorithmic structure for the actor–leader variant is:

$w \in W \subset \mathbb{R}^{d_w}$0 (Zheng et al., 2021)

Critical implementation aspects include the use of Pearlmutter’s trick for efficient Hessian-vector products; a fixed small number (e.g., 5–10) of CG iterations is empirically sufficient. Semi-gradient variants replace exact gradients with sampled-based approximations for stability when dealing with stochastic environments (Wen et al., 2021).

When instantiated in off-policy settings, e.g., for Soft Actor-Critic (SAC) or DDPG, the Stackelberg update modifies only the actor gradient step, while the critic’s target and architecture remain as in the baseline algorithm (Zheng et al., 2021).

4. Theoretical Analysis and Local Convergence

Under standard stochastic-approximation conditions—Lipschitz continuity and boundedness of gradients, suitable step-size schedules (actor learning slower than critic), and well-behaved gradient noise—the discrete-time Stackelberg AC iterates converge almost surely to a local Stackelberg equilibrium $q$7 of the associated Stackelberg ODE:

$q$8

A local Lyapunov analysis establishes the local stability of the Stackelberg equilibrium (Zheng et al., 2021). Explicitly incorporating the critic’s response in the actor’s update yields learning dynamics that are less susceptible to cycling and often exhibit accelerated convergence compared to naive (simultaneous) gradient interaction.

5. Empirical Evaluation and Observed Performance

Empirical studies encompass both tabular (FourRoom) and continuous control domains (Pendulum-v0, Reacher-v2, HalfCheetah-v2, and several OpenAI Gym benchmarks). Comparative baselines include standard Actorₒ–Critic, Actor_g–Critic, and off-policy SAC or DDPG methods.

Findings include:

• In basic tabular settings (FourRoom), Stackelberg AC matches standard AC in sample efficiency and final return; Residual AC (a correction-based method) substantially exceeds both in learning rate and final performance (Wen et al., 2021).
• In continuous control, embedding Stackelberg actor updates within Soft Actor-Critic (Stack-SAC) or DDPG (DDPG-SL) yields parity or outperformance relative to baselines. Stackelberg-based methods demonstrate up to 2×–5× faster convergence and up to 10–30% higher final return in off-policy settings (Zheng et al., 2021).
• In single-step toy problems, Stackelberg gradients eliminate parameter cycling and drive direct convergence (Zheng et al., 2021).
• A consistent qualitative advantage is decreased variance and smoother learning dynamics, attributed to superior conditioning of actor updates.

A key empirical takeaway is that, despite the theoretical appeal, the practical gains of Stackelberg AC hinge on the accuracy and stability of sample-based approximations to second-order terms. In scenarios with high variance, the Residual-AC approach, which directly learns a “residual critic” to estimate the policy gradient correction, provides the strongest empirical improvements (Wen et al., 2021).

6. Algorithmic Variants and Extensions

The Stackelberg perspective provides a general meta-framework, not restricted to specific surrogate objectives or update rules. Notably:

• The framework can accommodate on-policy and off-policy algorithms (e.g., A2C, SAC, DDPG) by adapting the critic’s loss and actor’s surrogate to the base algorithm.
• Unrolling the critic for multiple gradient steps (i.e., approximating the best-response more accurately) can further improve empirical performance, approaching the idealized best-response assumption (Zheng et al., 2021).
• Regularization (e.g., $q$9 damping in the Hessian or stabilizing learning rate schedules) is crucial for stable learning in high-dimensional or sample-constrained settings.

A summary of algorithmic features is given below:

Variant	Critic Objective	Actor Update
Stackelberg AC	Bellman residual or MSE	Total derivative
Standard AC	Bellman residual or MSE	Naive (individual) grad
Residual-AC	Bellman residual + residual	Corrected grad

Stackelberg AC meta-algorithms can be incorporated into existing infrastructure with minimal alteration to critic training and by replacing only the actor’s update rule.

7. Context and Significance

The Stackelberg Actor-Critic framework bridges the gap between classical actor-critic techniques and true policy-gradient methods by accounting for the interplay between policy and value approximation. Theoretical results confirm that the Stackelberg gradient recovers the unbiased policy gradient under standard assumptions, and local convergence of corresponding stochastic approximation schemes can be proven (Wen et al., 2021, Zheng et al., 2021). Empirical results indicate that Stackelberg AC algorithms often yield equal or superior learning stability and speed, although the magnitude of improvement is sensitive to implementation details, the fidelity of critic approximation, and the variance of higher-order estimator terms.

A plausible implication is that further research into robust, low-variance estimation of Stackelberg correction terms or hybrid optimization strategies (as with Residual-AC) may provide additional gains, especially in the large-scale, high-variance regimes typical of deep RL. The Stackelberg formulation also positions AC algorithms within a broader class of bilevel optimization and game-theoretic learning frameworks, opening avenues for cross-pollination across these domains.