Stackelberg Actor–Critic Algorithms

• Stackelberg actor–critic algorithms are reinforcement learning methods that reformulate the actor–critic paradigm as a leader–follower (bilevel) game, ensuring corrections via implicit differentiation.
• They utilize total (hyper-)gradient computations to adjust actor updates in response to the critic’s best-response dynamics, yielding improved sample efficiency and stabilization.
• In multi-agent settings, these methods achieve natural Pareto-optimal equilibria and outperform Nash-based approaches by mitigating symmetry and cycling issues.

Stackelberg Actor–Critic algorithms constitute a class of reinforcement learning (RL) and multi-agent RL (MARL) methods that interpret the actor–critic paradigm as a bi-level (bilevel) or hierarchical optimization, formalized as a Stackelberg game. In this framework, the actor (leader) selects its policy, cognizant of the fact that the critic (follower) adapts in response. This induces distinct actor updates based on the total (hyper-)gradient of the expected return with respect to the policy parameters, accounting for the implicit response of the critic to the policy. Stackelberg actor–critic approaches have been shown to yield improved theoretical guarantees, more stable optimization dynamics, enhanced sample efficiency, and—particularly in multi-agent settings—natural Pareto-optimality and improved equilibrium selection over Nash-based actor–critic counterparts.

1. Game-Theoretic and Bilevel Formulation

Stackelberg actor–critic methods formalize the actor–critic interaction as a two-player Stackelberg (leader–follower) game. Let $\theta$ denote the actor (leader) parameters and $\phi$ (or $q$, $\omega$) the critic (follower) parameters. The actor’s objective is usually the expected (discounted) return, written as $$J(\theta, \phi) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)]$$, and the critic’s objective is typically the value or Bellman residual loss, $L(\theta, \phi)$, possibly $$\frac{1}{2}\mathbb{E}_{s}[(V^{\pi_\theta}(s) - V_\phi(s))^2]$$ or variants.

The bilevel optimization becomes: $$\min_\theta F(\theta) \equiv -J(\theta, \phi^*(\theta))\qquad \text{s.t.}\quad \phi^*(\theta) \in \arg\min_\phi L(\theta, \phi)$$ The Stackelberg equilibrium is defined by:

• The critic best-responds: $\phi^*(\theta) = \arg\min_\phi L(\theta, \phi)$,
• The actor best-responds to the optimized critic: $$\theta^* = \arg\max_\theta J(\theta, \phi^*(\theta))$$.

This leader–follower setup generalizes to multi-agent Markov games, where one agent (leader) optimizes policy anticipating the follower's best response, yielding Stackelberg equilibria—a strict refinement over Nash equilibria in settings with multiple possible equilibria (Zhang et al., 2019).

2. Stackelberg Policy Gradient and Hypergradient

Distinguishing Stackelberg actor–critic updates from standard actor–critic and policy gradient methods requires differentiating through the critic’s best-response. The actor’s total derivative is: $\phi$0 Here, the correction term captures how changes in the actor parameters indirectly affect the expected return via the critic’s adjustment.

In the context of actor–critic with a Bellman-residual critic, this correction ensures the actor update is an unbiased estimator of the true policy gradient, under suitable weighting. In both finite (tabular RL) and continuous-control domains, this total derivative—termed the Stackelberg policy gradient—can be obtained via (i) implicit differentiation through the fixed-point equation defining the critic, or (ii) introducing a residual-valued critic that estimates the necessary correction term (Wen et al., 2021, Zheng et al., 2021).

3. Practical Algorithms: Meta-Frameworks and Variants

Several Stackelberg actor–critic algorithms instantiate the general framework described above:

Stack-AC: Directly computes the Stackelberg policy gradient using implicit differentiation, involving Hessian-inverse/Jacobian-vector computations for the correction term. Typically, the critic is trained to local optimality in an inner loop, after which the actor update uses the total derivative. Regularization may be applied for invertibility (Zheng et al., 2021, Wen et al., 2021, Prakash et al., 16 May 2025).

Residual Actor–Critic (Res-AC): Approximates the correction term by learning a separate “residual” critic to estimate the policy-gradient of the Bellman residual, then combining this with the standard actor–critic update. This avoids costly second-order computations (Wen et al., 2021).

Bi-Level Policy Optimization (BLPO) with Nyström Hypergradients: Recasts policy optimization as a Stackelberg bilevel problem and uses a nested critic best-response. To stably compute the hypergradient, this approach leverages a Nyström low-rank Hessian approximation for the inverse Hessian–vector product, reducing memory and time complexity versus direct Hessian inversion or conjugate-gradient methods (Prakash et al., 16 May 2025).

Bi-level Actor-Critic for Multi-agent Coordination: Explicitly implements the leader–follower hierarchy in MARL, maintaining leader and follower critics and a follower actor, and solving for Stackelberg equilibria in Markov games. During training, the follower actor optimizes against the observed leader actions, while the leader acts optimally given the follower’s best response (Zhang et al., 2019).

Algorithm	Key Mechanism	Second-Order Terms
Stack-AC	Implicit Differentiation	Yes
Res-AC	Auxiliary Residual Critic	No
BLPO-Nyström	Nyström IHVP for Hypergradient	Yes (Low-rank)
Bi-AC (Multi-agent)	Leader–Follower Policy Training	No or Yes, domain-dependent

Stackelberg actor–critic algorithms are unified by their leader–follower nesting and correction to the standard actor update via the total derivative.

4. Convergence Theory and Optimization Properties

Stackelberg actor–critic algorithms offer improved convergence and stability relative to vanilla actor–critic approaches:

• Contraction and Two-Timescale Analysis: Under suitable smoothness, convexity, and stochastic approximation conditions (e.g., $\phi$1), joint convergence to local Stackelberg equilibria is guaranteed. In the two-timescale setting, the critic (follower) is updated to near-optimality for each actor (leader) step (Zhang et al., 2019, Zheng et al., 2021).
• Local Strong Stackelberg Equilibrium: For strongly convex critic objectives and Lipschitz gradients, policy parameters converge in polynomial time to $\phi$2-stationary points of the bilevel objective using nested updates and accurate hypergradient computation (specifically, Nyström-based approximations satisfy provable error bounds with high probability) (Prakash et al., 16 May 2025).
• Bias Correction and Policy Gradient Consistency: Analytical results confirm that the Stackelberg policy gradient recovers the true policy gradient in expectation, eliminating the bias present in standard actor–critic under function approximation (Wen et al., 2021, Zheng et al., 2021).
• Cycling Mitigation and Asymptotic Stability: Leader–follower updates dampen the oscillatory and cycling dynamics found in alternating-gradient methods. The learning dynamics respect the ODE associated with the Stackelberg policy gradient, ensuring local asymptotic stability near a differentiable Stackelberg equilibrium (Zheng et al., 2021).

5. Multi-Agent Reinforcement Learning Applications

Stackelberg actor–critic algorithms are particularly salient in MARL, especially when equilibrium selection and Pareto efficiency are paramount:

• Equilibrium Selection: Standard MARL approaches such as MADDPG and Nash-Q converge to arbitrary Nash equilibria, which may be symmetric and Pareto-suboptimal. By introducing explicit leader–follower asymmetry, bi-level actor–critic algorithms achieve Stackelberg equilibria, providing “certitude” in equilibrium selection (Zhang et al., 2019).
• Empirical Evaluation: On cooperative matrix games (Escape, Maintain), Bi-AC achieved Stackelberg equilibria in up to 100% of runs and found asymmetric, Pareto-superior solutions unattainable by Nash-equilibrium algorithms (e.g., I-DQN 37%, MADDPG 56%, vs. Bi-AC 90–100%) (Zhang et al., 2019).
• Highway Merge Task: In this setting, Bi-AC assigned consistent passing priority (main-lane car passes first in about 72% of episodes, collisions <5%), while Nash-based baselines failed to break symmetry and had higher collision rates (Zhang et al., 2019).
• Practical Execution: Training is centralized (access to all agents and gradients), but execution is decentralized: follower actors can act independently, having learned to respond optimally to the leader.

The Stackelberg framework thus resolves symmetry/coordination failures, yielding asymmetric equilibria on the Pareto frontier without requiring test-time communication.

6. Empirical Performance and Practical Considerations

Empirical studies corroborate the theoretical benefits of Stackelberg actor–critic approaches in classical control, continuous control, and MARL:

• Sample Efficiency and Return: Stackelberg-corrected methods (e.g., Res-AC, Stack-AC, BLPO-Nyström) match or substantially exceed the sample efficiency and asymptotic return of baseline actor–critic or policy gradient algorithms in both tabular and deep control benchmarks (e.g., Pendulum, Reacher, HalfCheetah) (Wen et al., 2021, Zheng et al., 2021, Prakash et al., 16 May 2025).
• Ablation Studies: Experiments reveal that both nesting of the critic update and the hypergradient correction are necessary for performance gains; omitting either degrades sample efficiency or final return.
• Stability: Stackelberg actor–critic methods manifest reduced oscillations, mitigate cycling, and exhibit more stable convergence trajectories compared to alternating-gradient baselines (Zheng et al., 2021).
• Computational Overhead: Native Stackelberg updates incur additional costs (second-order derivative computation), although low-rank (e.g., Nyström) or residual-critic approaches ameliorate practical resource requirements (Prakash et al., 16 May 2025, Wen et al., 2021).
• Implementation: Architectures typically use multi-layer MLPs for policy and value approximators, replay buffers, target networks, and established exploration and stabilization heuristics.

7. Theoretical and Practical Implications

Stackelberg actor–critic algorithms represent a rigorous correction over standard actor–critic, leveraging game-theoretic structure and bilevel optimization to align the practical update rules with the true gradient of cumulative return. In multi-agent and single-agent RL, this yields algorithmic improvements in convergence, stability, and solution quality—especially in settings with multiple equilibria or significant nonstationarity induced by agent interaction. The Stackelberg approach also unifies disparate perspectives—policy gradient as Stackelberg game, MARL equilibrium refinement, and bilevel optimization—highlighting the value of hierarchical thinking in RL algorithm design (Wen et al., 2021, Zheng et al., 2021, Prakash et al., 16 May 2025, Zhang et al., 2019).