Mean Field Stackelberg AC-SMFG Framework

• Mean Field Stackelberg AC (AC-SMFG) is a framework that integrates actor–critic methods to compute equilibria in large-scale leader–follower mean field games.
• It employs a single-loop update scheme combining policy, critic, and mean-field estimation for scalable and sample-efficient optimization.
• The framework offers finite-time convergence guarantees with practical applications in economics and control, addressing challenges like gradient alignment and function approximation.

Mean Field Stackelberg AC (AC-SMFG) refers to a class of algorithms and theoretical frameworks for learning in Stackelberg Mean Field Games (SMFGs) using Actor–Critic (AC) methods. These frameworks address dynamic games where a single leader (major player) interacts strategically with an infinitely large population of followers (minor players), with the leader anticipating and shaping the population’s mean-field response. AC-SMFG algorithms are designed to efficiently optimize leader policies in large-scale or continuous-time environments via scalable and sample-efficient stochastic approximation, featuring convergence guarantees under realistic coupling between leader and follower objectives (Zeng et al., 18 Sep 2025, Bergault et al., 2023, Mi et al., 2024).

1. Stackelberg Mean Field Game Formulation

A Stackelberg Mean Field Game is a bi-level optimization problem involving a single leader and a continuum (or large finite set) of homogeneous followers. The components are:

• State Spaces: The leader controls a state $s^l \in \mathcal S^l$; each follower’s state is $s^f \in \mathcal S^f$. In canonical Markovian settings, all agents may share a common state space $S$.
• Action Spaces: The leader acts in $\mathcal A^l$, the followers in $\mathcal A^f$.
• Transition Dynamics: The system evolves with dynamics $P^{l, \mu}$ and $P^{f, \mu}$ for the leader and followers, respectively, both depending on the current mean field $\mu$.
• Mean Field: The empirical distribution $\mu \in \Delta_{\mathcal S^f}$ of follower states summarizes the aggregate effect of their actions.
• Reward Functions: The leader reward $r_l(s, b, \mu)$ and follower reward $s^f \in \mathcal S^f$0 may depend on both states, actions, and the mean field.
• Equilibrium Structure: The leader chooses a policy in anticipation of followers playing a Nash equilibrium in the induced MFG, i.e., followers optimize responses to the leader’s policies, and the leader optimizes knowing this reaction (Zeng et al., 18 Sep 2025, Mi et al., 2024).

The Stackelberg-MFG equilibrium is expressed as

$s^f \in \mathcal S^f$1

where $s^f \in \mathcal S^f$2, $s^f \in \mathcal S^f$3 are best-response operators for followers and leader, and the mean field is a fixed point.

2. Actor–Critic Algorithms for Stackelberg Mean Field Games

AC-SMFG methods use actor–critic stochastic approximation with online samples to optimize Stackelberg equilibria in mean field settings. The principal features are:

• Policy Parameterizations: Both leader and follower policies are parameterized functions (tabular softmax or neural networks), updated via policy gradients.
• Critic Networks/Values: Value functions (critics) estimate the (discounted) value functions under current policy parameters for both leader and representative follower.
• Mean-Field Estimation: The mean field $s^f \in \mathcal S^f$4 is recursively updated from observed state transitions, with tabular or parametric (e.g., Gaussian) updates.
• Update Schedule: AC-SMFG uses a single-loop update structure, alternating between actor (policy), critic (value), and mean-field updates within a single Markovian sample trajectory, avoiding expensive nested loops (Zeng et al., 18 Sep 2025).
• Gradient Alignment Property: The algorithm requires a gradient alignment condition that ensures optimizing the leader’s reward with respect to a partial gradient (holding mean-field fixed) still leads to improvement of the overall Stackelberg objective. This relaxes the restrictive independence requirement between the leader and the mean field seen in earlier works.

3. AC-SMFG Single-Loop Update Scheme

The AC-SMFG algorithm iteratively updates leader and follower policies ($s^f \in \mathcal S^f$5, $s^f \in \mathcal S^f$6), critics ($s^f \in \mathcal S^f$7, $s^f \in \mathcal S^f$8), and mean field ($s^f \in \mathcal S^f$9) using stochastic semi-gradient steps with two concurrent Markov chains. At each iteration $S$0:

• Sampling:
  • Path 1: $S$1 generated for policy/value updates.
  • Path 2: $S$2 generated for stationary mean field estimation.
• Leader Actor:

$S$3

• Follower Actor:

$S$4

• Mean-Field:

$S$5

• Critics: TD(0) updates for $S$6, $S$7.
• Step Sizes: Satisfy $S$8 (Zeng et al., 18 Sep 2025).

This structure preserves scalability and sample efficiency, as each iteration uses only two samples and all updates are online and incremental.

4. Theoretical Guarantees and Convergence Analysis

AC-SMFG possesses the first finite-time, non-asymptotic convergence guarantee for Stackelberg mean field games (Zeng et al., 18 Sep 2025). Specifically, under Lipschitz continuity, contraction of follower mean-field best-responses, ergodicity, and the gradient-alignment assumption, as well as properly chosen multiple-timescale diminishing step sizes, the algorithm achieves:

• Stationarity: The minimum leader-optimality residual $S$9.
• Sample Complexity: To achieve an $\mathcal A^l$0-stationary point, the algorithm needs $\mathcal A^l$1 samples.
• Lyapunov Analysis: The proof constructs a Lyapunov function combining policy sub-optimality, mean-field error, follower Bellman error, and value function error, and demonstrates one-step decay properties for each update (Zeng et al., 18 Sep 2025).

This is enabled by utilizing correlated parameter updates within the single loop and by exploiting the alignment between the true Stackelberg gradient and its tractable partial approximation.

5. Empirical Performance and Applications

AC-SMFG and related actor–critic SMFG algorithms have been validated on a variety of economic and control benchmarks:

• Environments: Market entrance games, location choice (beach-bar placement), and equilibrium pricing with continuous states and actions.
• Baselines: Compared to nested-loop actor–critic, fictitious-play mean field learning, and joint MARL (e.g., PPO-based joint policy updates).
• Results: AC-SMFG consistently outperforms baselines in both leader and follower rewards, convergence speed, and mean-field accuracy. It remains performant even when function approximation replaces tabular representations (e.g., neural networks for high-dimensional state/action spaces).
• Practical Implementation: Step sizes, projections, and mean field updates are robust and simple; no need for projection beyond simplex and value clipping. The approach generalizes to heterogeneous settings by modifying state/action/reward parameterization (Zeng et al., 18 Sep 2025, Mi et al., 2024).

The following table summarizes key algorithmic comparisons:

Algorithm	Update Structure	Convergence Guarantee
AC-SMFG	Single-loop AC	Finite-time provable (Zeng et al., 18 Sep 2025)
Nested AC	Inner-outer loops	Asymptotic only
Joint MARL (PPO)	Joint policy	Empirical only, no SMFG

6. Generalizations and Theoretical Extensions

AC-SMFG frameworks extend to settings with partial information and common noise, as in “Mean Field Games in a Stackelberg problem with an informed major player” (Bergault et al., 2023). In these models:

• The leader possesses private information, disclosed strategically through controls (“information signaling”).
• Followers play a Mean Field Nash equilibrium, conditional on the common signal (derived from leader actions).
• The leader’s optimization is over the induced posterior law (distributional control), with costs depending on the evolving mean field and posterior.
• Existence of solutions and approximate finite-population equilibrium can be established using stochastic control, propagation-of-chaos, and weak convergence techniques.

A plausible implication is that AC-SMFG provides a unifying learning-theoretic foundation for a wide class of Stackelberg control problems over populations, including those with incomplete information and continuous time.

7. Connections, Limitations, and Future Directions

AC-SMFG represents a scalable and theoretically grounded approach for solving Stackelberg equilibria in large-scale mean field systems. Its strengths include provable convergence, sample efficiency, and robustness to model coupling between leader and followers. Key limitations include the requirement of gradient alignment (which may be non-trivial in highly noncooperative or non-smooth domains), the Markovian sampling assumption, and the need for function approximation in high-dimensional settings.

Prominent directions for future research include:

• Relaxing alignment and regularity assumptions to broader dynamic settings.
• Extending to heterogeneous follower populations and adversarial leader–follower coupling.
• Incorporating richer information structures (common/private noise) and learning with partial observability (Bergault et al., 2023).
• Applying AC-SMFG to real-world problems in economics, infrastructure, and regulation, where computational tractability and non-asymptotic performance are critical (Mi et al., 2024).

By abstracting population-level interaction to tractable actor–critic updates, AC-SMFG provides a foundation for learning and controlling complex strategic systems with a leader–majority asymmetry.