Mean-Field Actor-Critic Flow

• MFAC flow is a continuous-time learning dynamics framework that integrates actor-critic methods with optimal transport to compute mean-field game equilibria.
• The framework updates policy, value function, and distribution through coupled PDE flows, ensuring global exponential convergence via Lyapunov analysis.
• Numerical experiments on LQ systemic risk, optimal execution, and flocking models demonstrate its robust performance in high-dimensional and non-linear settings.

The Mean-Field Actor-Critic (MFAC) flow is a continuous-time learning dynamics framework for solving mean-field games (MFGs), integrating reinforcement learning (RL) methods (specifically actor-critic algorithms) with optimal transport techniques. In this formulation, large populations of identical agents interact through an evolving population state distribution, and the learning process jointly updates individual control policies, value function estimations, and the distribution itself via interlinked gradient-based flows governed by partial differential equations (PDEs). A principal novelty is the Optimal Transport Geodesic Picard (OTGP) flow, which updates the distribution by interpolating along Wasserstein-2 geodesics towards the equilibrium. The MFAC flow admits a rigorous convergence analysis using Lyapunov functionals, establishing global exponential convergence under proper timescale separation. This unified approach yields both theoretical guarantees and practical algorithms for computing MFG equilibria, facilitating applications in high-dimensional and non-linear settings.

1. Structure and Principles of the MFAC Flow

MFAC flow models the solution procedure for MFGs as a set of coupled continuous-time flows in learning-time (denoted $\tau$), operating concurrently for the actor (policy), critic (value function), and the distribution. The central components evolve according to the following principles:

• Actor (Policy) Update: The control policy $\alpha^{\tau}(t, x)$ evolves according to a policy gradient update derived from the Hamiltonian $H$ of the underlying mean-field control problem:

$$\partial_{\tau} \alpha^{\tau}(t, x) = \beta_a \cdot H\left(t, x, \mu_t^{\tau}, \alpha^{\tau}(t, x), -G^{\tau}(t, x)\right)$$

where $G^{\tau}(t, x)$ is the critic's estimate of the gradient of the value function and $\beta_a$ is the actor learning rate.

• Critic (Value Function) Update: The critic estimates the value function $V^{\tau}$ and its spatial gradient $G^{\tau}$. Updates are based on shooting-method representations (via Itô’s formula):

$$\partial_{\tau} V_0^{\tau}(x) = \beta_c [V(0, x) - V_0^{\tau}(x)]$$

$$\partial_{\tau} G^{\tau}(t, x) = \beta_c \cdot 2D(t, x, \mu_t^{\tau}) \left[V(t, x) - G^{\tau}(t, x)\right]$$

with $\beta_c$ the critic learning rate and $D$ a problem-dependent matrix.

• Distribution (OTGP) Update: The key innovation is the update of the time-dependent population law $\mu_t^{\tau}$ via a flow in Wasserstein space:

$$\partial_{\tau} \mu_t^{\tau}(x) = \beta_\mu \nabla \cdot \left(\mu_t^{\tau}(x) \phi_t^{\tau}(x)\right)$$

where $\phi_t^{\tau}(x)$ is the Kantorovich potential (solving $x - T_t^{\tau}(x) = \phi_t^{\tau}(x)$), driving the distribution along the geodesic between the current and "target" measures. This approach efficiently ensures contraction towards equilibrium even in high-dimensional, non-linear scenarios (Zhou et al., 14 Oct 2025).

2. Mathematical Formulation and Coupled PDE Flow

The MFAC flow results in a dynamical system coupling three sets of PDEs for policy, value, and distribution evolution in the learning variable $\tau$ (distinct from physical time $t$):

Component	Update Equation (in $\tau$)	Function
Actor	$$\partial_{\tau} \alpha^{\tau}(t,x) = \beta_{a} H(t,x,\mu_t^{\tau},\alpha^{\tau}(t,x),-G^{\tau}(t,x))$$	Policy optimization
Critic	$$\partial_{\tau} V_0^{\tau}(x) = \beta_c [V(0,x) - V_0^{\tau}(x)]$$,<br> $$\partial_{\tau} G^{\tau}(t, x) = \beta_c \cdot 2D [V-G]$$	Value approximation
Distribution	$$\partial_{\tau} \mu_t^{\tau}(x) = \beta_\mu \nabla\cdot (\mu_t^{\tau}(x) \phi_t^{\tau}(x))$$	OTGP flow for PDF

Here, $H$ denotes the Hamiltonian for the MFG, $G$ approximates the gradient of the value function (used by the actor), and $\phi$ encodes the optimal transport direction derived from the Kantorovich potential for pushing the distribution towards the flux induced by the current policy control (Zhou et al., 14 Oct 2025).

The coupling of these flows models both agent-level response (actor and critic) and population response (distribution) within a unified mechanism, allowing the computation of mean-field Nash equilibria without explicit backward recursion.

3. Optimal Transport Geodesic Picard (OTGP) Flow

The OTGP flow is a central innovation for updating the distribution in MFAC:

• Rather than incrementally updating the distribution via samples or naive averaging, OTGP moves the law $\mu_t^{\tau}$ optimally (in the Wasserstein-2 metric) towards the measure $\rho_t$ induced by simulating the agent SDE under current policy parameters.
• The transport direction, specified via the Kantorovich potential, enables the evolution along geodesics, which is both computationally robust and theoretically justified by contraction properties in Wasserstein space.
• Efficient computation of the Kantorovich potential is feasible via algorithms such as the Hungarian algorithm, decoupling dimension dependence.

The OTGP flow ensures that distributional updates are consistent with the equilibrium structure induced by optimal control and provides a fixed-point style "Picard" iteration in measure space, analogous to fictitious play in standard game theory (Zhou et al., 14 Oct 2025).

4. Global Exponential Convergence: Lyapunov Analysis

MFAC flow's convergence is analyzed using a global Lyapunov function composed of actor, critic, and distribution error functionals:

• Actor Lyapunov: $$L_a^\tau = J^{\mu^\tau}[\alpha^\tau] - J^{\mu^\tau}[\alpha^{*,\mu^\tau}]$$ (cost gap versus optimal for fixed $\mu^\tau$)
• Critic Lyapunov: $L_c^\tau$ based on squared error between current and true value function (and gradient)
• Distribution Lyapunov: $L_\mu^\tau$ measured by a weighted Wasserstein-2 metric ($d_\beta(\mu^\tau,\rho)$) plus a terminal penalty

The derivative of the total Lyapunov function satisfies

$$\frac{d}{d\tau} L_{\text{total}}^\tau \leq -c_L L_{\text{total}}^\tau$$

for some $c_L > 0$, yielding

$$L_{\text{total}}^\tau \leq L_{\text{total}}^0 e^{-c_L \tau}$$

which guarantees global exponential convergence to the MFG equilibrium provided the speed ratios $(\beta_a, \beta_c, \beta_\mu)$ are chosen to satisfy mild constraints. The proof leverages performance-difference lemmas, stochastic Grönwall inequalities, and contraction properties from optimal transport (Zhou et al., 14 Oct 2025).

5. Algorithmic Realization and Numerical Experiments

The MFAC flow is discretized and implemented with neural network parameterizations for the actor (policy), critic (value), and a separate score network for the population distribution. Practical features include:

• Stochastic simulation of population dynamics using Euler–Maruyama integration for the SDEs.
• Distribution updating using score matching and integrated Langevin Monte Carlo with the learned score network.
• Neural architectures: Residual networks for function approximation; batch-based score estimation for robust distribution evolution.

Experiments demonstrate the framework's efficacy on:

• Linear–Quadratic (LQ) Systemic Risk Models: Recovery of Nash equilibrium policies and accurate estimation of value and state distributions, matching analytic solutions.
• Optimal Execution (Extended MFG): Accurate learning in settings where mean-field couplings occur via the action domain.
• Multi-dimensional Flocking models (Cucker–Smale): Scalability and effective approximation of equilibrium distributions in higher-dimensional complex systems.

Empirically, convergence curves for actor, critic, and distribution errors exhibit the predicted linear (exponential) decay, with performance consistent across both simple and high-dimensional non-linear MFG scenarios (Zhou et al., 14 Oct 2025).

6. Theoretical and Practical Implications

The MFAC flow provides a unified, scalable approach to model-free RL in MFGs:

• Single-timescale learning: The evolution of actor, critic, and measure can be performed concurrently in a single gradient-based pipeline, simplifying implementation compared to multi-timescale algorithms.
• Theory–practice alignment: The combination of Lyapunov-based exponential convergence guarantees and high empirical efficiency enables robust deployment.
• Methodological advances: Unification of RL and optimal transport enables tractable computation of measure-valued fixed points.

Anticipated research directions cited include:

• Extension to MFGs with common noise or non-convex costs
• More efficient or scalable parameterization and transport computation
• Relaxation of regularity or growth assumptions for broader applicability
• Enhanced gradient estimation for improved sample efficiency in high dimension

This framework sets the stage for advanced, theoretically grounded RL algorithms in large-scale, measure-coupled agent systems, with applications in economics, engineering, and beyond.