MF-SAC: Scalable Adaptive Control for Stochastic Systems

• MF-SAC is a distributed control framework for large-scale stochastic systems that uses mean field theory and adaptive estimation to achieve decentralized performance.
• It derives control laws via coupled Riccati equations and ODEs, yielding approximate Nash equilibria or socially optimal policies under uncertainty.
• The framework employs local observations and data-driven techniques, including RWLS and integral reinforcement learning, to ensure convergence and scalability.

Mean Field Stochastic Adaptive Control (MF-SAC) constitutes a distributed control framework designed for large-scale populations of interacting stochastic systems. It leverages mean field (MF) theory to provide scalable decentralized control strategies—yielding, in the limit, (approximate) Nash equilibria for noncooperative games or social-optimal policies for collectively optimizing populations. Critically, MF-SAC addresses parametric uncertainty and heterogeneity: agents adaptively estimate their own dynamics and key population statistics online, using local observations (and, optionally, sparse samples of the population), yielding strong theoretical guarantees in both Nash and social welfare contexts (Kizilkale et al., 2012, Xu et al., 2024).

1. Model Formulation and Problem Setting

The canonical MF-SAC problem is posed for a population of $N$ agents, each with independent continuous-time stochastic dynamics: $$dx_i(t) = \left(A_i x_i(t) + B_i u_i(t)\right)dt + D\,dw_i(t)$$ where $x_i(t)\in\mathbb{R}^n$ is the state, $u_i(t)\in\mathbb{R}^m$ is the control, $A_i,B_i$ are unknown agent-specific matrices, $D$ is a known noise matrix, and $w_i$ is a standard Wiener process (Kizilkale et al., 2012). The population is parameterized by unknown types $\theta_i = [A_i,B_i,Q_i]$, i.i.d. with respect to a distribution $F_\zeta(\theta)$ indexed by a hyperparameter $\zeta$. The agent’s performance is measured by its long-run average quadratic cost

$$dx_i(t) = \left(A_i x_i(t) + B_i u_i(t)\right)dt + D\,dw_i(t)$$0

where $$dx_i(t) = \left(A_i x_i(t) + B_i u_i(t)\right)dt + D\,dw_i(t)$$1 introduces a mean field (mass effect) term. In cooperative variants, a social cost sums or averages agent costs; for LQG networks with multiplicative noise, analogous structures arise with unknown coefficients $$dx_i(t) = \left(A_i x_i(t) + B_i u_i(t)\right)dt + D\,dw_i(t)$$2 and (potentially) social coupling matrices (Xu et al., 2024).

The MF-SAC paradigm assumes:

• Local observations: each agent measures only its own state and input.
• Population sampling: each agent may observe a vanishingly small random subset of other agents to estimate population statistics.
• Uncertain or unknown agent dynamics and cost parameters.
• Exogenous “mass effect” terms set by (possibly nonlinear) population averages, which must satisfy fixed-point self-consistency in the population limit.

2. Mean Field Equilibrium and Offline Solution

In the population limit ($$dx_i(t) = \left(A_i x_i(t) + B_i u_i(t)\right)dt + D\,dw_i(t)$$3), the mean field provides a Nash equilibrium for noncooperative agents (or a social optimum for cooperative problems), given knowledge of all relevant parameters and distributions. This equilibrium is characterized by:

• The agent’s optimal control,

$$dx_i(t) = \left(A_i x_i(t) + B_i u_i(t)\right)dt + D\,dw_i(t)$$4

where $$dx_i(t) = \left(A_i x_i(t) + B_i u_i(t)\right)dt + D\,dw_i(t)$$5 solves the agent-specific algebraic Riccati equation (ARE) and $$dx_i(t) = \left(A_i x_i(t) + B_i u_i(t)\right)dt + D\,dw_i(t)$$6 is an offset trajectory derived from an ODE depending on the mean field signal $$dx_i(t) = \left(A_i x_i(t) + B_i u_i(t)\right)dt + D\,dw_i(t)$$7.

• The self-consistency condition for the mass effect:

$$dx_i(t) = \left(A_i x_i(t) + B_i u_i(t)\right)dt + D\,dw_i(t)$$8

where $$dx_i(t) = \left(A_i x_i(t) + B_i u_i(t)\right)dt + D\,dw_i(t)$$9 is the mean trajectory of an agent with parameters $x_i(t)\in\mathbb{R}^n$0.

• Existence and uniqueness of this solution follow from standard controllability and observability conditions (A1–A4) and Lipschitz mass effect (Kizilkale et al., 2012).

For Linear-Quadratic-Gaussian (LQG) mean field social optimization with multiplicative noise, optimality is determined via two coupled algebraic Riccati equations (SARE and an indefinite ARE) yielding control laws

$x_i(t)\in\mathbb{R}^n$1

where $x_i(t)\in\mathbb{R}^n$2 and $x_i(t)\in\mathbb{R}^n$3 are feedback and feedforward gains, and $x_i(t)\in\mathbb{R}^n$4 evolves according to a deterministic ODE (Xu et al., 2024).

3. Adaptive Estimation and Online Implementation

Offline computation of the mean field equilibrium requires knowledge of individual and population parameters, which is typically unavailable. MF-SAC employs online adaptive estimation:

• Recursive Weighted Least Squares (RWLS): Each agent estimates its own dynamics $x_i(t)\in\mathbb{R}^n$5 via continuous-time RWLS:

$x_i(t)\in\mathbb{R}^n$6

with gain $x_i(t)\in\mathbb{R}^n$7 slowly increasing to ensure persistent excitation, and projection onto the controllable/observable set (Kizilkale et al., 2012).

• Population Parameter Estimation via MLE: By observing a random subset $x_i(t)\in\mathbb{R}^n$8 of the population, each agent estimates $x_i(t)\in\mathbb{R}^n$9 by maximizing the log-likelihood of the observed parameters:

$u_i(t)\in\mathbb{R}^m$0

and then solves for $u_i(t)\in\mathbb{R}^m$1.

• Certainty Equivalence Control: With current estimates $u_i(t)\in\mathbb{R}^m$2, each agent computes its control law using the Riccati equation and ODEs as in the offline equilibrium, augmented with a vanishing dither term for rich excitation:

$u_i(t)\in\mathbb{R}^m$3

where $u_i(t)\in\mathbb{R}^m$4 (Kizilkale et al., 2012).

In data-driven LQG settings with multiplicative noise, integral reinforcement learning (IRL) replaces model-based ARE solutions with off-policy least-squares regression on time-integrated trajectory data, updating gains $u_i(t)\in\mathbb{R}^m$5 using a single agent’s observations under exploratory excitation (Xu et al., 2024).

4. Mean Field Signal and Population Averaging

Computation of the mass effect $u_i(t)\in\mathbb{R}^m$6 or mean field signal $u_i(t)\in\mathbb{R}^m$7 is fundamental:

• In the adaptive control context, estimation of the mass effect relies on the agent’s inferred population distribution and its own type estimate, used to propagate the coupled mean field ODEs (Kizilkale et al., 2012).
• In IRL-based LQG settings, two main approaches are provided (Xu et al., 2024):
  • Monte Carlo averaging: Simulate multiple trajectories under the learned controller, set $u_i(t)\in\mathbb{R}^m$8; by the law of large numbers, this converges to $u_i(t)\in\mathbb{R}^m$9 as $A_i,B_i$0.
  • Online system identification: Using estimates for $A_i,B_i$1, form $A_i,B_i$2 if invertibility and data richness conditions are satisfied.

Both approaches ensure that the distributed implementation is feasible without global observation or synchronization, relying only on local data and minimal sampling.

5. Main Theoretical Results

Under the prescribed technical assumptions—controllability/observability, Lipschitz mass effect, identifiability of population parameter distribution, persistent excitation via dither, and bounded noise densities—MF-SAC guarantees the following properties (Kizilkale et al., 2012):

• Strong consistency of self parameter estimates: $A_i,B_i$3 almost surely as $A_i,B_i$4 for all $A_i,B_i$5.
• Strong consistency of MLE population parameter estimates: $A_i,B_i$6 almost surely as $A_i,B_i$7.
• Long-run average $A_i,B_i$8 stability: Uniformly bounded squared state trajectories almost surely.
• $A_i,B_i$9-Nash equilibrium: For every $D$0, there exists $D$1 such that for all $D$2, adaptive control achieves within $D$3 of the optimal non-adaptive Nash value.
• Cost-equivalence in the large-$D$4 limit: As $D$5, the long-run average cost under adaptive MF approaches the cost under full-information MF control.
• Best-response consistency: In the infinite population limit, each agent’s adaptive best-response cost matches the complete-information best response.

These results are established via uniform convergence of coefficient and parameter estimates, Lyapunov small-gain arguments for stability, and continuity of the equilibrium cost mapping in the large-population limit.

6. Extensions: Model-Free and Data-Driven Approaches

The MF-SAC framework has been extended to model-free settings using IRL and empirical system identification (Xu et al., 2024):

• Integral RL constructs off-policy equations whose expectation coincides with the steady-state Riccati and mean field equations, estimating Riccati solutions and gains $D$6 from time integrals of a single controlled trajectory, excited with probing noise.
• The population mean is estimated via either Monte Carlo over sample paths or direct identification of system matrices from time series.
• The algorithm is scalable and computationally efficient, requiring only "a unified state and input samples collected from a single agent" for all phases.
• Numerical experiments confirm convergence of the RL iterates and close agreement of simulated mean field signals with ground truth.

This model-free methodology allows MF-SAC to be applied when no parametric model of the agent dynamics is available, expanding applicability to broader classes of stochastic population systems.

7. Practical Workflow and Algorithmic Summary

The implementation of MF-SAC typically follows the steps:

1. Data Collection: Agents collect local trajectories and, optionally, sparse population observations.
2. Local Dynamics Identification: Each agent runs online RWLS (or IRL in LQG with multiplicative noise) to estimate its dynamics.
3. Population Parameter Estimation: Agents use sampled observations to fit $D$7 via MLE.
4. Solution of Control Equations: Agents solve Riccati and offset equations (or empirical RL regressions) with estimated parameters.
5. Mean Field Signal Computation: Apply Monte Carlo or system identification to compute the mean field trajectory.
6. Adaptive Control Law Application: Implement time-varying certainty equivalence law, periodically re-solving as estimates update.

A summary of components is provided below:

Component	Method	Output
Local dynamics estimation	RWLS (classic), IRL (LQG)	$D$8 or $D$9
Population statistics	MLE on sampled agents	$w_i$0 or empirical moments
Mean field computation	Riccati+ODE with estimates, MC average, or sys-id	$w_i$1 or $w_i$2
Control synthesis	Certainty-equivalence adaptive law	$w_i$3 (per agent)

This layered workflow ensures rigorous adaptivity and convergence to socially optimal or Nash equilibrium solutions in stochastic agent populations, both with known models and purely from data (Kizilkale et al., 2012, Xu et al., 2024).