Mean-Field Stochastic LQR Controller

• Mean-Field Stochastic LQR Controller is a control strategy that extends classical LQR by incorporating both individual-state and population-averaged interactions under stochastic uncertainty.
• It employs a Lagrangian dual formulation to decouple mean and fluctuation dynamics, solving coupled Riccati equations for robust, scalable state-feedback synthesis.
• Applications in microgrid frequency control and large-scale power networks demonstrate its ability to reduce overshoots and manage risk through variance-based constraints.

A mean-field stochastic linear quadratic regulator (MF-SLQR) controller generalizes the classical LQR approach to systems with both individual-state and mean-field (population-averaged) interactions under stochastic uncertainty, and further constrains state fluctuations to address risk. Such controllers are crucial in high-dimensional multi-agent networks, power grids, and large-scale coupled stochastic systems, particularly when low-probability, high-impact events must be systematically attenuated rather than averaged away as in the risk-neutral setting.

1. Problem Formulation: Dynamics, Cost, and Risk Constraint

Consider $n$ exchangeable agents indexed by $i$ with individual state $x_t^i \in \mathbb{R}^{d_x}$ and control $u_t^i \in \mathbb{R}^{d_u}$. Each agent's dynamics incorporate both local and mean-field coupling: $$x_{t+1}^i = A\,x_t^i + B\,u_t^i + \bar{A}\,\bar{x}_t + \bar{B}\,\bar{u}_t + w_t^i,$$ where $\bar{x}_t = \frac{1}{n}\sum_{j=1}^n x_t^j$, $\bar{u}_t = \frac{1}{n}\sum_{j=1}^n u_t^j$, and $w_t^i$ is an i.i.d. zero-mean noise sequence. In the $n\to\infty$ mean-field limit, agent states decompose into fluctuation and mean components: $$\begin{aligned} &\tilde{x}^i_{t+1} = A\,\tilde{x}_t^i + B\,\tilde{u}_t^i + \tilde{w}_t^i, \ &\bar{x}_{t+1} = (A+\bar{A})\,\bar{x}_t + (B+\bar{B})\,\bar{u}_t + \bar{w}_t, \end{aligned}$$ where $\tilde{x}_t^i = x_t^i - \bar{x}_t$, $\tilde{u}_t^i = u_t^i - \bar{u}_t$.

The infinite-horizon average quadratic cost per agent (risk-neutral) is

$$J = \limsup_{T\to\infty} \frac{1}{T} \mathbb{E}\!\left[ \sum_{t=0}^{T-1} \left( (x_t^i)^\top Q x_t^i + (u_t^i)^\top R u_t^i + \bar{x}_t^\top \bar{Q} \bar{x}_t + \bar{u}_t^\top \bar{R} \bar{u}_t \right) \right].$$

To control rare but impactful fluctuations, a variance-type risk constraint is imposed. Define

$$d_t^i = \left( (x_t^i)^\top Q x_t^i - \mathbb{E}\left[(x_t^i)^\top Q x_t^i | h_t^i \right] \right)^2,$$

where $h_t^i$ denotes agent $i$'s history. The per-player time-averaged variance is

$$J_c = \frac{1}{n}\sum_{i=1}^n \limsup_{T\to\infty} \frac{1}{T} \mathbb{E}\left[ \sum_{t=0}^{T-1} d_t^i \right].$$

The MF-SLQR controller seeks to

$$\begin{aligned} &\min_{\{u_t^i\}}~~ J \ &\text{s.t.}~~ (1),~~ J_c \leq \Gamma~, \end{aligned}$$

with risk budget $\Gamma > 0$ (Roudneshin et al., 2023).

2. Lagrangian Dual Formulation and Decomposition

A Lagrange multiplier $\lambda \geq 0$ combines the nominal and risk costs: $\mathcal{L} = J + \lambda (J_c - \Gamma).$ Owing to orthogonal mean–fluctuation decomposition, $\mathcal{L}$ decouples into two independent infinite-horizon LQR objectives: $$\mathcal{L} = \underbrace{\limsup_{T \to \infty} \frac{1}{T} \mathbb{E} \left[ \sum_t \tilde{x}_t^{i\top} Q_\lambda \tilde{x}_t^i + \tilde{u}_t^{i\top} R_\lambda \tilde{u}_t^i \right]}_{\text{fluctuation-LQR}} + \underbrace{\limsup_{T \to \infty} \frac{1}{T} \mathbb{E} \left[ \sum_t \bar{x}_t^\top \mathcal{Q}_{\bar\lambda} \bar{x}_t + \bar{u}_t^\top \mathcal{R}_{\bar\lambda} \bar{u}_t \right]}_{\text{mean-LQR}},$$ where

$$Q_\lambda = Q + \frac{4\lambda}{n} Q M_2 Q, \quad \mathcal{Q}_{\bar\lambda} = Q + \bar{Q} + 4\lambda Q M_2 Q, \quad R_\lambda = R,~\mathcal{R}_{\bar\lambda} = R + \bar{R}$$

and $$M_2 = \limsup_{T\to\infty} \frac{1}{T} \mathbb{E} \sum_{t=0}^{T-1}\left(x_t^\top Q x_t - \mathbb{E}[x_t^\top Q x_t]\right)^2$$. The dependence on $\lambda$ drives risk sensitivity (Roudneshin et al., 2023).

3. Coupled Riccati Equations and Controller Synthesis

Both mean and fluctuation subsystems admit closed-form LQR solutions. The optimal value of $\lambda = \lambda^*$ is enforced via a primal–dual algorithm.

Riccati Equations

$$\begin{aligned} P &= Q_\lambda + A^\top P A - A^\top P B (R + B^\top P B)^{-1} B^\top P A, \ \mathcal{P} &= \mathcal{Q}_{\bar\lambda} + (A+\bar{A})^\top \mathcal{P} (A+\bar{A}) \ & \qquad - (A+\bar{A})^\top \mathcal{P} (B+\bar{B}) [\mathcal{R}_{\bar\lambda} + (B+\bar{B})^\top \mathcal{P} (B+\bar{B})]^{-1} (B+\bar{B})^\top \mathcal{P} (A+\bar{A}). \end{aligned}$$

These equations admit a unique positive semidefinite stabilizing solution under standard MF-LQR stabilizability/detectability criteria (Roudneshin et al., 2023).

Controller Law

The optimal state-feedback is affine in the agent's own state and in the population mean: $$u_t^{i,*} = -K x_t^i - (\bar{K} - K)\bar{x}_t + k_0,$$ with

$$K = (R + B^\top P B)^{-1} B^\top P A, \qquad \bar{K} = [\mathcal{R}_{\bar\lambda} + (B+\bar{B})^\top \mathcal{P} (B+\bar{B})]^{-1} (B+\bar{B})^\top \mathcal{P} (A+\bar{A}),$$

and $k_0$ arises if the noise is nonzero-mean in the dual formulation but is zero otherwise (Roudneshin et al., 2023). The feedback structure matches that of classical mean-field LQR, but the gain matrices internalize the variance penalty via the modified cost matrices.

4. Risk Constraint Enforcement and Computational Aspects

The optimal Lagrange multiplier $\lambda^*$ is computed via a primal–dual loop. At each iteration:

1. Update $Q_\lambda$, $\mathcal{Q}_{\bar\lambda}$;
2. Solve the two Riccati equations for $P$, $\mathcal{P}$;
3. Form $K$, $\bar{K}$, $k_0$;
4. Simulate closed-loop dynamics and evaluate $J_c(\lambda)$;
5. Update $\lambda$ by a projected subgradient step: $$\lambda_{k+1} = [\lambda_k + \eta (J_c(\lambda_k) - \Gamma)]_+.$$ Each iteration requires $O(d_x^3)$ operations (Riccati/Lyapunov equations) and is independent of the agent population $n$. The resulting gains do not depend on $n$, which ensures scalability (Roudneshin et al., 2023).

5. Structural Properties and Solution Comparison

• Independence from Number of Players: All Riccati equations and controller gains depend only on local and mean-field coupling ($A, B, \bar{A}, \bar{B}$) and cost weights, not on $n$. This makes the approach viable for large-scale networks (Roudneshin et al., 2023).
• Existence and Uniqueness: Unique stabilizing Riccati solutions and dual multipliers $\lambda^*\geq 0$ are guaranteed by standard LQR system-theoretic conditions and strong duality (Roudneshin et al., 2023).
• Risk Parameter Influence: As $\Gamma$ is decreased (tighter variance constraint), $\lambda^*$ increases, which modifies $Q_\lambda$, leading to more conservative $K, \bar{K}$ (higher-magnitude feedback gains). This reduces the amplitude of risky excursions (e.g., overshoots) at the cost of modestly increased average cost $J$ (Roudneshin et al., 2023).
• Comparison to Risk-Neutral MF-LQR: Setting $\lambda=0$ recovers the standard mean-field LQR controller, which ignores state variance (Roudneshin et al., 2023).

6. Practical Example and Performance Analysis

A microgrid frequency control scenario is used as a high-dimensional case paper. Each area (agent) maintains a $4$-dimensional state including local frequency, generation, tie-line flow, and area control error integral. System matrices $A, \bar{A}, B$ are drawn from standard load frequency control (LFC) parameters and quadratic costs $Q, R$ are assigned.

Table: Impact of Variance Constraint on Controller Behavior | $\Gamma$ (risk tolerance) | $\lambda^*$ (dual) | Gain Magnitude | Overshoot | Average Cost $J$ | |--------------------------|--------------------|---------------|-----------|------------------| | High | $\approx 0$ | Baseline | Large | Baseline | | Moderate | Moderate | Increased | Reduced | Slightly higher | | Low | Large | Largest | Minimal | Highest |

Reducing $\Gamma$ leads to increased $\lambda^*$ and higher feedback gain magnitude, yielding faster disturbance damping and less vulnerability to rare high-variance events (Roudneshin et al., 2023).

7. Generalizations and Methodological Significance

Risk-constrained MF-SLQR controllers extend classic LQR and mean-field LQR by systematically regulating not only expected cost but also rare-event risk, via variance-type constraints. The affine law and dual-based synthesis parallel but augment the classical mean–fluctuation separation and are compatible with standard Riccati-based implementation. Scalability and algorithmic simplicity are preserved, and the general methodology readily extends to other risk proxies and limit regimes (Roudneshin et al., 2023).

These contributions are directly relevant for modern applications requiring explicit robustness to stochastic volatility and population-level coupling, such as large-scale energy systems and networked autonomous agents.