Population-Aware Linear Function Approximation

• PA-LFA is a framework that unifies the agent’s Q-function and the population distribution into a single parameterization for mean field games.
• It employs a single-loop semi-gradient descent algorithm to update both policy and population parameters asynchronously, ensuring efficient learning.
• Empirical studies on benchmark MFGs show that PA-LFA achieves faster convergence and lower steady-state error compared to traditional fixed-point iteration methods.

Population-Aware Linear Function Approximation (PA-LFA) is a framework for the efficient and stable learning of mean field games (MFGs) on continuous state–action spaces, unifying the agent’s action-value function and the population distribution into a single parameterization. This approach enables simultaneous, fully asynchronous updates using single-loop stochastic semi-gradient descent, avoiding the inefficiency and instability of traditional fixed-point iteration (FPI) methods. PA-LFA provides provable finite-time convergence guarantees in linear MFGs and controlled approximation error in non-linear regimes, validated by empirical results on benchmark MFGs (Zhang et al., 2024).

1. Unified Parameterization and Feature Representation

PA-LFA represents both the agent’s state–action value function (Q-function) and the population distribution using a joint parameter

$$\xi = (\theta; \eta) \in \mathbb{R}^{d_1} \times \mathbb{R}^{d_2},$$

where $\theta \in \mathbb{R}^{d_1}$ parameterizes $Q(s,a) \approx \theta^\top \phi(s,a)$, and $\eta \in \mathbb{R}^{d_2}$ parameterizes the population density $M(s) \approx \eta^\top \psi(s)$. The functions $\phi: S \times A \to \mathbb{R}^{d_1}$ and $\psi: S \to \mathbb{R}^{d_2}$ are known feature maps, with $\psi$ acting as a “measure basis” for density functions over the state space. In discrete MFGs, one may use one-hot encodings, whereby $\theta$ and $\eta$ correspond directly to table-valued $\theta \in \mathbb{R}^{d_1}$0 and $\theta \in \mathbb{R}^{d_1}$1.

2. Linear Function Approximation in Mean Field Games

Under the assumption of a linear-MFG structure, the equilibrium satisfies

$\theta \in \mathbb{R}^{d_1}$2

At each iteration $\theta \in \mathbb{R}^{d_1}$3, PA-LFA approximates

$\theta \in \mathbb{R}^{d_1}$4

Feature choices for $\theta \in \mathbb{R}^{d_1}$5 and $\theta \in \mathbb{R}^{d_1}$6 include radial basis functions, tile codings, and one-hot bases, with continuous-feature bases shown empirically to outperform naive discretization when dimensionality is low.

3. Single-Loop Semi-Gradient Descent Algorithm

PA-LFA jointly updates $\theta \in \mathbb{R}^{d_1}$7 and $\theta \in \mathbb{R}^{d_1}$8 in a single loop using a semi-gradient stochastic descent method. Given the observation $\theta \in \mathbb{R}^{d_1}$9 under the policy $Q(s,a) \approx \theta^\top \phi(s,a)$0 and population $Q(s,a) \approx \theta^\top \phi(s,a)$1, the update proceeds as follows:

• Compute:

$Q(s,a) \approx \theta^\top \phi(s,a)$2

$Q(s,a) \approx \theta^\top \phi(s,a)$3

$Q(s,a) \approx \theta^\top \phi(s,a)$4

• The semi-gradient is

$Q(s,a) \approx \theta^\top \phi(s,a)$5

• The projected SGD update is

$Q(s,a) \approx \theta^\top \phi(s,a)$6

where $Q(s,a) \approx \theta^\top \phi(s,a)$7 (Euclidean ball) and $Q(s,a) \approx \theta^\top \phi(s,a)$8 is projected onto the simplex $Q(s,a) \approx \theta^\top \phi(s,a)$9.

Algorithmic pseudocode:

$M(s) \approx \eta^\top \psi(s)$6

4. Finite-Time Convergence Properties

For linear MFGs under the standard contractivity and Lipschitz-ergodicity assumptions, PA-LFA admits the following finite-time convergence guarantee [Theorem 5.1]:

$\eta \in \mathbb{R}^{d_2}$0

where $\eta \in \mathbb{R}^{d_2}$1 is the contraction constant. With constant step-size $\eta \in \mathbb{R}^{d_2}$2,

$\eta \in \mathbb{R}^{d_2}$3

Setting $\eta \in \mathbb{R}^{d_2}$4 achieves

$\eta \in \mathbb{R}^{d_2}$5

With linearly decaying steps $\eta \in \mathbb{R}^{d_2}$6, log factors can be further reduced.

5. Approximation Error in Non-Linear Mean Field Games

When the MDP and population operators are not exactly linear in the chosen features, PA-LFA converges to a projected mean-field equilibrium parameterized by $\eta \in \mathbb{R}^{d_2}$7. The resulting errors are bounded as follows (Theorem 6.1):

$\eta \in \mathbb{R}^{d_2}$8

are controlled by feature approximation errors $\eta \in \mathbb{R}^{d_2}$9 and $M(s) \approx \eta^\top \psi(s)$0 and the contraction/Lipschitz constants of the problem:

$M(s) \approx \eta^\top \psi(s)$1

In the linear MFG case, $M(s) \approx \eta^\top \psi(s)$2, so PA-LFA converges exactly to the true mean field equilibrium.

6. Empirical Validation and Benchmark Comparisons

Experiments evaluated PA-LFA on benchmark continuous-state-action MFGs:

• Speed Control on a Ring: Discretization of $M(s) \approx \eta^\top \psi(s)$3 with 50 bins, Lighthill–Whitham–Richards-based reward, and a spatial bonus. Compared to FPI with entropy, FPI with fictitious play, and FPI with online mirror descent, PA-LFA converged 2–3× faster in MSE, exhibited no oscillations, and achieved lower steady-state error at matched basis sizes.
• Network Routing on Sioux Falls: Edge-based states/actions on a 74-edge graph with congestion and terminal rewards. Near-greedy policies yielded per-sample speed approximately 2× faster than FPI+FP, with similar stability but without the forward-backward looping of FPI.

In these benchmarks, the use of continuous-feature bases ($M(s) \approx \eta^\top \psi(s)$4, $M(s) \approx \eta^\top \psi(s)$5) led to superior results over naive grid discretization at matching feature dimension, and the single-loop update of PA-LFA eliminated oscillations commonly seen in traditional two-timescale or fixed-point methods.

7. Context and Impact

By treating the policy and population as a unified parameter vector, PA-LFA enables simultaneous, fully asynchronous updates for MFG learning, in contrast to the sequential, coupled updates of FPI-based schemes. This structural innovation addresses both inefficiency and instability—specifically oscillations—inherent to forward-backward MFG solvers. Empirical evidence substantiates accelerated convergence and improved error profiles on continuous-state-action MFGs, confirming the theoretical guarantees and demonstrating the practical utility of population-aware parameterization and feature-based approximation in large-scale multi-agent environments (Zhang et al., 2024).