Mean-Field Q-Learning

• Mean-field Q-learning is a reinforcement learning paradigm that replaces full joint state-action spaces with a population-level summary to reduce complexity.
• It approximates interactions using aggregated measures such as average actions or state distributions, enabling scalable learning in large populations.
• Algorithmic variants include two-timescale updating, decentralized approaches, and robust methods that handle partial observability and heterogeneity.

Searching arXiv for recent and foundational papers on mean-field Q-learning to ground the article in cited work. Mean-field Q-learning denotes a family of reinforcement-learning methods that replace explicit dependence on the joint state or joint action of many agents by dependence on a population-level summary, typically a distribution or an average action, thereby reducing the dimensionality of value estimation while preserving the dominant interaction structure. In the literature covered here, the term spans several related constructions: mean-action approximations for large-population stochastic games and partially observable multi-agent systems (Guo et al., 2019, Subramanian et al., 2020, Lin et al., 2024), lifted Q-functions on probability-measure spaces for mean-field control with a rigorously valid dynamic programming principle (Gu et al., 2019), two-timescale Q-learning schemes that jointly learn a value function and a mean field for mean field games (MFGs) or mean field control (MFC) (Angiuli et al., 2021, Angiuli et al., 2023), regularized and robust variants that stabilize best-response mappings or account for Wasserstein ambiguity (Anahtarci et al., 2020, Laurière et al., 18 Jun 2026), and extensions to decentralized heterogeneous agents and coalition games (Subramanian et al., 2021, Shao et al., 2024). Across these settings, the central idea is that the value function is no longer parameterized by a full joint configuration, but by a representative state together with a mean-field object whose dimension does not grow exponentially with population size.

1. Conceptual scope and problem classes

Mean-field Q-learning arises in problems where the direct multi-agent Q-function is intractable because the state-action description scales exponentially with the number of agents. In large-population stochastic games, a representative player can often treat the aggregate behavior of others through a mean field, such as a state distribution, a joint state-action distribution, or an average action (Guo et al., 2019, Subramanian et al., 2020). In cooperative control, the analogous construction appears in mean-field control, where the planner optimizes over the evolution of a population distribution rather than over isolated trajectories (Gu et al., 2019, Angiuli et al., 2021).

Several distinct but related regimes appear in the cited work. In general mean-field games, the representative agent solves an MDP parameterized by a population flow, and equilibrium is characterized by a fixed point between best response and the induced distribution (Guo et al., 2019). In mean-field control with learning, the control problem is lifted to the space of probability measures, and the correct Q-like object must be defined on the distribution state and a local policy rather than on an individual state-action pair (Gu et al., 2019). In partially observable or decentralized many-agent systems, the mean field is used as an approximation device that summarizes local interactions, often through an empirical mean action estimated from neighbors (Subramanian et al., 2020, Subramanian et al., 2021, Lin et al., 2024).

This breadth can create a misconception that “mean-field Q-learning” refers to a single algorithm. The cited literature instead shows a spectrum of formulations linked by a common dimensionality-reduction principle. One branch uses a mean action in a large but finite population (Subramanian et al., 2020, Lin et al., 2024). Another uses a full probability distribution as the state of the lifted control problem (Gu et al., 2019, Laurière et al., 18 Jun 2026). A plausible implication is that the phrase is best understood as an umbrella term for Q-learning schemes whose Bellman updates operate on a reduced mean-field state rather than the full joint system.

2. Core mathematical reductions

A standard mean-action approximation replaces the dependence of agent $j$'s value on the full joint action $\boldsymbol{a}_t$ by dependence on its own action and an average neighbor action. In partially observable mean-field reinforcement learning, this appears as

$$Q^{j}(s_t,\boldsymbol{a}_t) \approx Q^j(s_t, a^j_t, \overline{a}_t^j),$$

with

$$\overline{a}^j_t = \frac{1}{N^j} \sum_{k\neq j} a^k_t$$

for a neighborhood of size $N^j$ (Subramanian et al., 2020). In MFC-EQ, the local empirical action distribution is written

$$\bar{a}_t^j = \frac{1}{|\mathcal{N}^j|}\sum_{k\in\mathcal{N}^j} a_t^k,$$

and the scalarized multi-objective joint-Q is approximated by

$$\boldsymbol{\omega}^{\top}\mathbf{Q}(s_t^j,\mathbf{a}_t) \approx \boldsymbol{\omega}^{\top}\mathbf{Q}(s_t^j,a_t^j,\bar{a}_t^j)$$

(Lin et al., 2024). This yields a Q-input of fixed dimension independent of the number of agents.

A different reduction appears in the theory of mean-field control with learning, where the classical domain $(s,a)$ is insufficient because dynamics and rewards depend on the population distribution. The correct Q-function is defined on $(\mu,h)$, where $\mu\in\mathcal{P}(\mathcal{S})$ is a distribution and $\boldsymbol{a}_t$0 is a local policy. The resulting IQ function satisfies

$\boldsymbol{a}_t$1

(Gu et al., 2019). The paper argues that using a classical $\boldsymbol{a}_t$2 in this setting is mis-specified and leads to failure of the dynamic programming principle.

In general mean-field games, the mean field can be richer than a state distribution. The GMFG framework uses the joint state-action distribution

$\boldsymbol{a}_t$3

with policy and transition dynamics depending on $\boldsymbol{a}_t$4 rather than on $\boldsymbol{a}_t$5 alone (Guo et al., 2019). This suggests that the “right” mean field depends on the interaction structure of the problem: average action for local weak coupling, population state distribution for control on measures, and joint state-action laws when rewards and dynamics depend on both.

3. Bellman operators, dynamic programming, and equilibrium structure

The Bellman structure of mean-field Q-learning differs across the main settings, but in each case the mean field enters as part of the Bellman argument.

For fixed mean field $\boldsymbol{a}_t$6, the discounted Bellman operator in the finite-state MFG setting takes the standard form

$\boldsymbol{a}_t$7

(Albergaria et al., 2023). The nontrivial part is that $\boldsymbol{a}_t$8 must itself satisfy a consistency condition induced by the policy derived from $\boldsymbol{a}_t$9. Two-timescale algorithms exploit this by treating $$Q^{j}(s_t,\boldsymbol{a}_t) \approx Q^j(s_t, a^j_t, \overline{a}_t^j),$$0 and $$Q^{j}(s_t,\boldsymbol{a}_t) \approx Q^j(s_t, a^j_t, \overline{a}_t^j),$$1 as coupled variables with different learning rates (Angiuli et al., 2021, Albergaria et al., 2023).

In the GMFG framework, the mean-field game equilibrium is encoded by two maps: a best-response map $$Q^{j}(s_t,\boldsymbol{a}_t) \approx Q^j(s_t, a^j_t, \overline{a}_t^j),$$2 from a mean-field flow to an optimal policy, and a population update map $$Q^{j}(s_t,\boldsymbol{a}_t) \approx Q^j(s_t, a^j_t, \overline{a}_t^j),$$3 from policy and mean field to the next mean field. The equilibrium is a fixed point of

$$Q^{j}(s_t,\boldsymbol{a}_t) \approx Q^j(s_t, a^j_t, \overline{a}_t^j),$$4

(Guo et al., 2019). The resulting Nash equilibrium exists and is unique under Lipschitz assumptions summarized by the contraction condition

$$Q^{j}(s_t,\boldsymbol{a}_t) \approx Q^j(s_t, a^j_t, \overline{a}_t^j),$$5

(Guo et al., 2019).

For mean-field control with learning, the contribution of (Gu et al., 2019) is to show that the dynamic programming principle is valid only when the Q-like object is lifted to the probability-distribution space. The Bellman operator acts on $$Q^{j}(s_t,\boldsymbol{a}_t) \approx Q^j(s_t, a^j_t, \overline{a}_t^j),$$6, and the optimal policy is characterized by

$$Q^{j}(s_t,\boldsymbol{a}_t) \approx Q^j(s_t, a^j_t, \overline{a}_t^j),$$7

(Gu et al., 2019). This resolves a conceptual controversy in the area: whether one can simply import ordinary Q-learning into mean-field control. The paper’s answer is negative for the classical state-action formulation.

Robust and regularized variants further modify the Bellman operator. In regularized mean-field games, the reward is augmented by a strongly convex penalty $$Q^{j}(s_t,\boldsymbol{a}_t) \approx Q^j(s_t, a^j_t, \overline{a}_t^j),$$8, yielding a regularized Bellman operator and a unique optimal randomized action via strong concavity in $$Q^{j}(s_t,\boldsymbol{a}_t) \approx Q^j(s_t, a^j_t, \overline{a}_t^j),$$9 (Anahtarci et al., 2020). In robust mean-field control under common-noise uncertainty, the Bellman operator becomes a Bellman–Isaacs operator,

$$\overline{a}^j_t = \frac{1}{N^j} \sum_{k\neq j} a^k_t$$0

with ambiguity over the common-noise law inside a Wasserstein ball (Laurière et al., 18 Jun 2026).

4. Algorithmic families

The literature supports several algorithmic families under the common label.

A first family is alternating best-response learning in mean-field games. GMF-Q fixes a mean field $$\overline{a}^j_t = \frac{1}{N^j} \sum_{k\neq j} a^k_t$$1, runs ordinary Q-learning to approximate $$\overline{a}^j_t = \frac{1}{N^j} \sum_{k\neq j} a^k_t$$2, extracts a Boltzmann policy

$$\overline{a}^j_t = \frac{1}{N^j} \sum_{k\neq j} a^k_t$$3

then updates the population distribution and repeats (Guo et al., 2019). The use of Boltzmann rather than argmax is central: the paper shows that the naïve combination of Q-learning with discrete argmax is unstable because the best-response map becomes discontinuous (Guo et al., 2019).

A second family is two-timescale stochastic approximation. In the unified approach of (Angiuli et al., 2021), and its convergence analysis in (Albergaria et al., 2023), the algorithm simultaneously updates a Q-function and a mean-field estimate: $$\overline{a}^j_t = \frac{1}{N^j} \sum_{k\neq j} a^k_t$$4

$$\overline{a}^j_t = \frac{1}{N^j} \sum_{k\neq j} a^k_t$$5

Depending on the learning-rate ratio, the same scheme converges either to an MFG equilibrium or to an MFC solution (Angiuli et al., 2021). In the MFG regime, $$\overline{a}^j_t = \frac{1}{N^j} \sum_{k\neq j} a^k_t$$6, so the Q-update is fast; in the MFC regime, $$\overline{a}^j_t = \frac{1}{N^j} \sum_{k\neq j} a^k_t$$7, so the mean-field update is fast (Angiuli et al., 2021). The later convergence paper clarifies that MFC requires several mean-field distributions, one for each state-action pair, and therefore needs a separate algorithmic treatment (Albergaria et al., 2023).

A third family is local mean-action Q-learning for large partially observed populations. In partially observable mean-field reinforcement learning, the agent replaces the exact mean action by a sampled estimate $$\overline{a}^j_t = \frac{1}{N^j} \sum_{k\neq j} a^k_t$$8 drawn from a Bayesian belief over the mean field, giving updates of the form

$$\overline{a}^j_t = \frac{1}{N^j} \sum_{k\neq j} a^k_t$$9

(Subramanian et al., 2020). In the fixed-observation-radius setting, the belief is Dirichlet; in the probabilistic distance-observability setting, a Gamma variable is added to model visibility uncertainty (Subramanian et al., 2020).

A fourth family is decentralized mean-field Q-learning with learned mean-field models. In decentralized mean field games, each heterogeneous agent learns its own Q-function

$N^j$0

and its own local mean-field estimate

$N^j$1

where $N^j$2 is an opponent/mean-field modeling network (Subramanian et al., 2021). The paper emphasizes that this breaks the “chicken-and-egg” dependence of earlier empirical mean-field Q-learning methods that used lagged global mean fields (Subramanian et al., 2021).

A fifth family is task-specific integration with multi-objective or decentralized planning components. MFC-EQ combines a mean-action Q-approximation with envelope Q-learning to handle bi-objective decentralized formation control. The algorithm uses replay, target networks, Boltzmann exploration, and a homotopy loss

$N^j$3

to learn a preference-conditioned vector Q-function (Lin et al., 2024).

A sixth family is quantized or lifted-state Q-learning on probability-simplex spaces. In regularized mean-field games, fitted Q-iteration is run on a lifted action space $N^j$4 with regularized reward $N^j$5 (Anahtarci et al., 2020). In robust mean-field control with common noise, Q-learning is performed on a finite grid over $N^j$6, where $N^j$7, with projection and quantization of both lifted states and lifted policies (Laurière et al., 18 Jun 2026).

5. Regularization, smoothing, and robustness

Regularization is a recurring device for making mean-field Q-learning well-posed. In regularized mean-field games, a differentiable, $N^j$8-strongly convex function $N^j$9 is subtracted from the stage reward on the relaxed action space $$\bar{a}_t^j = \frac{1}{|\mathcal{N}^j|}\sum_{k\in\mathcal{N}^j} a_t^k,$$0, producing

$$\bar{a}_t^j = \frac{1}{|\mathcal{N}^j|}\sum_{k\in\mathcal{N}^j} a_t^k,$$1

(Anahtarci et al., 2020). The resulting Q-function is strongly concave in $$\bar{a}_t^j = \frac{1}{|\mathcal{N}^j|}\sum_{k\in\mathcal{N}^j} a_t^k,$$2, implying a unique maximizer and a Lipschitz policy map $$\bar{a}_t^j = \frac{1}{|\mathcal{N}^j|}\sum_{k\in\mathcal{N}^j} a_t^k,$$3 via Fenchel duality (Anahtarci et al., 2020). This lets the authors prove contraction of the mean-field equilibrium operator without imposing strong concavity directly on the physical reward or transition kernels (Anahtarci et al., 2020).

Smoothing by softmin or Boltzmann policy plays a similar role in two-timescale schemes. In (Albergaria et al., 2023), the policy map uses

$$\bar{a}_t^j = \frac{1}{|\mathcal{N}^j|}\sum_{k\in\mathcal{N}^j} a_t^k,$$4

to make the mapping from Q-values to policies Lipschitz, which is crucial for ODE-based stochastic approximation analysis. The paper then quantifies the bias induced by smoothing: the error between the smoothed fixed point and the hard equilibrium decays like $$\bar{a}_t^j = \frac{1}{|\mathcal{N}^j|}\sum_{k\in\mathcal{N}^j} a_t^k,$$5, where $$\bar{a}_t^j = \frac{1}{|\mathcal{N}^j|}\sum_{k\in\mathcal{N}^j} a_t^k,$$6 is an action gap (Albergaria et al., 2023).

Robustness enters in a different way in (Laurière et al., 18 Jun 2026). There, the planner does not trust the reference common-noise law and replaces the Bellman expectation by a worst-case expectation over a Wasserstein ball. The robust term is dualized: $$\bar{a}_t^j = \frac{1}{|\mathcal{N}^j|}\sum_{k\in\mathcal{N}^j} a_t^k,$$7 (Laurière et al., 18 Jun 2026). This dual reformulation converts an infinite-dimensional ambiguity set into a scalar optimization over $$\bar{a}_t^j = \frac{1}{|\mathcal{N}^j|}\sum_{k\in\mathcal{N}^j} a_t^k,$$8, making robust lifted-space Q-learning tractable.

A plausible synthesis is that regularization and smoothing control policy sensitivity, while robustness controls model sensitivity. Both become more important in mean-field settings because small changes in policy or transition laws feed back through the population distribution.

6. Partial observability, decentralization, and heterogeneous agents

A common misconception is that mean-field Q-learning requires access to an exact global population statistic. Several cited works directly challenge that assumption.

Partially observable mean-field reinforcement learning maintains a belief distribution over the mean field instead of an exact point estimate. In the fixed-radius setting, visible actions update a Dirichlet posterior,

$$\bar{a}_t^j = \frac{1}{|\mathcal{N}^j|}\sum_{k\in\mathcal{N}^j} a_t^k,$$9

from which $$\boldsymbol{\omega}^{\top}\mathbf{Q}(s_t^j,\mathbf{a}_t) \approx \boldsymbol{\omega}^{\top}\mathbf{Q}(s_t^j,a_t^j,\bar{a}_t^j)$$0 is sampled and used in the Q-update (Subramanian et al., 2020). The paper proves that the resulting Q-function stays close to the Nash Q-value under a common set of assumptions, with explicit sampling-error dependence (Subramanian et al., 2020).

Decentralized mean field games go further by dropping the interchangeability assumption. Each heterogeneous agent $$\boldsymbol{\omega}^{\top}\mathbf{Q}(s_t^j,\mathbf{a}_t) \approx \boldsymbol{\omega}^{\top}\mathbf{Q}(s_t^j,a_t^j,\bar{a}_t^j)$$1 has its own action space $$\boldsymbol{\omega}^{\top}\mathbf{Q}(s_t^j,\mathbf{a}_t) \approx \boldsymbol{\omega}^{\top}\mathbf{Q}(s_t^j,a_t^j,\bar{a}_t^j)$$2, reward $$\boldsymbol{\omega}^{\top}\mathbf{Q}(s_t^j,\mathbf{a}_t) \approx \boldsymbol{\omega}^{\top}\mathbf{Q}(s_t^j,a_t^j,\bar{a}_t^j)$$3, policy $$\boldsymbol{\omega}^{\top}\mathbf{Q}(s_t^j,\mathbf{a}_t) \approx \boldsymbol{\omega}^{\top}\mathbf{Q}(s_t^j,a_t^j,\bar{a}_t^j)$$4, and local mean-field estimate $$\boldsymbol{\omega}^{\top}\mathbf{Q}(s_t^j,\mathbf{a}_t) \approx \boldsymbol{\omega}^{\top}\mathbf{Q}(s_t^j,a_t^j,\bar{a}_t^j)$$5 (Subramanian et al., 2021). The solution concept is a decentralized mean-field equilibrium, where $$\boldsymbol{\omega}^{\top}\mathbf{Q}(s_t^j,\mathbf{a}_t) \approx \boldsymbol{\omega}^{\top}\mathbf{Q}(s_t^j,a_t^j,\bar{a}_t^j)$$6 satisfies both best response and consistency for that agent’s own mean-field estimate (Subramanian et al., 2021). This contrasts with classical MFG symmetry and shows that mean-field Q-learning can be adapted to fully decentralized, heterogeneous, even competitive settings (Subramanian et al., 2021).

MFC-EQ provides another decentralized example under partial observation and limited communication. Each agent sees only a local field of view, but agents exchange relative positions so that each can compute the global formation deviation in $$\boldsymbol{\omega}^{\top}\mathbf{Q}(s_t^j,\mathbf{a}_t) \approx \boldsymbol{\omega}^{\top}\mathbf{Q}(s_t^j,a_t^j,\bar{a}_t^j)$$7 time using only relative coordinates (Lin et al., 2024). The mean field is still an average local action distribution, but the state representation mixes local perception and lightweight communication (Lin et al., 2024).

In cooperative games with a major global agent and many local agents, mean-field subsampling offers a further relaxation: the central agent observes only $$\boldsymbol{\omega}^{\top}\mathbf{Q}(s_t^j,\mathbf{a}_t) \approx \boldsymbol{\omega}^{\top}\mathbf{Q}(s_t^j,a_t^j,\bar{a}_t^j)$$8 local states per time step and reacts to the empirical distribution of that subsample (Anand et al., 4 Mar 2026). This suggests that observability constraints can be absorbed into the mean-field definition itself, rather than treated as a nuisance external to the RL formulation.

7. Empirical evidence and representative application areas

The empirical record in the supplied papers is broad rather than uniform, so direct cross-paper benchmarking is not possible. Still, recurring patterns emerge.

In repeated ad auctions, GMF-Q is reported as efficient and robust in terms of convergence and learning accuracy, and it outperforms independent learners and MF-Q in convergence, stability, and learning ability (Guo et al., 2019). In the finite-$$\boldsymbol{\omega}^{\top}\mathbf{Q}(s_t^j,\mathbf{a}_t) \approx \boldsymbol{\omega}^{\top}\mathbf{Q}(s_t^j,a_t^j,\bar{a}_t^j)$$9 ad auction experiments, the closeness-to-Nash metric satisfies approximately $(s,a)$0 for IL, $(s,a)$1 for MF-Q, and $(s,a)$2 for GMF-Q when $(s,a)$3 and $(s,a)$4 (Guo et al., 2019).

In economics and finance, the two-timescale framework is illustrated on accumulated consumption with HARA utility and a trader’s optimal liquidation problem, where changing only the relative learning rates allows the same algorithm to recover either the MFG Nash solution or the MFC social optimum (Angiuli et al., 2021). The later convergence paper confirms this interpretation theoretically in finite-state, infinite-horizon tabular settings (Albergaria et al., 2023).

In decentralized formation control, MFC-EQ is evaluated on $(s,a)$5, $(s,a)$6, and $(s,a)$7 four-neighbor grids with obstacle densities 10%, 15%, and 20%, field of view $(s,a)$8, and training over 500k episodes with batch size 192 (Lin et al., 2024). The method maintains high success rates, often outperforms SWARM-MAPF in $(s,a)$9, and handles dynamic formation changes that centralized baselines cannot easily address (Lin et al., 2024). The paper also reports that for each evaluation preference $(\mu,h)$0, the lowest $(\mu,h)$1 is obtained by running the learned policy with $(\mu,h)$2, supporting the preference-agnostic interpretation of envelope mean-field Q-learning (Lin et al., 2024).

In partially observable many-agent games in the MAgent framework, POMFQ variants consistently outperform MFQ, MFAC, independent Q-learning, and recurrent baselines across Multibattle, Battle-Gathering, and Predator–Prey, under both fixed-radius and probabilistic-distance visibility models (Subramanian et al., 2020). The paper’s theoretical result that POMFQ-FOR remains within $(\mu,h)$3 of the Nash Q-value in the limit provides a rare near-Nash guarantee under partial observability (Subramanian et al., 2020).

In mean-field type games with coalitions, exploitability is the central metric. Deep DDPG-MFTG reduces exploitability relative to baselines across five environments, including a four-room crowd-aversion problem with mean-field input dimension 200 and a cyber-security game where attacker exploitability improves by approximately 55% relative to a baseline (Shao et al., 2024). The tabular DNashQ-MFTG also converges to the Nash equilibrium Q-functions of the quantized game under standard assumptions (Shao et al., 2024).

Robust lifted-space Q-learning is illustrated on systemic risk and epidemic models. The asynchronous implementation tracks an idealized Bellman iteration closely, and the experiments show a clear robustness–performance tradeoff as the Wasserstein radius $(\mu,h)$4 varies under common-noise misspecification (Laurière et al., 18 Jun 2026).

8. Theory of convergence and representation

Convergence theory in mean-field Q-learning is fragmented across settings, but several lines are now well-developed.

For GMFGs, convergence follows from contraction of the mean-field fixed-point map under Lipschitz assumptions $(\mu,h)$5, combined with standard Q-learning convergence on each fixed mean field (Guo et al., 2019). The stabilizing role of Boltzmann best responses is central because argmax discontinuity can destroy the regularity required for fixed-point iteration (Guo et al., 2019).

For two-timescale MFG/MFC learning, convergence relies on Borkar-style stochastic approximation. The fast component tracks the equilibrium of the frozen slow variable, and the slow component tracks an ODE whose right-hand side substitutes in that fast equilibrium (Angiuli et al., 2021, Albergaria et al., 2023). The 2023 convergence paper proves almost sure convergence in the tabular, infinite-horizon case for both synchronous and asynchronous variants, with separate treatments for MFG and MFC because MFC requires several mean-field distributions $(\mu,h)$6 (Albergaria et al., 2023).

For mean-field control with learning, (Gu et al., 2019) establishes the dynamic programming principle on the lifted probability space. Its main theoretical message is not a sample-based convergence theorem but a structural one: the correct state of dynamic programming is the distribution $(\mu,h)$7, and therefore any valid Q-learning theory in MFC must operate in the lifted space (Gu et al., 2019).

For regularized mean-field games, the contraction constant of the equilibrium map depends explicitly on the strong convexity parameter $(\mu,h)$8 of the regularizer,

$(\mu,h)$9

and uniqueness follows when $\mu\in\mathcal{P}(\mathcal{S})$0 (Anahtarci et al., 2020). The fitted Q-iteration approximation error is then propagated through the equilibrium operator, yielding bounds on the learned mean field and policy (Anahtarci et al., 2020).

For robust mean-field control with common noise, the lifted-space Q-learning algorithm converges almost surely to the discretized robust Q-function, and the finite-time error decomposes into discretization error plus a stochastic approximation term of order

$\mu\in\mathcal{P}(\mathcal{S})$1

(Laurière et al., 18 Jun 2026). This is one of the few finite-time rates available for a robust mean-field control Q-learning scheme.

A separate but relevant strand concerns representation learning in neural Q-learning. The mean-field theory of overparameterized two-layer networks shows that Q-learning and TD globally minimize the projected Bellman error in the mean-field limit of the network-width distribution, with the parameter distribution evolving in Wasserstein space (Zhang et al., 2020). This paper is not about mean-field games; rather, the “mean field” is over neural network parameters. Its relevance here is conceptual: it shows that the phrase “mean-field Q-learning” can refer either to population-interaction reductions or to infinite-width neural-network limits. The two uses are distinct and should not be conflated (Zhang et al., 2020).

9. Limitations, controversies, and open directions

The most persistent controversy concerns what the correct Q-object should be. The classical single-agent state-action Q-function is insufficient for MFC with learning because it omits the distribution-state information needed for time consistency (Gu et al., 2019). By contrast, large-population MARL papers often retain a single-agent Q-function but augment it with a mean action or mean-field statistic (Subramanian et al., 2020, Lin et al., 2024). These are not contradictory results; they address different approximations. A plausible interpretation is that the mean-action formulation is an approximation for large finite games, whereas the IQ formulation is the exact object for lifted mean-field control.

Another limitation is the strength of regularity assumptions. GMF-Q requires contraction of the mean-field best-response composition (Guo et al., 2019). Two-timescale convergence requires Lipschitz population dynamics, sufficient exploration, and globally asymptotically stable limiting ODEs (Angiuli et al., 2021, Albergaria et al., 2023). Regularized MFG learning requires strong convexity of the regularizer and contraction of the induced equilibrium operator (Anahtarci et al., 2020). Robust lifted-space Q-learning additionally requires finite-space quantization and bounded common-noise ambiguity (Laurière et al., 18 Jun 2026). These assumptions are analytically useful but may restrict applicability.

Scalability remains a bifurcated story. Mean-action methods scale well in population size because the Q-input dimension is fixed (Lin et al., 2024, Subramanian et al., 2021). Lifted-space methods avoid exponential dependence on agent number but replace it with dependence on discretized probability-simplex dimensions (Anahtarci et al., 2020, Laurière et al., 18 Jun 2026, Shao et al., 2024). The deep DDPG-MFTG results suggest one path around quantization, but convergence theory for deep mean-field Q-learning in games remains open (Shao et al., 2024).

Several future directions are explicitly identified in the data. MFC-EQ suggests mixing networks, non-linear preference handling, attention-based architectures, and learned selective communication (Lin et al., 2024). The two-timescale MFG/MFC work points to continuous-space extensions and deep RL approximations as ongoing directions (Angiuli et al., 2021). The MFTG work highlights partial observation and continuous state-action spaces as unresolved extensions (Shao et al., 2024). Robust mean-field control under Wasserstein uncertainty suggests broader applications to large populations under shared shocks, but currently depends on quantization and tabular structure (Laurière et al., 18 Jun 2026).

Overall, the literature shows that mean-field Q-learning is no longer a narrow heuristic for many-agent scaling. It has become a technically diverse research area connecting stochastic games, control on measure spaces, robust RL, multi-objective learning, and decentralized planning, with a common structural aim: to preserve the essential feedback from a large population while avoiding the combinatorial burden of explicit joint-state Q-learning (Guo et al., 2019, Gu et al., 2019, Angiuli et al., 2021, Lin et al., 2024).