Model-free policy gradient for discrete-time mean-field control

Abstract: We study model-free policy learning for discrete-time mean-field control (MFC) problems with finite state space and compact action space. In contrast to the extensive literature on value-based methods for MFC, policy-based approaches remain largely unexplored due to the intrinsic dependence of transition kernels and rewards on the evolving population state distribution, which prevents the direct use of likelihood-ratio estimators of policy gradients from classical single-agent reinforcement learning. We introduce a novel perturbation scheme on the state-distribution flow and prove that the gradient of the resulting perturbed value function converges to the true policy gradient as the perturbation magnitude vanishes. This construction yields a fully model-free estimator based solely on simulated trajectories and an auxiliary estimate of the sensitivity of the state distribution. Building on this framework, we develop MF-REINFORCE, a model-free policy gradient algorithm for MFC, and establish explicit quantitative bounds on its bias and mean-squared error. Numerical experiments on representative mean-field control tasks demonstrate the effectiveness of the proposed approach.

• The paper introduces MF-REINFORCE, a breakthrough model-free policy gradient method that estimates gradients without explicit environment derivatives.
• It decomposes the policy gradient into REINFORCE, measure derivative, and mean-field derivative terms for precise and scalable control.
• Numerical experiments validate its convergence in both simplified and complex scenarios, highlighting key bias-variance trade-offs.

Model-Free Policy Gradient Methods for Discrete-Time Mean-Field Control

Introduction

The paper "Model-free policy gradient for discrete-time mean-field control" (2601.11217) presents the first rigorous development of policy gradient algorithms for mean-field control (MFC) in discrete-time settings with finite state spaces and compact action spaces. Unlike the majority of existing literature, which predominantly uses value-based approaches, this work establishes a model-free paradigm allowing policy optimization solely via simulated trajectories and auxiliary estimation of population state distribution sensitivities.

Policy Gradient in Mean-Field Control

Mean-field control abstracts large-population stochastic control problems by replacing agent-level interactions with couplings through population state distributions. The intrinsic dependence of rewards and transitions on these evolving distributions impedes direct use of standard policy gradient estimators. The authors derive an exact trajectory-based formula for the policy gradient that includes three key terms:

• REINFORCE term: Analogous to classical policy gradient methods, involving likelihood-ratio weighting.
• Measure derivative (MD) term: Incorporates sensitivity of population rewards to the state distribution flow.
• Mean-field derivative (MFD) term: Accounts for dependencies in transition kernel and policy on the population distribution.

This decomposition shows that mean-field structure breaks the amenability of naive policy-gradient methods due to the need for derivatives with respect to the measure argument, which are not accessible model-free.

Logits Parametrization and Perturbation Estimation

To enable practical model-free estimation, the paper proposes a perturbation method leveraging a logits-based parameterization of state distributions:

• Logits mapping: Probability distributions are mapped to log-probability vectors, which can be perturbed in $\mathbb{R}^d$ and then mapped back via softmax.
• Perturbation scheme: Introduce stochastic Gaussian perturbations to logits at each time step, yielding perturbed population flows that enable the construction of an approximate policy gradient estimator independent of explicit environment or reward derivatives.

The authors rigorously prove that the gradient of the perturbed value function converges to the true policy gradient as the magnitude of the perturbation diminishes.

MF-REINFORCE Algorithm: Implementation and Theory

Based on their analytic foundation, the authors introduce MF-REINFORCE, the first model-free policy gradient algorithm for mean-field control in discrete time. The algorithm involves:

1. Sampling population trajectories under both nominal and perturbed logits flows.
2. Estimating the sensitivity of state distributions via additional trajectory rollouts and solving a triangular system for their gradients.
3. Accumulating policy gradient estimates by reweighting rewards along sampled trajectories with both classical and mean-field correction terms.

Explicit upper bounds on the estimator's bias and mean squared error (MSE) are proved, quantifying:

• The impact of perturbation scale, sample complexity, and time horizon on accuracy.
• A strong dependence ("curse of time") where the estimator's variance may grow exponentially with horizon in presence of nontrivial mean-field coupling.
• Regimes where bias and variance can be simultaneously controlled by hyperparameter tuning, and parallelization may be leveraged to offset computational costs.

Numerical Experiments

Two-State Two-Action System

MF-REINFORCE is benchmarked on a simple two-state two-action setup where optimal policies are analytically tractable. Results show that the estimator reliably converges to the optimal policy, with final errors strongly dependent on the perturbation parameter. Too large perturbations degrade the learned policy, while too small perturbations slow convergence due to increased estimator variance.

Figure 1: Evolution of validation rewards on two-state two-action problem with MF-REINFORCE for different values of $\,\varepsilon\,$.

Cybersecurity Mean-Field Control

The algorithm is applied to a discrete-time variant of a mean-field cybersecurity model, with four population states and dynamically evolving infection and defense probabilities. MF-REINFORCE achieves convergence to policies that closely reproduce both optimal population state trajectories and rewards obtained by value-based methods (mean-field Q-learning), showing the practical viability of the approach for complex, multi-agent nonstationary environments.

Figure 2: Evolution of validation rewards on cybersecurity problem with MF-REINFORCE for different values of $\,\varepsilon\,$.

Figure 3: Flow of state distribution on cybersecurity problem using the learned policy (MF-REINFORCE, $\,\varepsilon=1.0$); dashed lines: Q-learning reference.

Distribution Planning

A distribution planning scenario over a discrete $10$-state space demonstrates MF-REINFORCE's robustness. Here, larger perturbations lead to faster convergence and more stable training, but final population distributions differ qualitatively depending on the perturbation magnitude, further illustrating the bias-variance tradeoff predicted by theory.

Figure 4: Evolution of validation rewards on distribution planning problem with MF-REINFORCE for different values of $\,\varepsilon\,$.

Implications, Limitations, and Outlook

This work marks a significant expansion of the toolbox for mean-field control, specifically by showing that policy gradient optimization is viable in model-free discrete settings—with rigorous error control. The logits perturbation approach effectively transposes the strengths of REINFORCE-like estimators to population-level control environments. The theoretical analyses clarify the sample complexity, optimality, and limitations of such estimators, especially as the horizon and population coupling scale increases.

Practically, MF-REINFORCE offers immediate applicability to multi-agent systems, networked control, and economics where explicit models of transition or reward are unavailable, but efficient simulation is possible. Its bias-variance analysis and algorithmic structure provide guidelines for hyperparameter choice and estimator scaling.

Future directions may include:

• Extending the approach to continuous spaces and infinite horizons.
• Incorporating entropy regularization or additional variance reduction strategies.
• Tightening complexity bounds and convergence analysis under function approximation or deep RL architectures.
• Adapting the framework to competitive multi-agent mean-field games and heterogeneous agent populations.

Conclusion

The paper establishes both the theory and implementation of model-free policy gradient methods in finite-state mean-field control, providing an estimator construction, convergence guarantees, explicit bias and variance analysis, and empirical validation on distinct test problems. The results strengthen the foundation for scalable policy-gradient approaches in population-based RL settings and pose new questions regarding optimization and sample complexity in high-dimensional, long-horizon, and strongly coupled environments.