Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations

Abstract: This paper investigates the continuous-time counterpart of the Q-function for entropy-regularized mean-field control (MFC) with controlled common noise, coined as q-function by Jia and Zhou (2023) in the single agent's model. We first show that, under discretely sampled actions, the value function in the exploratory formulation converges to the one in the relaxed control formulation as the time grid refines. Leveraging the relaxed control formulation, we derive the exploratory Hamilton-Jacobi-Bellman (HJB) equation, in which the controlled common noise gives rise to an additional nonlinear functional of policy, rendering the policy iteration intricate. Under certain concavity condition, we establish the existence and uniqueness of the optimal one-step policy iteration via a first-order condition using the partial linear functional derivative with respect to policy. The policy improvement at each iteration is verified by relating to an entropy-regularized optimization problem over the space of policies. In the mean-field setting, we introduce the integrated q-function (Iq-function) defined on the state distribution and the policy, and it is shown that an optimal policy is identified as a two-layer fixed point to the argmax operator of the Iq-function. Finally, we provide the explicit characterization of an optimal policy as a Gaussian distribution in the general linear-quadratic (LQ) setting.

• The paper establishes a rigorous theoretical framework for continuous-time q-learning in mean-field control with common noise.
• It introduces an integrated q-function and a two-layer fixed point strategy to characterize optimal policies in stochastic control.
• It provides explicit analysis for the LQ case, justifying Gaussian policies and reinforcing policy improvement iterations in complex systems.

Foundations of Continuous-Time q-Learning for Mean-Field Control with Common Noise

---

Introduction and Motivation

The paper "Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations" (2604.27372) develops the theoretical underpinnings of continuous-time q-learning in mean-field control (MFC) problems subject to common noise. The MFC framework is critical in modeling and optimizing large population systems where the agents’ dynamics depend not only on local actions and idiosyncratic randomness but also on aggregate (mean-field) effects and exogenous common noise sources. This setup is prevalent in a wide array of applications, including systemic risk control in finance, resource allocation in engineering, and swarm robotics.

Despite the advances in reinforcement learning (RL) in continuous-time settings for single-agent models and, to a lesser extent, in mean-field games (MFG) and MFC, the presence of correlated, controlled common noise introduces formidable challenges both in the mathematical characterization of optimal policies and in the construction of learning algorithms. This paper rigorously addresses these challenges, formulating the exact structure of the integrated q-function (Iq-function) in the continuous-time MFC framework with controlled common noise and establishing fixed-point characterizations for the associated policy iteration procedures.

---

Problem Formulation

The paper considers a mean-field stochastic control setting in continuous time, where a population of agents is coordinated by a central "social planner" to maximize an aggregate welfare objective. Each agent’s state evolves according to a controlled McKean-Vlasov SDE that incorporates both idiosyncratic and common (Brownian) noise. Crucially, the drift, diffusion, and reward functions can all depend on the individual state, the conditional population distribution, and the agent’s action.

In practical applications, explicit knowledge of the system dynamics is seldom available, motivating the use of RL. The paper utilizes entropy regularization (through a Shannon entropy term) to facilitate randomized policy exploration and exploits relaxed controls (randomized action distributions) as analytical tools. Two formulations are discussed:

• Relaxed control formulation: Randomized controls are viewed as action distributions, and the dynamics are reformulated accordingly, necessitating auxiliary Brownian motions to preserve key distributional properties in the presence of controlled common noise.
• Exploratory formulation with discretely sampled actions: Actions are resampled from the policy only at discrete time points on a grid; taking the grid mesh to zero establishes equivalence to the continuous-time relaxed formulation.

---

Theoretical Contributions

Exploratory HJB Equation and Policy Iteration

One key technical advance is the derivation of the proper exploratory Hamilton–Jacobi–Bellman (HJB) equation in this mean-field, common-noise context. The presence of controlled common noise fundamentally transforms the structure of the HJB equation, introducing an additional nonlinear, functional dependence on the policy in the drift and diffusion terms. As a result, updating the policy during iteration (policy improvement) no longer admits a closed-form solution (such as the standard Gibbs update).

The authors establish:

• Existence and uniqueness of the (implicit) optimal one-step policy iteration: By formulating a first-order optimality condition using the partial linear functional derivative with respect to the policy, they demonstrate that, under sufficient concavity assumptions (in both the classical and displacement sense), a unique improved policy exists at each iteration.
• Policy improvement guarantee: Given the implicit structure of the policy update (as a solution to a nonlinear fixed-point equation), the authors nonetheless rigorously prove that each update yields a non-decreasing value function, generalizing standard policy improvement results to this more complex, measure-argument setting.

Integrated q-Function (Iq-function)

The correct definition of the continuous-time Iq-function for MFC with common noise is established for the first time. The Iq-function is constructed as the derivative (in the time increment step) of the value under a policy perturbed on a short interval using an alternative action distribution. Unlike in standard or common-noise-free mean-field RL, the Iq-function here depends inextricably on the nonlinear policy functional introduced through common noise. This is formalized as follows:

$$q^\gamma(t,\mu,{\bm h}; {\bm \pi}) = \frac{\partial \tilde J}{\partial t}(t, \mu; {\bm \pi}) - \beta \tilde J(t, \mu; {\bm \pi}) + \mathscr{H}^\gamma(t, \mu, {\bm h}; {\bm \pi})$$

where $\tilde J$ is the value function, $\mathscr{H}^\gamma$ is the entropy-regularized integrated Hamiltonian, and ${\bm h}$ is a probability transition kernel representing the exploratory policy.

Two-Layer Fixed Point Structure and Policy Characterization

A significant conceptual advancement is the demonstration that the optimal policy is characterized by a two-layer fixed-point problem over the space of policies:

1. For a given reference policy ${\bm \pi}$, the optimal response is obtained as a fixed point of a nonlinear map $\Phi_{{\bm \pi}}$, expressed in a generalized Gibbs-type form involving the partial derivative of the (unregularized) Iq-function.
2. The overall optimal policy is then given as a fixed point of the operator $\mathcal{I}$ that maps a policy to its iterated improvement under this procedure.

This contrasts sharply with the explicit Gibbs policy updates found in the classical and mean-field control settings without controlled common noise and illustrates the profound additional complexity introduced by common noise.

---

Linear-Quadratic (LQ) MFC with Common Noise

The paper provides a highly nontrivial analytical result for the general linear-quadratic MFC case with controlled common noise. In this setting, the authors explicitly solve for:

• The quadratic value function parameters (evolving according to a coupled system of matrix Riccati ODEs).
• The unique optimal policy, which is shown to be a Gaussian with mean and covariance expressed as two-layer fixed points involving the value function and its derivatives. This provides rigorous justification for adopting Gaussian policies in continuous-time RL even when common noise is present and controlled.

---

Implications and Further Research Directions

The findings of this paper have immediate theoretical and practical implications:

• Algorithmic design: These results provide rigorous foundations for model-free continuous-time q-learning in high-dimensional distributed control problems subject to exogenous systemic risk factors.
• Policy iteration and actor-critic methods: The generalized two-layer fixed point framework paves the way for new algorithmic architectures in RL for complex stochastic control beyond standard actor-critic or policy gradient methods.
• Financial engineering, network systems, and robotics: Directly relevant models include mean-variance portfolio optimization with correlated risk, city-scale traffic management, and resource allocation in large interconnected systems.
• Extension to risk-sensitive and nonlinear reward regimes: The measure-derivative and fixed-point tools developed herein facilitate further generalizations to risk-sensitive, time-inconsistent, and path-dependent mean-field control problems under common noise.

Future developments may focus on the design and analysis of finite-sample and neural network-based approximations under the established theoretical machinery, as well as empirical validation in applied domains.

---

Conclusion

This paper (2604.27372) rigorously establishes the theoretical framework for continuous-time q-learning in mean-field control systems with controlled common noise. The main advances include the derivation of the appropriate HJB equation with nonlinear policy functionals, the existence and uniqueness of the (implicit) policy improvement operator, and the fixed-point-based characterization of optimal policies in both general and LQ settings. These findings fill a critical gap in the RL literature for systems with aggregate interactions and correlated disturbances and lay a robust foundation for further research at the intersection of stochastic control, RL, and high-dimensional mean-field systems.