Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithms

Abstract: This paper is a continuation work of Ren et al. (2026) aiming to further devise q-learning algorithms for mean-field control (MFC) with controlled common noise. Based on the relaxed control formulation, we first establish the martingale condition of the value function and the Iq-function by evaluating along the conditional state distributions generated by all test policies. As the data in the relaxed control formulation are not observable in practice, we quantify the error incurred when they are replaced by the observable ones in the exploratory formulation under discretely sampled actions. This, together with a two-layer fixed point characterization of an optimal policy in Ren et al. (2026), allows us to propose several algorithms including the Actor-Critic q-learning algorithm, in which the policy is updated in the Actor-step based on the iteration rule induced by the improved Iq-function, and the value function and Iq-function are updated in the Critic-step based on the martingale orthogonality condition using the data from the exploratory formulation. We also establish the convergence of the inner iterations in the Actor-step in an infinite-horizon linear quadratic (LQ) framework. In two examples, within and beyond LQ framework, our q-learning algorithms are implemented with satisfactory performance.

• The paper introduces novel continuous-time q-learning and actor-critic schemes that learn optimal policies without explicit dynamics via a new Iq-function.
• It establishes martingale characterizations and fixed-point conditions that provide rigorous convergence guarantees, especially in linear-quadratic settings.
• Numerical examples show rapid convergence and low error, highlighting the scalability and practical relevance for large-scale stochastic systems.

Continuous-Time Q-Learning Algorithms for Mean-Field Control with Common Noise

Overview and Motivation

This paper addresses the construction and analysis of reinforcement learning algorithms for mean-field control (MFC) in continuous time, specifically focusing on systems with common noise. The principal aim is the development of q-learning and actor-critic schemes that operate without explicit knowledge of system dynamics, thus enabling the social planner to learn optimal policies via data-driven methods in large-scale stochastic systems. The theoretical underpinnings are built upon the martingale characterization of value functions and a novel integrated q-function (Iq-function), especially accommodating the complexities introduced by common noise and the two-layer fixed point structure in policy characterization.

Martingale Characterization and Iq-Function Construction

The foundation for continuous-time MFC with common noise is established via two frameworks: a relaxed control formulation for theoretical analysis and an exploratory formulation suitable for practical learning with discretely sampled actions. A detailed martingale characterization of the value function and Iq-function is provided, extending prior single-agent results. The Iq-function, defined as the first-order time derivative of the advantage function, encapsulates the combined effects of reward structure, transition dynamics, and entropy regularization. Importantly, this paper quantifies the error incurred when martingale conditions are computed with observed data instead of theoretically ideal data, highlighting the nuanced divergence between mean-field and single-agent RL settings.

Q-Learning Algorithm Design

Optimal Q-Learning

When the policy fixed point structure admits explicit solutions (particularly in linear quadratic (LQ) settings), the paper proposes an optimal q-learning algorithm. Here, the value function and Iq-function can be parameterized cohesively, and the learning algorithm is driven by a temporal difference update based on the averaged martingale orthogonality condition over a set of test policies. Strong convergence guarantees are provided, with explicit parameterizations enabling direct policy learning.

Actor-Critic Q-Learning

For general cases lacking explicit two-layer fixed point solutions, the paper develops a continuous-time actor-critic algorithm. The actor step updates policy parameters using a stochastic gradient ascent over an implicit policy improvement operator defined via the improved Iq-function, relying on partial functional derivatives. The critic step updates the value and Iq-function parameters based on the martingale orthogonality condition with observed data from discretely sampled actions. The procedure alternates between evaluating and improving the policy, and introduces inner fixed-point iterations in the actor step with rigorous convergence proofs in the LQ infinite-horizon setting.

Figure 1: Convergence of parameters $\theta$ during q-learning for MFC with common noise in the LQ framework.

Figure 2: Convergence of parameters $\theta$ in the actor-critic algorithm illustrating the update dynamics in the generalized setting.

Numerical Implementation and Results

Two numerical examples are provided to demonstrate algorithmic efficacy: a linear-quadratic MFC benchmark and a non-LQ MFC model. For both, explicit forms of optimal policy, value function, and Iq-function are derived, facilitating precise parameterization. The implementation compares the optimal q-learning algorithm, actor-critic without inner iterations, and actor-critic with inner iterations. All algorithms show rapid convergence and high parameter accuracy, with value function error metrics substantiating performance.

Figure 3: Convergence of parameters $\theta$ in actor-critic q-learning with inner iterations, affirming the stability of the fixed point procedure.

Figure 4: Value function $L^1$ error in the optimal q-learning setup, quantifying the precise convergence of learning.

Figure 5: Value function $L^1$ error for actor-critic q-learning, validating robust learning of the value function.

Figure 6: Value function $L^1$ error in actor-critic q-learning with inner iterations, demonstrating convergence to optimality under sequential updates.

Notably, in both examples, inner fixed-point iterations in the actor step were observed to be non-essential, as alternative critic-actor step iterations sufficed to converge to the two-layer fixed point. This is a strong numerical claim, diverging from the intuition that explicit inner iterations are required for learning in implicit fixed-point settings.

Theoretical Implications and Practical Impact

The paper contributes several formal results:

• Martingale characterizations for value and q-functions under common noise;
• Quantitative error analysis for substituting theoretical data with observed data in learning;
• Two-layer fixed-point characterization for optimal policies in continuous-time MFC with common noise, extending the Gibbs measure structure;
• Rigorous convergence theorems for actor-critic iterations in infinite-horizon LQ settings.

From a theoretical standpoint, the work clarifies the structural complexity introduced by common noise, notably rendering optimal policies as implicit solutions of a nested fixed-point operator. Practically, the algorithms are highly scalable, requiring no model knowledge of the dynamics, and robust to discretization errors—a significant step for RL in large population stochastic systems (e.g., centralized traffic or economic policy).

Future Directions

Open questions remain regarding the full theoretical convergence analysis in non-LQ finite-horizon settings, the generalization of the inner fixed-point iteration convergence to arbitrary mean-field controls, and extensions to more complex reward structures or additional noise sources. Expansion toward neural function representations could enhance expressivity for real-world environments and accelerate learning in high-dimensional spaces. The methodology also suggests potential cross-applications in multi-agent RL with weakly coupled dynamics, portfolio optimization under systemic risk, and optimal execution models.

Conclusion

This paper provides a comprehensive framework for continuous-time q-learning and actor-critic algorithms in mean-field control settings with common noise, introducing a novel Iq-function, martingale-based learning schemes, and rigorous convergence analysis. Numerical implementation confirms algorithmic performance, while theoretical results emphasize the distinctive complexities introduced by mean-field interactions and common perturbations. This paradigm offers significant practical relevance for scalable population-level RL, and opens avenues for further exploration in both theory and applied domains (2604.27378).