Mean Field Langevin Dynamics

• Mean Field Langevin Dynamics is a stochastic process that governs the evolution of probability measures and serves as the infinite-width limit of noisy gradient descent in interacting particle systems.
• It is characterized by a McKean–Vlasov stochastic differential equation and a nonlinear Fokker–Planck PDE, with convergence ensured by convexity and log-Sobolev conditions.
• The framework enables precise analysis of both continuous and discrete flows, providing applications in large-scale optimization and empirical risk minimization through a proximal Gibbs law formulation.

Mean Field Langevin Dynamics (MFLD) is a nonlinear stochastic process governing the evolution of probability measures on $\R^d$, which arises as the infinite-width (mean field) limit of noisy gradient descent in interacting particle systems, including the training of infinitely wide neural networks. Formally, MFLD describes the gradient flow in the 2-Wasserstein metric of a convex, entropy-regularized functional, and it is characterized by a McKean–Vlasov stochastic differential equation (SDE) or an associated nonlinear Fokker–Planck partial differential equation (PDE). The dynamics combine mean-field-dependent drifts with isotropic noise, yielding global exponential convergence under appropriate convexity and log-Sobolev conditions. The framework admits rigorous analyses in both continuous and discrete time and underpins numerous recent advances in large-scale optimization, sampling, and learning over probability measures.

1. Definition and Fundamental Equations

Let $P$ denote the set of probability densities $q(\theta)$ on $\R^d$ with finite entropy and finite second moment. For a fixed convex, differentiable energy functional $E(q)=\int f(\theta)q(\theta)d\theta$ and a positive regularization parameter $\lambda>0$, the entropy regularized objective is

$$F(q) = E(q) + \lambda H(q), \quad H(q)=\int q(\theta)\log q(\theta)d\theta.$$

The first variation (Wasserstein gradient) of $F$ at $q$ is

$\frac{\delta F}{\delta q}(q):\R^d\to\R.$

The mean-field Langevin SDE is

$P$0

Its law $P$1 evolves via the nonlinear Fokker–Planck PDE

$P$2

or equivalently

$P$3

where $P$4 is the (time-dependent) proximal Gibbs law.

2. Variational Structure and Proximal Gibbs Law

The MFLD is the 2-Wasserstein gradient flow of the free energy $P$5. At the density level, the Wasserstein gradient reads

$P$6

The proximal Gibbs distribution $P$7 is defined for any $P$8 as

$P$9

Equivalently,

$q(\theta)$0

Importantly, $q(\theta)$1 is the unique minimizer (over $q(\theta)$2) of the Bregman-proximal update

$q(\theta)$3

The optimization gap satisfies

$q(\theta)$4

where $q(\theta)$5 is the minimizer of $q(\theta)$6.

3. Convergence Results in Continuous and Discrete Time

Continuous Time

Assuming convexity/smoothness of $q(\theta)$7 and a uniform log-Sobolev inequality (LSI) for each $q(\theta)$8 with constant $q(\theta)$9,

$\R^d$0

the continuous mean-field Langevin flow satisfies

$\R^d$1

i.e., exponential convergence of the functional gap at rate $\R^d$2.

Discrete Time

Consider the discretization

$\R^d$3

Letting $\R^d$4 denote the law at step $\R^d$5, and $\R^d$6 the per-step drift discretization error, the objective gap satisfies

$\R^d$7

If $\R^d$8, the discrete process inherits a linear convergence rate up to $\R^d$9 bias and has iteration complexity

$E(q)=\int f(\theta)q(\theta)d\theta$0

to reach $E(q)=\int f(\theta)q(\theta)d\theta$1.

4. Proof Strategy and Key Functional Inequalities

The core convergence mechanism is the energy-dissipation identity

$E(q)=\int f(\theta)q(\theta)d\theta$2

which provides monotonic decay of $E(q)=\int f(\theta)q(\theta)d\theta$3. The log-Sobolev inequality for $E(q)=\int f(\theta)q(\theta)d\theta$4 ensures

$E(q)=\int f(\theta)q(\theta)d\theta$5

The convexity optimality gap lemma ties KL divergence to the functional gap, $E(q)=\int f(\theta)q(\theta)d\theta$6, yielding the exponential decay ODE

$E(q)=\int f(\theta)q(\theta)d\theta$7

Discrete analysis is performed via a perturbed Fokker–Planck equation; the discretization error is controlled by smoothness bounds, and a discrete Grönwall argument gives geometric convergence in $E(q)=\int f(\theta)q(\theta)d\theta$8.

5. Application to Empirical Risk Minimization: Duality Gap

In empirical risk minimization,

$E(q)=\int f(\theta)q(\theta)d\theta$9

The Fenchel dual variable $\lambda>0$0 satisfies

$\lambda>0$1

where $\lambda>0$2 is the proximal Gibbs law associated to $\lambda>0$3. Hence, the KL divergence to $\lambda>0$4 is the exact primal-dual gap, providing a directly computable and efficiently decaying stopping criterion in convex empirical risk optimization (Nitanda et al., 2022).

6. Interpretations, Extensions, and Practical Consequences

MFLD provides the formal mean-field limit for noisy gradient descent on highly overparameterized systems, notably infinite-width neural networks, as recently formalized in the context of convex lifting to measure space. The functional $\lambda>0$5 encodes risk or energy, and the entropy regularization $\lambda>0$6 ensures strong convexity, uniqueness, and tractable convergence analysis. The introduction of the proximal Gibbs law allows precise quantification of algorithmic gaps and enables connections to duality.

The convergence analysis is robust to both continuous- and discrete-time implementations, and in practice, time and space discretizations can be matched to the regularization and LSI constants to control both statistical and computational error. These results underpin rigorous algorithms for distributional optimization and learning in machine learning, providing both sharp rates and diagnostics for algorithmic progress (Nitanda et al., 2022).