---
title: Monotone Gradient Networks (mGradNets)
url: https://www.emergentmind.com/topics/monotone-gradient-networks-mgradnets
type: topic
---

# Monotone Gradient Networks (mGradNets)

Searching arXiv for the primary mGradNet papers and closely related work.
Monotone Gradient Networks (mGradNets) are neural architectures for vector-valued maps \(f:\mathbb R^d\to\mathbb R^d\) that are constrained to satisfy \(f(x)=\nabla F(x)\) for some convex, twice continuously differentiable potential \(F\in C^2(\mathbb R^d)\). In the formulations introduced as C-MGN and M-MGN, and later generalized as mGradNet-C and mGradNet-M, the defining property is the global positive semidefiniteness of the Jacobian, \(J_f(x)\succeq 0\) for all \(x\), which makes the network a direct parameterization of monotone gradients rather than of scalar convex functions that must then be differentiated [2301.10862], [2404.07361]. This design was motivated by applications in gradient-based optimization, normalizing flows, and especially optimal transport, where Brenier’s theorem identifies the squared-Euclidean Monge map as the gradient of a convex potential. Subsequent work used mGradNets to learn such transport maps directly through Monge–Ampère-based objectives and applied them to robot swarm control [2507.13191].

## 1. Mathematical characterization and motivation

A vector-valued map \(g:\mathbb R^n\to\mathbb R^n\) is monotone if
\[
(g(x)-g(y))^\top (x-y)\ge 0,\qquad \forall x,y\in\mathbb R^n.
\]
By classical results cited in the original formulation—Rockafellar (1970) and Boyd & Vandenberghe (2004)—a continuously differentiable scalar function \(f:\mathbb R^n\to\mathbb R\) is convex if and only if its gradient \(\nabla f\) is monotone [2301.10862]. The later GradNet framework restated this in differential form: an mGradNet is precisely a gradient network whose Jacobian is symmetric positive semidefinite everywhere, \(J_f(x)\succeq 0\) [2404.07361].

The immediate motivation for learning monotone gradients directly is operational. Many optimization algorithms require only a gradient oracle. In the Monge problem with squared-Euclidean cost, Brenier’s theorem guarantees that the optimal transport map is the gradient of a convex function. Direct modeling of \(\nabla f\) therefore targets the object of interest itself. This differs from Input Convex Neural Networks (ICNNs), which parameterize a convex potential and require a subsequent differentiation step, and from Input Convex Gradient Networks (ICGNs), which use Hessian factorization [2301.10862].

In the optimal-transport setting, the Monge formulation seeks a map \(T:\mathbb R^N\to\mathbb R^N\) minimizing
\[
\inf_{T:\,T_\# p=q}\int \|x-T(x)\|^2 p(x)\,dx.
\]
Under the regularity conditions stated in the GradNetOT exposition, the unique minimizer has the form
\[
T^*(x)=\nabla \phi(x),
\qquad \phi:\mathbb R^N\to\mathbb R \text{ convex},
\]
and the pushforward condition is equivalent to the Monge–Ampère relation
\[
p(x)=q(\nabla\phi(x))\,\det(D^2\phi(x)).
\]
This places monotone gradients at the center of data-driven transport-map learning [2507.13191].

## 2. Canonical architectures: cascaded and modular forms

The original paper introduced two architectures for directly learning gradients of convex functions: the Cascaded Monotone Gradient Network (C-MGN) and the Modular Monotone Gradient Network (M-MGN) [2301.10862]. Their later counterparts in the GradNet framework are denoted mGradNet-C and mGradNet-M [2404.07361].

| Architecture | Core structure | Monotonicity mechanism |
|---|---|---|
| C-MGN / mGradNet-C | Deep cascaded network with shared \(W\) | Jacobian factors as \(W^\top D(x)W\) |
| M-MGN / mGradNet-M | Sum of parallel modules | Each module contributes PSD terms |

For C-MGN, the 2023 construction uses a shared linear weight \(W\in\mathbb R^{n\times n}\), bias vectors \(b_\ell\), and layer-wise monotonic activations \(\sigma_\ell\). With \(L\) hidden layers,
\[
z_0=Wx+b_0,
\]
\[
z_\ell=Wx+\sigma_\ell(z_{\ell-1})+b_\ell,\qquad \ell=1,\dots,L-1,
\]
and
\[
g(x)=W^\top \sigma_L(z_{L-1})+V^\top Vx+b_L.
\]
If each \(\sigma_\ell\) is element-wise increasing, then
\[
J_g(x)=W^\top \Bigl(\sum_{\ell=1}^L \prod_{i=\ell}^L \operatorname{diag}(\sigma_i'(z_{i-1}))\Bigr)W+V^\top V \succeq 0.
\]
The 2024 mGradNet-C variant preserves the same shared-\(W\) cascaded logic while introducing nonnegative layer-wise coefficients \(\alpha_\ell,\beta_\ell,\gamma_L\), yielding the factorization \(J_f(x)=W^\top D(x)W\) with diagonal PSD \(D(x)\) [2404.07361].

For M-MGN, the network is built from \(K\) independent modules. Each module has \(W_k\in\mathbb R^{n\times n}\), a bias \(b_k\), and a scalar convex function \(s_k:\mathbb R^n\to\mathbb R_+\) with \(\nabla s_k=\sigma_k\). The model is
\[
z_k=W_kx+b_k,\qquad k=1,\dots,K,
\]
\[
g(x)=a+V^\top Vx+\sum_{k=1}^K \bigl[s_k(z_k)\cdot W_k^\top \sigma_k(z_k)\bigr].
\]
Since each \(s_k\) is convex, \(J_{\sigma_k}(z_k)=H_{s_k}(z_k)\succeq 0\), and the Jacobian becomes
\[
J_g(x)=V^\top V+\sum \bigl[s_k\,W_k^\top H_{s_k}W_k + (W_k^\top \sigma_k)(W_k^\top \sigma_k)^\top\bigr]\succeq 0.
\]
The corresponding mGradNet-M formulation in 2024 writes
\[
z_m=W_mx+b_m,
\qquad
h(x)=a+\sum_{m=1}^M \rho_m(\phi_m(z_m))\,W_m^\top \sigma_m(z_m),
\]
with \(\phi_m\in C^2\) convex, \(\sigma_m=\nabla\phi_m\), and \(\rho_m\) nonnegative and nondecreasing, which again ensures that every module contributes PSD terms [2404.07361].

The contrast between the two families is structural rather than semantic. C-MGN is described as a “deep” cascaded structure with one shared \(W\) and potentially more nonlinear depth; M-MGN is a “wide” structure with parallel shallow modules whose contributions are summed [2301.10862].

## 3. Theoretical guarantees and representational scope

The foundational guarantee is monotonicity. Propositions 1 and 2 in the original paper prove \(J_g(x)\succeq 0\) for all \(x\), hence the learned map is guaranteed to be \(\nabla f\) for some convex \(f\) [2301.10862]. In the later GradNet formalization, this becomes the defining criterion of an mGradNet, and several closure constructions are given: a single-layer recipe \(f(x)=W^\top \sigma(Wx+a)+b\) with \(\sigma=\nabla\psi\) and \(J_\sigma\succeq 0\); a “Lipschitz flip” \(g(x)=Lx-f(x)\) when a GradNet is \(L\)-Lipschitz; conical combinations of mGradNets; and strong-convex shifts \(g(x)=f(x)+\mu x\), which produce gradients of \(\mu\)-strongly-convex potentials [2404.07361].

Strong convexity has immediate consequences for invertibility. The original work notes that adding a linear term \(V^\top Vx\) can enforce \(J_g(x)\succeq \gamma I\) when \(V^\top V\succeq \gamma I\). In that case the map is \(\gamma\)-strongly monotone, the underlying potential is \(\gamma\)-strongly convex, and the resulting \(g\) is bijective and invertible, which is explicitly identified as useful in normalizing flows [2301.10862].

A notable point in the development of the literature concerns universal approximation. The original 2023 exposition states that no formal universal-approximation bound was proved for monotone gradients, although empirical expressivity was strong [2301.10862]. The 2024 GradNet paper then established a universal approximation theorem for gradients of convex functions. Specifically, for the class \(\mathcal C=C^1_{\mathrm{cvx}}([0-\delta,1+\delta]^d)\), the single-layer mGradNet of Proposition 8 with scaled softmax activation,
\[
\sigma(z)=\operatorname{softmax}(tz),\qquad t\to\infty,
\]
universally approximates \(\nabla\mathcal C\) on \([0,1]^d\). The proof sketch proceeds through uniform approximation of convex functions by scaled LogSumExp of finitely many affine hyperplanes and then differentiates that representation [2404.07361].

The same paper also characterizes a larger function family: “monotone convex transformation of a sum of convex ridges,”
\[
F(x)=T\Bigl(\sum_{i=1}^N \psi_i(a_i^\top x+b_i)\Bigr),
\]
with \(\psi_i,T\) convex \(C^k\) and \(T\) nondecreasing. The corresponding gradient can be implemented as an mGradNet by combining a ridge-sum gradient \(g(x)\) with a scalar multiplier \(\gamma(u)=T'(u)\) [2404.07361]. This suggests that the architecture is not restricted to a narrow template of convex potentials.

## 4. Training objectives and computational profile

Three training modes recur across the mGradNet literature. The first is direct gradient regression. In the original formulation, when samples of the true gradient are available, the network is supervised with
\[
L_{\mathrm{grad}}=\mathbb E_x\|g(x)-\nabla f(x)\|_2^2
\]
or an \(L_1\) variant [2301.10862]. The 2024 GradNet study uses the same principle in the form
\[
L(\theta)=\mathbb E_x\|f_\theta(x)-\nabla F(x)\|^2,
\]
and emphasizes that no extra Jacobian-symmetry penalty is needed because symmetry and PSD structure are built into the architecture [2404.07361].

The second mode is density matching or optimal transport via a change-of-variable objective. In the 2023 paper, \(g\) is treated as a normalizing flow and trained by minimizing a KL-based objective,
\[
L_{\mathrm{OT}}=\mathbb E_{x\sim p_X}\bigl[-\log p_Y(g(x))-\log |\det J_g(x)|\bigr].
\]
Optimization uses standard SGD or Adam, with the same batch sizes and learning rates for all competing models in the experiments [2301.10862].

The third mode, developed in GradNetOT, imposes the pushforward condition in logarithmic residual form:
\[
\log p(x)=\log q(T_\theta(x))+\log\det(J_{T_\theta}(x)).
\]
With residual
\[
r(\theta;x)=\log\det(J_{T_\theta}(x))-\bigl[\log p(x)-\log q(T_\theta(x))\bigr],
\]
the optimization problem is
\[
\theta^*=\arg\min_\theta \mathbb E_{x\sim p(x)}[r(\theta;x)]^2.
\]
The reported implementation uses Adam with batch size \(B=1000\), a decaying learning rate from \(10^{-2}\) to \(10^{-4}\), and automatic differentiation to compute both \(T_\theta(x)\) and \(\log\det J_{T_\theta}(x)\) [2507.13191].

Computationally, the architecture is intended to avoid the burdens that accompany indirect convex-potential learning. The 2024 study states that forward/backward cost is comparable to an MLP of the same size and that no costly numerical integration or PSD-projection is used. Nonnegativity constraints, such as those on \(\alpha,\beta\) in mGradNet-C and on \(\rho_m'\) in mGradNet-M, are enforced by parameterization as squares or via softplus [2404.07361]. This is consistent with the original motivation that direct parameterization of \(\nabla f\) avoids both the extra differentiation step of ICNNs and the heavy Hessian-factorization required by ICGNs [2301.10862].

## 5. Empirical performance on gradient fields and transport tasks

The earliest experiments focused on a synthetic two-dimensional gradient field. In the 2023 comparison, ICGN used 15 parameters, ICNN used 78 and 163 parameters, C-MGN used 14 parameters, and M-MGN used 22 parameters. The reported MSEs in dB were: ICGN \(-15.00\), ICNN (78p) \(-4.15\), ICNN (163p) \(-30.88\), C-MGN (14p) \(-39.10\), and M-MGN (22p) \(-32.31\). The paper states that C-MGN beats all baselines with an order-of-magnitude fewer parameters [2301.10862].

The broader 2024 GradNet evaluation used comparable parameter counts in 2D—mGradNet-C with \(L=3\), hidden dim \(\approx 7\), had \(\sim 100\) parameters; mGradNet-M with \(M=4\) modules, each dim \(\approx 7\), had \(\sim 94\); ICNN had \(\approx 109\), ICGN \(\approx 96\), and CRR \(\approx 102\). On the convex field benchmark over \([0,1]^2\), the RMSEs in dB were ICNN \(-17.1\), ICGN \(-10.6\), CRR \(-15.7\), mGradNet-C \(-24.1\), and mGradNet-M \(-23.6\), corresponding to an improvement of approximately \(6\)–\(7\) dB over the best baseline [2404.07361].

Optimal-transport experiments in the original paper evaluated Gaussian optimal coupling in dimensions \(d=2\) and \(d=16\), with baselines Whitening Transform (WT), Invertible Autoregressive Flow (IAF), and Convex Potential Flow (CP Flow). For \(d=2\), the reported NLL and transport cost were WT \(2.83, 4.13\); IAF \(3.00, 4.04\); CP Flow \(3.10, 3.47\); C-MGN \(2.86, 3.30\); M-MGN \(2.87, 3.21\). For \(d=16\), they were WT \(22.70, 235.28\); IAF \(22.53, 234.82\); CP Flow \(22.71, 210.10\); C-MGN \(22.67, 212.66\); M-MGN \(22.61, 211.84\). The paper concludes that both C-MGN and M-MGN match or outperform CP Flow and IAF in NLL and incur lower transport cost than IAF, with simpler architectures [2301.10862].

GradNetOT extended the empirical range to Gaussian-to-Gaussian transport up to \(d=50\). In the 2D case, a standard unconstrained feed-forward net is reported to fail because points “cross” under the mapping, whereas both mGradNet-C and mGradNet-M recover the analytic whitening transform. In higher dimensions, the mean squared error between the learned map and the true whitening transform remains near zero, with mGradNet-M slightly outperforming mGradNet-C [2507.13191]. A plausible implication is that the structural bias imposed by PSD Jacobians is especially relevant when the target map is known a priori to be a Brenier map.

## 6. Applications, distinctions, and open directions

A first applied use case is color domain adaptation for autonomous driving. The task is to map RGB pixel colors of daytime road scenes to sunset-style scenes by learning the Monge map between their color distributions. The reported pipeline fits a multivariate Gaussian to target pixel colors, learns a map \(g\) from day to Gaussian via M-MGN or C-MGN by minimizing Gaussian NLL, and applies the learned map to test images for visual comparison against a kernel-OT baseline (OTKer). The stated outcome is that C-MGN and M-MGN produce color transforms on par with more complex convex-relaxation OT methods while being learned end-to-end with few parameters [2301.10862].

A second application is robot swarm control. GradNetOT encodes initial and desired swarm layouts as smooth densities on \([0,1]^2\) via MNIST grayscale images and KDE, then learns an mGradNet-M transport map that tracks the exact Wasserstein geodesic between the distributions. In comparison to barycentric projection of the discrete Kantorovich plan, the averaged MSE over 1,000 test points is reported to be on the order of \(10^{-3}\) across \(5\times 5\) source-target digit pairs [2507.13191].

A recurrent point of clarification concerns terminology. mGradNets are not the same object as monotone variational-inequality training methods. The latter construct a monotone operator \(F(\theta)\) in parameter space, solve a VI of the form “find \(\theta^*\in\Theta\) such that \(\langle F(\theta^*),\theta-\theta^*\rangle\ge 0\) for all \(\theta\in\Theta\),” and apply projected or extragradient schemes for training or fine-tuning FC, GNN, and CNN models [2202.08876]. By contrast, mGradNets constrain the input-output Jacobian \(J_f(x)\) so that the network itself represents a monotone gradient field. The shared word “monotone” therefore refers to different mathematical objects in the two lines of work.

Open directions were already identified in the original formulation: convolutional generalization by replacing fully connected \(W\) or \(W_k\) with convolution and its adjoint while preserving PSD Jacobians; richer module compositions; theoretical error bounds; scaling to very high dimensions; and integration into end-to-end task training [2301.10862]. The later GradNet framework partially addressed one earlier limitation by supplying a universal approximation theorem [2404.07361]. This suggests that current research is progressively shifting from existence and feasibility questions toward scalability, compositional design, and task-specific transport objectives.

Source: https://www.emergentmind.com/topics/monotone-gradient-networks-mgradnets