---
title: ReLU Monotone Operator Equilibrium Net
url: https://www.emergentmind.com/topics/relu-monotone-operator-equilibrium-network
type: topic
---

# ReLU Monotone Operator Equilibrium Net

Searching arXiv for relevant papers on ReLU monotone operator equilibrium networks and related formulations.
A ReLU Monotone Operator Equilibrium Network is an implicit-depth neural model in which the hidden state is defined by the solution of a ReLU fixed-point equation, typically $z^*=\mathrm{ReLU}(Wz^*+Ux_0+u)$ or $z^*=\mathrm{ReLU}(Wz^*+b)$. The central idea is to replace explicit depth with an equilibrium computation while imposing monotonicity or strong monotonicity on the underlying operator so that the equilibrium is well posed, the solver converges stably, and the model can be differentiated by implicit methods rather than by storing all intermediate iterates [2006.08591]. In inverse problems, the same principle appears in monotone operator learning and in local relaxations such as MnM-MOL, where a ReLU CNN is embedded in a learned reconstruction update and iterated to convergence [2111.11380], [2312.00386].

## 1. Origin and scope of the model class

Monotone Operator Equilibrium Networks, or monDEQs, were introduced as a class of implicit-depth models motivated by two limitations of earlier equilibrium formulations: unstable convergence to a solution and lack of guarantees that a solution exists. The monDEQ construction uses the theory of monotone operators to connect equilibrium computation with monotone operator splitting, thereby obtaining efficient solvers with guaranteed, stable convergence and a parameterization that ensures the existence of a unique equilibrium point [2006.08591]. The same general program was later specialized to model-based inverse problems, where monotone operator learning (MOL) recasts reconstruction as a deep equilibrium computation with explicit monotonicity constraints on a learned operator [2111.11380].

Within this literature, the phrase “ReLU Monotone Operator Equilibrium Network” refers to several closely related instantiations. In the basic monDEQ form, the equilibrium equation is a ReLU implicit layer. In MOL, the equilibrium arises from a reconstruction iteration whose nonlinear block is a ReLU CNN. In semialgebraic certification work, the same ReLU fixed point is treated as an exact polynomially constrained object for SDP-based robustness analysis. In non-Euclidean formulations, the equilibrium is analyzed under weighted $\ell_\infty$ or $\ell_1$ norms rather than the Euclidean norm. These variants share the same structural theme: ReLU provides the nonlinear fixed-point relation, while monotonicity controls existence, uniqueness, convergence, and robustness [2106.01453], [2303.11273], [2106.03194].

A recurring distinction in the literature is between explicit monotonicity of the learned nonlinear block and monotonicity of a larger composite operator. Global MOL constrains the CNN itself to be monotone, whereas MnM-MOL relaxes this by requiring monotonicity only for the sum of the data-term gradient and the CNN block, and only in a local neighborhood of the image manifold [2312.00386]. This addresses a common misconception that every useful ReLU equilibrium model must impose a global monotonicity constraint directly on the CNN.

## 2. Monotone-operator foundations

The operator-theoretic basis of the model is the equivalence between a ReLU fixed point and the zero of a sum of monotone operators. For the equilibrium equation
$$
z^*=\mathrm{ReLU}(Wz^*+b),
$$
one may define
$$
F(z)=(I-W)z-b,\qquad G=\partial f,
$$
where $f$ is the indicator of the nonnegative orthant, so that $\prox_f^1(x)=\max(x,0)=\mathrm{ReLU}(x)$. Solving the fixed-point equation is then equivalent to solving
$$
0\in(F+G)(z^*).
$$
Because $\partial f$ is maximal monotone and $F$ is strongly monotone when $I-W\succeq mI$ with $m>0$, the sum has a unique zero, and the equilibrium equation has exactly one solution [2006.08591].

This same logic underlies the inverse-problem formulation used in MOL. There the target image $x$ is defined by the variational problem
$$
x^*=\arg\min_x \;\frac{\lambda}{2}\|\mathcal A x-b\|_2^2+\varphi(x),
$$
with first-order optimality condition
$$
\lambda\,\mathcal A^T(\mathcal A x^*-b)+\nabla\varphi(x^*)=0.
$$
MOL identifies $F(x)=\nabla\varphi(x)$ and implements it as a residual CNN of the form $F(x)=x-H(x)$, with monotonicity enforced by a Lipschitz bound on $H$ [2111.11380]. MnM-MOL modifies this arrangement by defining the combined operator
$$
Q_\theta(x)=\nabla \ell(x)+N_\theta(x),
$$
where $\ell(x)=\tfrac12\lambda\|Ax-b\|^2$ and $N_\theta$ is a feed-forward CNN with ReLU activations. The monotonicity requirement is placed on $Q_\theta$ rather than on $N_\theta$ itself [2312.00386].

Several parameterizations were proposed to make these conditions trainable. In the original monDEQ construction, strong monotonicity of $I-W$ is equivalent to
$$
W=(1-m)I-A^\top A+(B-B^\top),
$$
for some trainable $A,B\in\mathbb R^{n\times n}$ [2006.08591]. In Lipschitz-Bounded Equilibrium Networks, a weighted monotonicity LMI is enforced by the unconstrained parameterization
$$
W=I-\Psi\bigl(V^TV+\epsilon I+S\bigr),
$$
with diagonal positive-definite $\Psi$, skew-symmetric $S$, and arbitrary $V$, yielding
$$
2\Lambda-\Lambda W-W^T\Lambda\succ0,\qquad \Lambda=\Psi^{-1}.
$$
Under this condition, the ReLU equilibrium is well posed and globally Lipschitz [2010.01732].

## 3. Equilibrium computation and implicit training

The standard solver for a ReLU monDEQ is forward-backward splitting. For the basic equilibrium equation, the update is
$$
z^{k+1}=\mathrm{ReLU}\bigl((1-\alpha)z^k+\alpha(Wz^k+b)\bigr).
$$
If $I-W$ is $m$-strongly monotone and $L$-Lipschitz, forward-backward converges linearly for any step size
$$
\alpha\le\frac{2m}{L^2}.
$$
In a dense $n$-dimensional model, the per-iteration cost is $O(n^2)$ for the matrix-vector product $Wz$ and $O(n)$ for the ReLU operation [2006.08591]. The same paper also discusses Peaceman–Rachford splitting, which often converges in fewer iterations and requires solving a linear system involving $I+\alpha(I-W)$ [2006.08591].

In MOL, the deep equilibrium layer is a forward-backward reconstruction step rather than a direct ReLU recurrence. The fixed-point map is
$$
T(x)=(I+\alpha\lambda\,\mathcal A^T\mathcal A)^{-1}\Bigl(x-\alpha F(x)+\alpha\lambda\,\mathcal A^T b\Bigr),
$$
and with the residual form $F(x)=x-H(x)$ this becomes
$$
x_{k+1}=(I+\alpha\lambda\,\mathcal A^T\mathcal A)^{-1}\Bigl((1-\alpha)x_k+\alpha H(x_k)+\alpha\lambda\,\mathcal A^T b\Bigr).
$$
If $F$ is $m$-monotone and $L_F\le 2-m$, the iteration converges for
$$
0<\alpha<\frac{2m}{(2-m)^2},
$$
and the fixed point is unique [2111.11380].

The training procedure exploits implicit differentiation. For an equilibrium defined by $x^*=T(x^*)$, the gradient satisfies
$$
\frac{d x^*}{d\theta}=\bigl[I-\partial_x T(x^*)\bigr]^{-1}\partial_\theta T(x^*).
$$
Rather than storing all forward iterates, the corresponding linear system is solved by Jacobian iterations, which is the basis for the fixed memory cost associated with DEQ-style backpropagation [2111.11380]. MnM-MOL uses the same implicit-layer viewpoint: inference initializes $x_0$ within a local neighborhood, iterates $x_{k+1}=T(x_k)$ until convergence, or uses Anderson acceleration, and then differentiates through the equilibrium solve [2312.00386].

## 4. ReLU CNN blocks, Lipschitz control, and local monotonicity

In the original MOL design, the nonlinear block is a ReLU CNN embedded in a residual operator $F(x)=x-H(x)$. If $H$ is $L_H$-Lipschitz with $L_H<1-m$, then
$$
\langle x-y,F(x)-F(y)\rangle\ge m\|x-y\|_2^2,
$$
so $F$ is strictly monotone. The two enforcement mechanisms described in the MOL formulation are spectral normalization of each convolutional layer and a Lipschitz regularizer
$$
\ell_{\rm Lip}(x^*)=\max_{\|\epsilon\|\le\delta}\frac{\|H(x^*+\epsilon)-H(x^*)\|_2^2}{\|\epsilon\|_2^2},
$$
which keeps the local Lipschitz estimate below $1-m$ [2111.11380]. The architectural recipe uses a small number of convolution–ReLU layers, no batch-norm, one final residual connection, and the condition $\prod_i \sigma_{\max}(W_i)\le L_H$ with $L_H<1-m$; a typical choice is $m\approx0.1\ldots0.3$ [2111.11380].

MnM-MOL relaxes both the scope and the target of the monotonicity constraint. Instead of constraining the CNN itself, it defines
$$
Q_\theta(x)=\nabla\ell(x)+N_\theta(x)
$$
and requires $Q_\theta$ to be $m$-monotone in a $\delta$-neighborhood of each ground-truth image $x^*$ on the data manifold $\mathcal M$:
$$
\operatorname{Re}\langle Q_\theta(x)-Q_\theta(y),x-y\rangle\ge m\|x-y\|^2,\qquad x,y\in B_\delta(x^*).
$$
Equivalently, with the residual $H_\theta(x)=I-Q_\theta(x)$, a sufficient condition is
$$
\max_{x,y\in B_\delta(x^*)}\|H_\theta(x)-H_\theta(y)\|\le 1-m.
$$
The local Lipschitz constant
$$
L[H_\theta](x^*)=\max_{x,y\in B_\delta(x^*)}\frac{\|H_\theta(x)-H_\theta(y)\|}{\|x-y\|}
$$
is estimated by projected gradient ascent in the ball, and training penalizes any $L[H_\theta](x^*)>1-m$ rather than enforcing the constraint through global spectral norms [2312.00386].

The training objective in MnM-MOL is
$$
C=\|x^*-x_{\rm true}\|^2+\beta\cdot\mathrm{ReLU}\bigl(L[H_\theta](x^*)-(1-m)\bigr)^2.
$$
Under the local monotone condition and initialization $x_0\in B_\delta(x^*)$, the paper states three guarantees: the fixed-point equation $\nabla\ell(x)+N_\theta(x)=0$ has at most one solution in $B_\delta(x^*)$; if $\eta\le 1/\lambda$, then
$$
\|x_k-x^*\|\le (1-\eta m\lambda)^k\|x_0-x^*\|;
$$
and if the measurements are perturbed by $n$ with $\|A^H n\|\le m\delta$, then
$$
\|\Delta\|\le\frac{\|A^H n\|}{m}.
$$
The role of ReLU in this framework is explicit: ReLUs make $N_\theta$ piecewise-linear but generally non-monotone, while the combined operator remains locally monotone [2312.00386].

## 5. Robustness and certification

Robustness is one of the principal motivations for monotone equilibrium models. In MOL, monotonicity implies perturbation bounds on the equilibrium solution. For measurement sets $b$ and $\tilde b$, the small-$\alpha$ limit gives
$$
\|x^*(b)-x^*(\tilde b)\|\le \frac{\lambda}{m}\|b-\tilde b\|,
$$
so larger $m$ implies greater robustness [2111.11380]. MnM-MOL reports that the relaxed local constraints retain robustness to input perturbations similar to MOL and preserve strong robustness to both adversarial and Gaussian noise, while yielding PSNR/SSIM only marginally below unrolled methods like MoDL and keeping the same DEQ memory footprint [2312.00386].

For ReLU-based monDEQs, robustness certification can be made exact at the level of the ReLU graph. The semialgebraic formulation writes the hidden-state equilibrium
$$
z=\mathrm{ReLU}(Wz+Ux_0+u),\qquad F(x_0)=Cz+c,
$$
together with the ReLU graph constraints
$$
z_i(z_i-t_i)=0,\qquad z_i\ge 0,\qquad z_i\ge t_i.
$$
Under the strong monotonicity condition
$$
I_p-W\succeq mI_p,
$$
the equilibrium is unique and the input-output map is single-valued and Lipschitz. This allows certification problems to be posed as polynomial optimization problems and relaxed to SDPs through Lasserre or Shor relaxations [2106.01453].

Three certification tasks are described in that framework. CertMON bounds score gaps over an $\ell_q$ perturbation region to certify robustness of the top-score label. LipMON upper-bounds the Lipschitz constant via a POP derived from Clarke’s generalized Jacobian. EllipMON computes a minimum-volume ellipsoid containing the reachable score set and then checks pairwise label separation [2106.01453]. The reported empirical pattern is norm-dependent: on a monDEQ trained on MNIST with $p_0=784$, $p=87$, $m=20$, and $\epsilon=0.1$ in the $L_2$ norm, all first 100 test points are certified robust, with $99\,\%$ certification by CertMON and $99\,\%$ by EllipMON; for $\epsilon=0.1$ in the $L_\infty$ norm, none are certified and PGD finds attacks on $85\,\%$; reducing $\epsilon$ to $0.01$ in $L_\infty$ restores $\approx99\,\%$ certification [2106.01453]. The same study identifies the main limitation of this line of work as SDP scalability, with interior-point solvers struggling once $p_0+p\gtrsim 5000$ and per-instance costs on the order of tens of minutes [2106.01453].

An alternative robustness-oriented route is the Lipschitz-Bounded Equilibrium Network. There the stronger LMI
$$
2\Lambda-\Lambda W-W^T\Lambda-\tfrac1\gamma W_o^T W_o-\tfrac1\gamma \Lambda U U^T\Lambda\succ0
$$
guarantees that the equilibrium network is well posed and satisfies
$$
\|y(x_1)-y(x_2)\|_2\le \gamma\|x_1-x_2\|_2
$$
for all inputs, with $\gamma$ treated as a prescribed gain [2010.01732].

## 6. Non-Euclidean formulations and physical realizations

A significant extension of the monDEQ idea replaces Euclidean monotonicity with monotonicity under non-Euclidean norms. In the weighted $\ell_\infty$ setting, one studies equilibrium equations such as
$$
0\in D\,x^*+W\,\mathrm{ReLU}(x^*)+b
$$
or, in residual form,
$$
F(x;u)=x-\sigma(Wx+Uu+b),
$$
with monotonicity characterized by log norms and weak pairings rather than inner products [2303.11273], [2106.03194]. For the weighted norm $\|\cdot\|_{\infty,[\eta]^{-1}}$, the matrix measure is
$$
\mu_{\infty,[\eta]^{-1}}(A)=\max_i\Bigl(A_{ii}+\sum_{j\neq i}|A_{ij}|\tfrac{\eta_j}{\eta_i}\Bigr),
$$
and monotonicity can be tested through inequalities involving $\mu_{\infty,[\eta]^{-1}}(D-W)$ [2303.11273].

These formulations produce alternative convergence and Lipschitz results. In the non-Euclidean monotone operator theory, the resolvent $J_{\alpha F}=(I+\alpha F)^{-1}$ is a contraction when $F$ is strongly monotone, and the forward-backward iteration
$$
x^{k+1}=(I+\gamma D)^{-1}\bigl(x^k-\gamma(W\,\mathrm{ReLU}(x^k)+b)\bigr)
$$
satisfies
$$
\|x^{k+1}-x^*\|\le (1-\gamma c)\|x^k-x^*\|
$$
whenever $0<\gamma\le 1/\max_i d_i$ [2303.11273]. In the NEMON formulation, the sufficient well-posedness condition is
$$
\mu_\infty(W)<1,
$$
and the Krasnosel’skiĭ–Mann average iteration
$$
x^{k+1}=(1-\alpha)x^k+\alpha\,\sigma(Wx^k+Uu+b)
$$
converges linearly to the unique equilibrium, with contraction rate
$$
\rho=1-\alpha(1-\mu_\infty(W))<1.
$$
The same framework gives the input-output bound
$$
\Bigl\|\frac{dx^*}{du}\Bigr\|_\infty\le \frac{\|U\|_\infty}{1-\mu_\infty(W)}.
$$
These results are motivated partly by adversarial analysis under pixel-wise perturbations, where weighted $\ell_\infty$ bounds can be less conservative than Euclidean estimates [2106.03194].

The ReLU monDEQ has also been realized directly in analog hardware. A resistor-diode network can be arranged so that its port behavior corresponds to the solution of a ReLU monotone operator equilibrium network, with equilibrium condition
$$
x=\mathrm{ReLU}(Wx+b)
$$
obtained from an inclusion of the form
$$
0\in Hx+\psi(x)+Bu.
$$
In this setting, ReLU is the resolvent of a maximal monotone diode operator, and hardware linearization computes gradients in situ by linearizing each diode at its operating point, injecting parameter-offset sources, and reading the resulting linearized response [2509.13793]. The same work extends the construction to cascades of resistor-diode networks, which can implement feedforward and other asymmetric networks, and introduces a “diode ReLU” induced by a non-ideal Schottky diode model [2509.13793].

Across these variants, the defining feature remains unchanged: a ReLU equilibrium relation is treated not as an unrolled computation but as the zero of a monotone or strongly monotone operator. What changes from one formulation to another is the geometry in which monotonicity is imposed, the strictness of the constraint, and the level at which guarantees are pursued—global well-posedness, local manifold-restricted convergence, certified robustness, or even circuit-level realization.

Source: https://www.emergentmind.com/topics/relu-monotone-operator-equilibrium-network