ReLU Monotone Operator Equilibrium Net

• The paper introduces ReLU Monotone Operator Equilibrium Networks, leveraging ReLU fixed points and monotonicity to ensure unique, stable, and robust equilibrium computation.
• It details the use of implicit differentiation and forward-backward splitting to efficiently train the network while providing convergence guarantees and robustness in inverse problems.
• The framework extends into non-Euclidean formulations and analog implementations, broadening applications in adversarial analysis and hardware-based realizations.

Searching arXiv for relevant papers on ReLU monotone operator equilibrium networks and related formulations. A ReLU Monotone Operator Equilibrium Network is an implicit-depth neural model in which the hidden state is defined by the solution of a ReLU fixed-point equation, typically $z^*=\mathrm{ReLU}(Wz^*+Ux_0+u)$ or $z^*=\mathrm{ReLU}(Wz^*+b)$. The central idea is to replace explicit depth with an equilibrium computation while imposing monotonicity or strong monotonicity on the underlying operator so that the equilibrium is well posed, the solver converges stably, and the model can be differentiated by implicit methods rather than by storing all intermediate iterates (Winston et al., 2020). In inverse problems, the same principle appears in monotone operator learning and in local relaxations such as MnM-MOL, where a ReLU CNN is embedded in a learned reconstruction update and iterated to convergence (Pramanik et al., 2021, John et al., 2023).

1. Origin and scope of the model class

Monotone Operator Equilibrium Networks, or monDEQs, were introduced as a class of implicit-depth models motivated by two limitations of earlier equilibrium formulations: unstable convergence to a solution and lack of guarantees that a solution exists. The monDEQ construction uses the theory of monotone operators to connect equilibrium computation with monotone operator splitting, thereby obtaining efficient solvers with guaranteed, stable convergence and a parameterization that ensures the existence of a unique equilibrium point (Winston et al., 2020). The same general program was later specialized to model-based inverse problems, where monotone operator learning (MOL) recasts reconstruction as a deep equilibrium computation with explicit monotonicity constraints on a learned operator (Pramanik et al., 2021).

Within this literature, the phrase “ReLU Monotone Operator Equilibrium Network” refers to several closely related instantiations. In the basic monDEQ form, the equilibrium equation is a ReLU implicit layer. In MOL, the equilibrium arises from a reconstruction iteration whose nonlinear block is a ReLU CNN. In semialgebraic certification work, the same ReLU fixed point is treated as an exact polynomially constrained object for SDP-based robustness analysis. In non-Euclidean formulations, the equilibrium is analyzed under weighted $\ell_\infty$ or $\ell_1$ norms rather than the Euclidean norm. These variants share the same structural theme: ReLU provides the nonlinear fixed-point relation, while monotonicity controls existence, uniqueness, convergence, and robustness (Chen et al., 2021, Davydov et al., 2023, Jafarpour et al., 2021).

A recurring distinction in the literature is between explicit monotonicity of the learned nonlinear block and monotonicity of a larger composite operator. Global MOL constrains the CNN itself to be monotone, whereas MnM-MOL relaxes this by requiring monotonicity only for the sum of the data-term gradient and the CNN block, and only in a local neighborhood of the image manifold (John et al., 2023). This addresses a common misconception that every useful ReLU equilibrium model must impose a global monotonicity constraint directly on the CNN.

2. Monotone-operator foundations

The operator-theoretic basis of the model is the equivalence between a ReLU fixed point and the zero of a sum of monotone operators. For the equilibrium equation

$z^*=\mathrm{ReLU}(Wz^*+b),$

one may define

$F(z)=(I-W)z-b,\qquad G=\partial f,$

where $f$ is the indicator of the nonnegative orthant, so that $\prox_f^1(x)=\max(x,0)=\mathrm{ReLU}(x)$. Solving the fixed-point equation is then equivalent to solving

$0\in(F+G)(z^*).$

Because $\partial f$ is maximal monotone and $z^*=\mathrm{ReLU}(Wz^*+b)$0 is strongly monotone when $z^*=\mathrm{ReLU}(Wz^*+b)$1 with $z^*=\mathrm{ReLU}(Wz^*+b)$2, the sum has a unique zero, and the equilibrium equation has exactly one solution (Winston et al., 2020).

This same logic underlies the inverse-problem formulation used in MOL. There the target image $z^*=\mathrm{ReLU}(Wz^*+b)$3 is defined by the variational problem

$z^*=\mathrm{ReLU}(Wz^*+b)$4

with first-order optimality condition

$z^*=\mathrm{ReLU}(Wz^*+b)$5

MOL identifies $z^*=\mathrm{ReLU}(Wz^*+b)$6 and implements it as a residual CNN of the form $z^*=\mathrm{ReLU}(Wz^*+b)$7, with monotonicity enforced by a Lipschitz bound on $z^*=\mathrm{ReLU}(Wz^*+b)$8 (Pramanik et al., 2021). MnM-MOL modifies this arrangement by defining the combined operator

$z^*=\mathrm{ReLU}(Wz^*+b)$9

where $\ell_\infty$0 and $\ell_\infty$1 is a feed-forward CNN with ReLU activations. The monotonicity requirement is placed on $\ell_\infty$2 rather than on $\ell_\infty$3 itself (John et al., 2023).

Several parameterizations were proposed to make these conditions trainable. In the original monDEQ construction, strong monotonicity of $\ell_\infty$4 is equivalent to

$\ell_\infty$5

for some trainable $\ell_\infty$6 (Winston et al., 2020). In Lipschitz-Bounded Equilibrium Networks, a weighted monotonicity LMI is enforced by the unconstrained parameterization

$\ell_\infty$7

with diagonal positive-definite $\ell_\infty$8, skew-symmetric $\ell_\infty$9, and arbitrary $\ell_1$0, yielding

$\ell_1$1

Under this condition, the ReLU equilibrium is well posed and globally Lipschitz (Revay et al., 2020).

3. Equilibrium computation and implicit training

The standard solver for a ReLU monDEQ is forward-backward splitting. For the basic equilibrium equation, the update is

$\ell_1$2

If $\ell_1$3 is $\ell_1$4-strongly monotone and $\ell_1$5-Lipschitz, forward-backward converges linearly for any step size

$\ell_1$6

In a dense $\ell_1$7-dimensional model, the per-iteration cost is $\ell_1$8 for the matrix-vector product $\ell_1$9 and $z^*=\mathrm{ReLU}(Wz^*+b),$0 for the ReLU operation (Winston et al., 2020). The same paper also discusses Peaceman–Rachford splitting, which often converges in fewer iterations and requires solving a linear system involving $z^*=\mathrm{ReLU}(Wz^*+b),$1 (Winston et al., 2020).

In MOL, the deep equilibrium layer is a forward-backward reconstruction step rather than a direct ReLU recurrence. The fixed-point map is

$z^*=\mathrm{ReLU}(Wz^*+b),$2

and with the residual form $z^*=\mathrm{ReLU}(Wz^*+b),$3 this becomes

$z^*=\mathrm{ReLU}(Wz^*+b),$4

If $z^*=\mathrm{ReLU}(Wz^*+b),$5 is $z^*=\mathrm{ReLU}(Wz^*+b),$6-monotone and $z^*=\mathrm{ReLU}(Wz^*+b),$7, the iteration converges for

$z^*=\mathrm{ReLU}(Wz^*+b),$8

and the fixed point is unique (Pramanik et al., 2021).

The training procedure exploits implicit differentiation. For an equilibrium defined by $z^*=\mathrm{ReLU}(Wz^*+b),$9, the gradient satisfies

$F(z)=(I-W)z-b,\qquad G=\partial f,$0

Rather than storing all forward iterates, the corresponding linear system is solved by Jacobian iterations, which is the basis for the fixed memory cost associated with DEQ-style backpropagation (Pramanik et al., 2021). MnM-MOL uses the same implicit-layer viewpoint: inference initializes $F(z)=(I-W)z-b,\qquad G=\partial f,$1 within a local neighborhood, iterates $F(z)=(I-W)z-b,\qquad G=\partial f,$2 until convergence, or uses Anderson acceleration, and then differentiates through the equilibrium solve (John et al., 2023).

4. ReLU CNN blocks, Lipschitz control, and local monotonicity

In the original MOL design, the nonlinear block is a ReLU CNN embedded in a residual operator $F(z)=(I-W)z-b,\qquad G=\partial f,$3. If $F(z)=(I-W)z-b,\qquad G=\partial f,$4 is $F(z)=(I-W)z-b,\qquad G=\partial f,$5-Lipschitz with $F(z)=(I-W)z-b,\qquad G=\partial f,$6, then

$F(z)=(I-W)z-b,\qquad G=\partial f,$7

so $F(z)=(I-W)z-b,\qquad G=\partial f,$8 is strictly monotone. The two enforcement mechanisms described in the MOL formulation are spectral normalization of each convolutional layer and a Lipschitz regularizer

$F(z)=(I-W)z-b,\qquad G=\partial f,$9

which keeps the local Lipschitz estimate below $f$0 (Pramanik et al., 2021). The architectural recipe uses a small number of convolution–ReLU layers, no batch-norm, one final residual connection, and the condition $f$1 with $f$2; a typical choice is $f$3 (Pramanik et al., 2021).

MnM-MOL relaxes both the scope and the target of the monotonicity constraint. Instead of constraining the CNN itself, it defines

$f$4

and requires $f$5 to be $f$6-monotone in a $f$7-neighborhood of each ground-truth image $f$8 on the data manifold $f$9:

$\prox_f^1(x)=\max(x,0)=\mathrm{ReLU}(x)$0

Equivalently, with the residual $\prox_f^1(x)=\max(x,0)=\mathrm{ReLU}(x)$1, a sufficient condition is

$\prox_f^1(x)=\max(x,0)=\mathrm{ReLU}(x)$2

The local Lipschitz constant

$\prox_f^1(x)=\max(x,0)=\mathrm{ReLU}(x)$3

is estimated by projected gradient ascent in the ball, and training penalizes any $\prox_f^1(x)=\max(x,0)=\mathrm{ReLU}(x)$4 rather than enforcing the constraint through global spectral norms (John et al., 2023).

The training objective in MnM-MOL is

$\prox_f^1(x)=\max(x,0)=\mathrm{ReLU}(x)$5

Under the local monotone condition and initialization $\prox_f^1(x)=\max(x,0)=\mathrm{ReLU}(x)$6, the paper states three guarantees: the fixed-point equation $\prox_f^1(x)=\max(x,0)=\mathrm{ReLU}(x)$7 has at most one solution in $\prox_f^1(x)=\max(x,0)=\mathrm{ReLU}(x)$8; if $\prox_f^1(x)=\max(x,0)=\mathrm{ReLU}(x)$9, then

$0\in(F+G)(z^*).$0

and if the measurements are perturbed by $0\in(F+G)(z^*).$1 with $0\in(F+G)(z^*).$2, then

$0\in(F+G)(z^*).$3

The role of ReLU in this framework is explicit: ReLUs make $0\in(F+G)(z^*).$4 piecewise-linear but generally non-monotone, while the combined operator remains locally monotone (John et al., 2023).

5. Robustness and certification

Robustness is one of the principal motivations for monotone equilibrium models. In MOL, monotonicity implies perturbation bounds on the equilibrium solution. For measurement sets $0\in(F+G)(z^*).$5 and $0\in(F+G)(z^*).$6, the small-$0\in(F+G)(z^*).$7 limit gives

$0\in(F+G)(z^*).$8

so larger $0\in(F+G)(z^*).$9 implies greater robustness (Pramanik et al., 2021). MnM-MOL reports that the relaxed local constraints retain robustness to input perturbations similar to MOL and preserve strong robustness to both adversarial and Gaussian noise, while yielding PSNR/SSIM only marginally below unrolled methods like MoDL and keeping the same DEQ memory footprint (John et al., 2023).

For ReLU-based monDEQs, robustness certification can be made exact at the level of the ReLU graph. The semialgebraic formulation writes the hidden-state equilibrium

$\partial f$0

together with the ReLU graph constraints

$\partial f$1

Under the strong monotonicity condition

$\partial f$2

the equilibrium is unique and the input-output map is single-valued and Lipschitz. This allows certification problems to be posed as polynomial optimization problems and relaxed to SDPs through Lasserre or Shor relaxations (Chen et al., 2021).

Three certification tasks are described in that framework. CertMON bounds score gaps over an $\partial f$3 perturbation region to certify robustness of the top-score label. LipMON upper-bounds the Lipschitz constant via a POP derived from Clarke’s generalized Jacobian. EllipMON computes a minimum-volume ellipsoid containing the reachable score set and then checks pairwise label separation (Chen et al., 2021). The reported empirical pattern is norm-dependent: on a monDEQ trained on MNIST with $\partial f$4, $\partial f$5, $\partial f$6, and $\partial f$7 in the $\partial f$8 norm, all first 100 test points are certified robust, with $\partial f$9 certification by CertMON and $z^*=\mathrm{ReLU}(Wz^*+b)$00 by EllipMON; for $z^*=\mathrm{ReLU}(Wz^*+b)$01 in the $z^*=\mathrm{ReLU}(Wz^*+b)$02 norm, none are certified and PGD finds attacks on $z^*=\mathrm{ReLU}(Wz^*+b)$03; reducing $z^*=\mathrm{ReLU}(Wz^*+b)$04 to $z^*=\mathrm{ReLU}(Wz^*+b)$05 in $z^*=\mathrm{ReLU}(Wz^*+b)$06 restores $z^*=\mathrm{ReLU}(Wz^*+b)$07 certification (Chen et al., 2021). The same study identifies the main limitation of this line of work as SDP scalability, with interior-point solvers struggling once $z^*=\mathrm{ReLU}(Wz^*+b)$08 and per-instance costs on the order of tens of minutes (Chen et al., 2021).

An alternative robustness-oriented route is the Lipschitz-Bounded Equilibrium Network. There the stronger LMI

$z^*=\mathrm{ReLU}(Wz^*+b)$09

guarantees that the equilibrium network is well posed and satisfies

$z^*=\mathrm{ReLU}(Wz^*+b)$10

for all inputs, with $z^*=\mathrm{ReLU}(Wz^*+b)$11 treated as a prescribed gain (Revay et al., 2020).

6. Non-Euclidean formulations and physical realizations

A significant extension of the monDEQ idea replaces Euclidean monotonicity with monotonicity under non-Euclidean norms. In the weighted $z^*=\mathrm{ReLU}(Wz^*+b)$12 setting, one studies equilibrium equations such as

$z^*=\mathrm{ReLU}(Wz^*+b)$13

or, in residual form,

$z^*=\mathrm{ReLU}(Wz^*+b)$14

with monotonicity characterized by log norms and weak pairings rather than inner products (Davydov et al., 2023, Jafarpour et al., 2021). For the weighted norm $z^*=\mathrm{ReLU}(Wz^*+b)$15, the matrix measure is

$z^*=\mathrm{ReLU}(Wz^*+b)$16

and monotonicity can be tested through inequalities involving $z^*=\mathrm{ReLU}(Wz^*+b)$17 (Davydov et al., 2023).

These formulations produce alternative convergence and Lipschitz results. In the non-Euclidean monotone operator theory, the resolvent $z^*=\mathrm{ReLU}(Wz^*+b)$18 is a contraction when $z^*=\mathrm{ReLU}(Wz^*+b)$19 is strongly monotone, and the forward-backward iteration

$z^*=\mathrm{ReLU}(Wz^*+b)$20

satisfies

$z^*=\mathrm{ReLU}(Wz^*+b)$21

whenever $z^*=\mathrm{ReLU}(Wz^*+b)$22 (Davydov et al., 2023). In the NEMON formulation, the sufficient well-posedness condition is

$z^*=\mathrm{ReLU}(Wz^*+b)$23

and the Krasnosel’skiĭ–Mann average iteration

$z^*=\mathrm{ReLU}(Wz^*+b)$24

converges linearly to the unique equilibrium, with contraction rate

$z^*=\mathrm{ReLU}(Wz^*+b)$25

The same framework gives the input-output bound

$z^*=\mathrm{ReLU}(Wz^*+b)$26

These results are motivated partly by adversarial analysis under pixel-wise perturbations, where weighted $z^*=\mathrm{ReLU}(Wz^*+b)$27 bounds can be less conservative than Euclidean estimates (Jafarpour et al., 2021).

The ReLU monDEQ has also been realized directly in analog hardware. A resistor-diode network can be arranged so that its port behavior corresponds to the solution of a ReLU monotone operator equilibrium network, with equilibrium condition

$z^*=\mathrm{ReLU}(Wz^*+b)$28

obtained from an inclusion of the form

$z^*=\mathrm{ReLU}(Wz^*+b)$29

In this setting, ReLU is the resolvent of a maximal monotone diode operator, and hardware linearization computes gradients in situ by linearizing each diode at its operating point, injecting parameter-offset sources, and reading the resulting linearized response (Chaffey, 17 Sep 2025). The same work extends the construction to cascades of resistor-diode networks, which can implement feedforward and other asymmetric networks, and introduces a “diode ReLU” induced by a non-ideal Schottky diode model (Chaffey, 17 Sep 2025).

Across these variants, the defining feature remains unchanged: a ReLU equilibrium relation is treated not as an unrolled computation but as the zero of a monotone or strongly monotone operator. What changes from one formulation to another is the geometry in which monotonicity is imposed, the strictness of the constraint, and the level at which guarantees are pursued—global well-posedness, local manifold-restricted convergence, certified robustness, or even circuit-level realization.