DEQN: Implicit Deep Equilibrium Networks

Updated 23 June 2026

Deep Equilibrium Nets are implicit models that define outputs as fixed points of nonlinear transformations, enabling constant-memory depth.
They utilize efficient root-finding algorithms (e.g., Broyden's method) and implicit differentiation to perform scalable training.
Architectural variants like MDEQ and Lipschitz DEQN improve convergence guarantees, adversarial robustness, and inference efficiency.

Deep Equilibrium Networks (DEQN) are a class of implicit deep learning architectures in which the output is characterized as the fixed point of a nonlinear transformation, rather than the result of stacking an explicit set of layers. This approach allows for a decoupling of the forward and backward passes, offering significant memory efficiency—constant in effective depth—by leveraging the implicit function theorem for backpropagation. DEQNs have been developed to match or outperform explicit deep networks on language modeling, computer vision, and structured prediction, and have inspired a rich set of theoretical, algorithmic, and applied research directions.

1. Mathematical Formulation and Fixed-Point Inference

A DEQN layer parameterized by $\theta \in \mathbb{R}^{d_\theta}$ maps an input $x \in \mathbb{R}^{d_x}$ and hidden state $z \in \mathbb{R}^{d_z}$ to an implicit equilibrium via the fixed-point equation:

$z^* = f_\theta(x, z^*)$

Alternatively, this can be stated as a root-finding problem:

$g_\theta(x, z^*) = f_\theta(x, z^*) - z^* = 0$

Forward inference seeks $z^*$ by invoking a black-box root-finding algorithm. Broyden's method is common, maintaining a low-rank approximation to the inverse Jacobian of $g_\theta$ :

Initialization: $z_0 \leftarrow 0;\ B_0^{-1} \leftarrow -I$
Iteration:
1. $g_t \leftarrow g_\theta(x, z_t)$
2. $z_{t+1} \leftarrow z_t - B_t^{-1} g_t$
3. Update $x \in \mathbb{R}^{d_x}$ 0 via Sherman–Morrison to maintain a low-rank inverse Jacobian approximation.
Terminate when $x \in \mathbb{R}^{d_x}$ 1 or $x \in \mathbb{R}^{d_x}$ 2; set $x \in \mathbb{R}^{d_x}$ 3.

This approach generalizes across domains and retains $x \in \mathbb{R}^{d_x}$ 4 memory with respect to "depth" since only the last state and Jacobian estimate are kept (Bai et al., 2019, Bai et al., 2020).

2. Backpropagation via Implicit Differentiation

Once the equilibrium $x \in \mathbb{R}^{d_x}$ 5 is computed, gradients can be obtained without unrolling the entire fixed-point iteration. Using the implicit function theorem, for a loss $x \in \mathbb{R}^{d_x}$ 6 with $x \in \mathbb{R}^{d_x}$ 7, the gradient with respect to parameters is:

$x \in \mathbb{R}^{d_x}$ 8

In practice, direct inversion of $x \in \mathbb{R}^{d_x}$ 9 is avoided due to $z \in \mathbb{R}^{d_z}$ 0 cost. Instead, vector–Jacobian solves (e.g., using another Broyden or Anderson solver) are employed for efficient computation. Recent work proposes reusing the inverse Jacobian approximation $z \in \mathbb{R}^{d_z}$ 1 from the forward pass, enabling a constant-time and constant-memory backward pass with minimal loss in gradient quality (Nguyen et al., 2023).

Method	Backward Time	Memory (w.r.t. depth)
Implicit solve	$z \in \mathbb{R}^{d_z}$ 2	$z \in \mathbb{R}^{d_z}$ 3
GDEQ (reuse $z \in \mathbb{R}^{d_z}$ 4)	$z \in \mathbb{R}^{d_z}$ 5	$z \in \mathbb{R}^{d_z}$ 6

Empirical studies demonstrate cosine similarity $z \in \mathbb{R}^{d_z}$ 7 between GDEQ and exact gradients, and consistent $z \in \mathbb{R}^{d_z}$ 8 speedups in training (Nguyen et al., 2023).

3. Algorithmic and Architectural Variants

3.1 Multiscale Deep Equilibrium Networks (MDEQ)

MDEQs extend the basic DEQN by solving for a joint equilibrium over multi-resolution feature vectors:

$z \in \mathbb{R}^{d_z}$ 9

Each branch processes its scale with a weight-tied block, and fusion occurs across scales with up/downsampling operators. Joint equilibrium is found via quasi-Newton solvers (e.g., Broyden) (Bai et al., 2020).

3.2 Lipschitz and Monotone DEQNs

Restricting $z^* = f_\theta(x, z^*)$ 0 to be Lipschitz with constant $z^* = f_\theta(x, z^*)$ 1 (e.g., by weight-projection and formulating all operations with bounded constants) guarantees unique fixed points and linear convergence (Banach theorem). Monotone-operator theory parametrizes $z^* = f_\theta(x, z^*)$ 2 to ensure $z^* = f_\theta(x, z^*)$ 3 and applies operator-splitting algorithms for provable stability and efficiency (Sato et al., 3 Feb 2026, Winston et al., 2020).

Positive Concave DEQ (pcDEQ) models enforce $z^* = f_\theta(x, z^*)$ 4 and concave nonnegative activations, utilizing nonlinear Perron–Frobenius theory. For these, the spectral radius of the asymptotic map $z^* = f_\theta(x, z^*)$ 5, ensuring existence, uniqueness, and geometric convergence of the iteration $z^* = f_\theta(x, z^*)$ 6 (Gabor et al., 2024).

3.3 Distributional and Hybrid Architectures

Distributional DEQs (DDEQs) promote inputs and latents to measures, with equilibrium in Wasserstein space. The fixed point is found via Wasserstein gradient flows, supporting permutation-invariance for sets and point clouds, and leveraging measure-to-measure maps (Geuter et al., 3 Mar 2025).

Hierarchical and joint-inference/inference–input optimization (JIIO) DEQNs embed bilevel optimization directly into augmented fixed-point systems, dramatically reducing the cost of inner-optimization loops (generative modeling, adversarial training, meta-learning) (Gurumurthy et al., 2021).

4. Theory of Representation, Kernel Dynamics, and Training

4.1 Neural Tangent Kernels and Spectral Theory

The NTK for infinite-depth DEQ is deterministic even as both width and depth tend to infinity; it can be computed by root-finding in a one-dimensional fixed-point recurrence. The limiting kernel depends only on activation, variance hyperparameters, and input covariance via a nonlinear system, and matches certain shallow explicit architectures' kernels under high-dimensional Gaussian mixtures. This determines generalization behavior and ensures DEQs can be kernel-regression–analyzable (Feng et al., 2023, Ling et al., 2024).

4.2 Neural Collapse and Representation Geometry

DEQNs exhibit classic Neural Collapse (NC) properties under balanced data: class mean features form a simplex equiangular tight frame, with self-duality between features and classifier weights. Under class imbalance, DEQNs yield features that are closer to the ideal simplex structure than explicit models, showing advantages in minority class alignment and representation robustness (Sun et al., 2024).

4.3 Initialization and Stability

Higher order initialization statistics critically affect stability of both the forward fixed-point and backward solve. Orthogonal and symmetric (GOE) initializations enable higher weight variances before divergence, with sharper spectral boundaries and reduced variance in equilibrium states. Prescriptions for initialization directly target these trade-offs, especially in large-scale settings (Agarwala et al., 2022).

4.4 Regularization and Convergence Acceleration

Jacobian regularization imposes a Frobenius-norm penalty on $z^* = f_\theta(x, z^*)$ 7, reducing its spectral radius and thus enforcing contractivity, accelerating both forward and backward solver convergence. Empirically, this reduces the number of function evaluations (NFEs) by $z^* = f_\theta(x, z^*)$ 8– $z^* = f_\theta(x, z^*)$ 9 and narrows the train/inference time gap to explicit models with minimal accuracy loss. Explicit contraction in the architecture (e.g., via Lipschitz MDEQ) or loss (e.g., Jacobian reg) provides theoretical and empirical control over stability/speed/expressivity trade-offs (Bai et al., 2021, Sato et al., 3 Feb 2026).

5. Adversarial Robustness and Neural Dynamics

DEQNs exhibit adversarial robustness competitive with explicit deep networks under strong white-box attacks, particularly when training and evaluation include mechanism to align forward/backward solver states and attack gradients. Methods include:

Entropy reduction via input updates along DEQ trajectories, linked to reduced local expansion via Lyapunov exponents, directly leveraging DEQ's structure as a discrete-time dynamical system (Yang et al., 2023).
Training on losses sampled at random intermediate solver states, enforcing robustness along the entire input-to-equilibrium trajectory.
Jacobian regularization further stabilizes the trajectory and mitigates gradient obfuscation; intermediate state ensembling and early exit further boost robustness at practically no cost (Yang et al., 2023).

6. Acceleration and Inference Efficiency

Standard DEQ inference incurs high latency due to expensive iterative fixed-point solves. Recent advances employ consistency distillation—training a student network (C-DEQ) to map intermediate trajectory states (from Anderson-accelerated ODE paths) to the true equilibrium in one (or few) steps. C-DEQ achieves up to $g_\theta(x, z^*) = f_\theta(x, z^*) - z^* = 0$ 0– $g_\theta(x, z^*) = f_\theta(x, z^*) - z^* = 0$ 1 accuracy gains over vanilla DEQ at the same few-step inference budget, retains the memory efficiency of implicit models, and supports multi-step trade-off control (Lin et al., 3 Feb 2026).

Model	Pre-training	Test Accuracy (CIFAR-10)	Speedup (vs. exact)
Implicit DEQ	yes/no	92.29 / 91.64	1×
GDEQ	yes/no	93.08 / 92.01	2.16×
Lipschitz MDEQ	no (L=0.03)	90.47	4.9× (forward)
C-DEQ	NFE=1	47.90 (WT103)	1.8× (faster)

Further research investigates ODE-inspired continuous–time acceleration, trust-region solvers, or unrolled intermediate regulations at both training and test time. For task generality, all these methods preserve $g_\theta(x, z^*) = f_\theta(x, z^*) - z^* = 0$ 2-memory backprop via the implicit function theorem.

7. Practical Guidelines, Limitations, and Extensions

To implement and deploy DEQNs:

Ensure the contraction condition (on $g_\theta(x, z^*) = f_\theta(x, z^*) - z^* = 0$ 3 or its Jacobian) via spectral norm enforcement, initialization, or regularization.
Use robust root solvers (Broyden/Anderson) with low-rank Jacobian histories.
When targeting adversarial robustness or rapid inference, combine Jacobian regularization, input trajectory regulation, and consistency distillation.
For non-sequential or set-structured data, adopt DDEQs via measure-valued latent spaces and Wasserstein gradient flows.
Create explicit-equivalent shallow networks for speed where DEQ is provably redundant (e.g., under GMM inputs, kernel matchings).
Monitor fixed-point success rates and abort/decay learning rates if instabilities are detected late in training (Agarwala et al., 2022, Nguyen et al., 2023, Lin et al., 3 Feb 2026).

Open challenges include spurious fixed points in $g_\theta(x, z^*) = f_\theta(x, z^*) - z^* = 0$ 4 settings, fine-grained analysis of the trade-off between stability and model capacity, more efficient Wasserstein solvers for DDEQs, and scalable certified robust DEQN training for large vision/language tasks.