Input Convex Recurrent Neural Networks

• ICRNNs are specialized recurrent neural networks that guarantee convexity in their input variables using nonnegative weights and convex, non-decreasing activations.
• They integrate seamlessly with convex optimization frameworks such as model predictive control, offering efficient and tractable solutions for dynamic systems.
• Recent extensions with Lipschitz constraints (ICLRNNs) enhance robustness and stability, making them effective for noise-resistant and real-time control applications.

Input Convex Recurrent Neural Networks (ICRNNs) are a class of neural architectures designed to address the need for tractable, globally convex models of dynamical systems amenable to efficient optimization in control and prediction tasks. By constraining the functional form of the network, ICRNNs guarantee convexity with respect to their input variables, enabling direct integration with convex optimization frameworks such as model predictive control (MPC). Recent developments have extended ICRNNs by incorporating explicit Lipschitz continuity constraints—yielding Input Convex Lipschitz RNNs (ICLRNNs)—to further enhance robustness without sacrificing computational efficiency. These models have demonstrated empirical advantages in both process control and sequence prediction domains, particularly where rapid, stable, and noise-resistant solutions are required (Wang et al., 2024, Chen et al., 2018).

1. Mathematical Foundation and Convexity Guarantees

ICRNNs restrict the parameterization and activation functions of standard recurrent neural networks to ensure convexity in the input sequence. The generic ICRNN recurrence adopts the form: $h_t = g_1(W^x_t x_t + W^h_t h_{t-1} + b^h_t)$

$y_t = g_2(W^y_t h_t + b^y_t)$

where all weight matrices $W_i$ are elementwise non-negative ($W_i \ge 0$), and all activation functions $g_i$ are convex and non-decreasing (e.g., ReLU, linear, softmax) (Wang et al., 2024).

For the feedforward ICNN, the $K$-layer form is: $$f_{\text{ICNN}}(u) = z_K,\quad z_0 = u,\quad z_k = \sigma(W_k z_{k-1} + D_k u + b_k),\ k=1,\dots,K$$ where $W_k, D_k \ge 0$ and $\sigma$ is convex and non-decreasing (Chen et al., 2018).

Convexity in the unrolled recurrent formulation is established by induction: non-negative weighted sums and the compositional closure property for convex non-decreasing functions guarantee that the network output is convex in all input variables (Wang et al., 2024). The input duplication scheme ($\hat{u}_t = [u_t; v_t]$ with $y_t = g_2(W^y_t h_t + b^y_t)$0) ensures the network can model both positive and negative input dependencies while preserving convexity (Chen et al., 2018).

2. Lipschitz Constraints and Robustness Enhancement

While input convexity yields tractability for optimization, it does not inherently control the network's sensitivity to input perturbations. Enforcing a global Lipschitz continuity constraint ensures that for all $y_t = g_2(W^y_t h_t + b^y_t)$1,

$y_t = g_2(W^y_t h_t + b^y_t)$2

where $y_t = g_2(W^y_t h_t + b^y_t)$3 is the Lipschitz constant (Wang et al., 2024). In ICLRNNs, a two-stage procedure is applied after each gradient update:

• Spectral normalization (via power iteration) constrains the spectral norm $y_t = g_2(W^y_t h_t + b^y_t)$4 for each $y_t = g_2(W^y_t h_t + b^y_t)$5.
• The Björck orthogonalization algorithm is applied to guarantee all singular values $y_t = g_2(W^y_t h_t + b^y_t)$6.
• Nonnegative clipping replaces any negative entries in $y_t = g_2(W^y_t h_t + b^y_t)$7 with zero.

Convex, non-decreasing activations with Lipschitz constant $y_t = g_2(W^y_t h_t + b^y_t)$8 (e.g., ReLU, linear) further bound the network's global Lipschitz constant. Compatibility between nonnegativity and spectral constraints is theoretically justified: replacing negative entries in a $y_t = g_2(W^y_t h_t + b^y_t)$9 matrix increases $W_i$0 at most to $W_i$1, preserving the $W_i$2-Lipschitz property (Wang et al., 2024). This constraint regime not only bounds the input-output sensitivity of the model, enhancing robustness to exogenous noise, but also mitigates the risk of exploding gradients in deep or large-scale recurrent networks.

3. Training Protocols and Practical Implementation

ICRNNs and ICLRNNs are trained using standard stochastic gradient descent methods (e.g., Adam optimizer), with losses appropriate to the prediction task (mean squared error for regression, cross-entropy for classification) (Wang et al., 2024, Chen et al., 2018). Input and output data are typically normalized to $W_i$3 per coordinate, and sequence data are windowed to appropriate memory lengths. Network parameters are constrained to be non-negative via per-step projections ($W_i$4) after each update.

For ICLRNNs, the constraint enforcement pipeline—spectral normalization, Björck orthogonalization, then non-negative clipping—is applied after each weight update. No auxiliary variables or slack terms are needed, resulting in minimal computational overhead. Empirically, the time per training epoch is comparable to unconstrained RNNs. For ICRNNs, negative input modeling is achieved by concatenating duplicated negative inputs ($W_i$5) at each timestep (Chen et al., 2018).

Backpropagation through time is used for gradient computation. The dominant computational cost remains matrix multiplication; the additional constraint operations (spectral normalization, clipping) are elementwise or involve power iteration, contributing negligibly in total GPU or CPU time (Wang et al., 2024).

4. Integration with Convex Optimal Control and Engineering Applications

A primary application of ICRNNs is their direct integration with convex model predictive control frameworks. Because the network output is convex in the input sequence, the MPC problem: $W_i$6 subject to state dynamics $W_i$7, and input constraints, becomes a globally tractable convex program if the stage cost $W_i$8 is convex and network constraints are enforced (Chen et al., 2018). Standard solvers (CVX, CVXPY, OSQP) or projected gradient methods can solve the resulting program efficiently.

Empirical studies demonstrate that ICRNN-based MPC achieves significant reductions in solve-time and improved sample efficiency compared to shooting methods or black-box RNNs. In MuJoCo locomotion benchmarks, ICNN-based MPC obtained 10–20% higher returns while using 5–10$W_i$9 less wall-clock time relative to traditional model-based reinforcement learning controllers (Chen et al., 2018). In a commercial building HVAC scenario, a recurrent ICNN enabled energy reductions up to 23.3% over unconstrained baselines, with smooth, non-oscillatory controls and no instability observed (Chen et al., 2018).

For ICLRNNs, case studies include nonlinear chemical process modeling and solar irradiance forecasting. In exothermic CSTR control, ICLRNNs achieved test MSE $W_i \ge 0$0 with fastest convergence in Lyapunov-MPC, and CPU solve times of approximately 20 minutes per study, outperforming nonconvex alternatives. In PV system forecasting, ICLRNNs delivered lowest computational footprint (FLOPs: 399,362), robustly tracking sudden irradiance changes and enabling deployment on resource-constrained edge hardware (Wang et al., 2024).

5. Empirical Performance: Efficiency and Robustness

Empirical benchmarks across several domains illustrate the architectural tradeoffs:

Architecture	CSTR FLOPs	PV Forecast FLOPs	Robustness under Noise	Gradient Stability
RNN	406,548	399,362	Moderate	Potentially unstable
LSTM	1,597,460	1,596,418	Moderate	Improved
ICRNN	1,204,233	1,195,010	Better	Susceptible to explosion
LRNN	2,505,748	2,498,562	High	Stable
ICLRNN	406,548	399,362	Highest	Stable (bounded gradients)

ICLRNNs offer the lowest computational cost while providing maximal robustness to input noise and gradient explosion. In comparative evaluations, ICLRNNs sustained the smallest degradation in test MSE as input SNR was reduced and exhibited more stable prediction and closed-loop control behaviors than unconstrained or even purely convex-only RNN models (Wang et al., 2024).

6. Theoretical Guarantees and Limitations

Theoretical results formally establish that an RNN is convex, non-decreasing, and $W_i \ge 0$1-Lipschitz in its inputs if and only if all weights are non-negative, spectrally normalized, and all activations are convex, non-decreasing, and 1-Lipschitz (Theorem 1 in (Wang et al., 2024)). The composition closure properties and properties of spectral normalization under nonnegative projections are rigorously established (Propositions 1–3).

A limitation noted is that pure ICRNNs—without Lipschitz constraints—may suffer from gradient explosion when scaled to larger networks; adding Lipschitz normalization as in ICLRNNs mitigates this issue. Another practical consideration is the representational capacity reduction imposed by non-negativity; the input duplication trick ($W_i \ge 0$2) partially compensates for this, but may increase network width or training complexity (Chen et al., 2018).

7. Summary and Outlook

Input Convex Recurrent Neural Networks and their Lipschitz-constrained extensions provide neural architectures with provable convexity and robustness suitable for applications requiring reliable, real-time optimization and prediction. Their integration into MPC and real-world engineering workflows offers both theoretical tractability and empirical gains in computational efficiency, stability, and robustness under uncertainty (Wang et al., 2024, Chen et al., 2018). Future extensions may include further architectural innovations to enhance expressivity without compromising convexity, as well as broader deployment in safety-critical and resource-constrained settings.