---
title: Input Convex Neural Networks
url: https://www.emergentmind.com/topics/input-convex-neural-networks-icnns
type: topic
---

# Input Convex Neural Networks

Input Convex Neural Networks (ICNNs) are a specialized class of feedforward neural architectures that enforce convexity with respect to some or all of their inputs by design. Developed to bridge the expressivity of deep networks with the tractability and regularity of convex optimization, ICNNs have become foundational in modern data-driven optimization, control, inverse problems, and physics-inspired modeling. The defining feature is architectural and parametric constraints that guarantee that the output is a convex function of designated input variables, ensuring global optima for input inference and enabling their integration as function surrogates in structured convex programs.

## 1. Formal Definition and Architectural Principles

Let $f_\theta\colon \mathbb{R}^d \rightarrow \mathbb{R}$ be a scalar-valued function modeled by a feed-forward network parameterized by $\theta$. $f_\theta$ is convex in $x$ if, for any $x, y \in \mathbb{R}^d$ and $t \in [0,1]$,
$$
f_\theta(t x + (1-t) y) \leq t f_\theta(x) + (1-t) f_\theta(y).
$$
An Input Convex Neural Network (ICNN) guarantees this property via layerwise constraints. For an $L$-layer ICNN with input $x$,
\[
\begin{aligned}
z_0 &= x \\
z_1 &= \sigma_0(W_0 x + b_0) \\
z_{k+1} &= \sigma_k(W_k z_k + U_k x + b_k), \quad k=1,\ldots,L-1 \\
f(x) &= z_{L+1} = \sigma_L(W_L z_L + U_L x + b_L)
\end{aligned}
\]
where

- $W_k \geq 0$ (entrywise) for $k \geq 1$ (and optionally for $k=0$),
- $U_k \geq 0$ (entrywise) for $k \geq 0$,
- each $\sigma_k$ is convex and nondecreasing (e.g., ReLU, leaky-ReLU, softplus).
  
Convexity follows by induction: each $z_k$ is a convex function of $x$, and compositions with convex, nondecreasing activations preserve convexity. The same approach extends to partially input-convex networks (PICNNs), where convexity is enforced only with respect to a subset of inputs [1609.07152], [2401.07121].

## 2. Parameter Constraints and Convexity Guarantees

Core to the ICNN is the parametric structure:

- Nonnegativity in $W_k$ and $U_k$ ensures that each hidden layer’s preactivation is a nonnegative weighted sum of convex functions, preserving convexity.
- Convex, nondecreasing activations (e.g. ReLU, leaky-ReLU, softplus, ELU) are required. These may be chosen for additional regularization properties or smoothness [1609.07152], [2401.07121].

Parameter enforcement is typically achieved via two methods: (a) post-gradient-step clipping ($W_k \gets \max(W_k,0)$), or (b) nonnegative parametrization (e.g. $W_k = \exp(A_k - \epsilon)$, where $A_k$ is unconstrained) [2401.07121].

Where strong convexity is desired (e.g., in modeling mirror potentials), a positive definite quadratic term may be added [2206.06733].

## 3. Theoretical Expressivity and Limitations

ICNNs are universal approximators of continuous convex functions. Any Lipschitz convex function on a compact domain can be approximated arbitrarily well by an ICNN with sufficient width and depth, provided nonnegative weights and convex, nondecreasing activations [1609.07152], [1805.11835].

However, ICNNs represent only a subset of convex functions implementable with ReLU multi-layer perceptrons (MLPs): for 1-hidden-layer ReLU networks, ICNNs are sufficient to realize any convex continuous piecewise-linear (CPWL) function. For depth $\geq 3$, there exist convex ReLU networks not reparameterizable into the ICNN form due to structural constraints—ICNNs cover only a small subset of convex ReLU networks in this regime [2501.03017].

ICNNs cannot faithfully approximate nonconvex functions; the minimal sup-norm error can be significant for highly nonconvex targets. They also cannot capture some system structures (e.g., linear time delays) without input expansion or lifting [2103.17231].

## 4. Training, Initialization, and Inference

Training follows standard supervised paradigms (e.g., mean-squared error minimization for regression), employing stochastic optimization (Adam, SGD). After each update, nonnegativity constraints are re-imposed as above [2401.07121], [1805.11835]. For structured learning tasks (e.g., max-margin structured prediction, Q-function learning), the architecture enables convex inference and loss computation [1609.07152].

Initialization of nonnegative weights deviates from classical methods (e.g. Xavier, He) due to nonzero mean and strictly positive entries. Recent principled schemes adjust mean, variance, and correlations to control signal propagation (e.g. centering preactivation means, using nonnegative log-normal samples), resulting in accelerated learning and obviating the need for skip connections under proper statistical calibration [2312.12474].

Inference exploits convexity: input optimization becomes a convex program, solvable by projected gradient, bundle methods, or converted to linear/quadratic programs as needed [1609.07152], [2410.00796]. For partially input-convex architectures, convexity guarantees tractable optimization over the targeted variables, even when nonconvexity remains in other coordinates [2505.11047].

## 5. Integration in Optimization, Control, and Model Reduction

ICNNs are widely used to learn convex surrogates for control, hybrid optimization, and inverse problems, particularly when global optimality or robust feasibility is required. For instance:

- In constrained parametric optimization, ICNNs coupled with augmented Lagrangian methods can learn solution mappings that converge to $\epsilon$-KKT points, empirically rivaling classic QP and ACOPF solvers in gap (0.15–2%) and runtime ($<$1 ms) [2505.04037].
- In energy systems and contingency screening, ICNNs enable the data-driven characterization of polyhedral feasible sets, with convex programs enforcing reliability constraints (e.g., zero false negative guarantees in $N-k$ screening) [2410.00796], [2505.11047].
- In model predictive control (MPC), ICNNs enable convex optimization over control inputs, facilitating fast and globally optimal receding-horizon control in physical systems (e.g. MuJoCo and large-scale building HVACs), with up to 5$\times$ less compute time compared to nonconvex shooting methods and >10% higher empirical reward or energy savings [1805.11835], [2603.22095].
- Recent architectures combine input convexity with modern sequence models—e.g. the Input Convex Encoder-only Transformer (IC-EoT) maintains input convexity across temporal horizons, overcoming gradient instability in recurrent ICNNs and delivering 3–8$\times$ faster MPC solution times at similar predictive accuracy [2603.22095].
- For structure-preserving physical modeling, convexity ensures mathematical well-posedness of PDEs (e.g. non-Newtonian Stokes flow), as data-driven ICNN surrogates automatically obey monotonicity and growth conditions necessary for unique solvability [2401.07121].
- In nonlinear model reduction for real-time deformable simulation, symmetric ICNNs encode both convexity and physical oddness to ensure stability and plausible generalization under out-of-distribution loads or sparse training [2511.18241].

## 6. Application to Convex Learning, Inverse Problems, and Optimal Transport

ICNNs are routinely applied to the learning of convex potentials (e.g., in optimal transport), where gradient maps of an ICNN instantiate monotone transport operators. In Wasserstein distance estimation, adversarial training with ICNNs over Kantorovich dual pairs directly yields Brenier maps, with rigorous consistency and uniqueness guarantees—empirically outperforming or matching regularized GAN and classical OT approaches across a range of synthetic and high-dimensional benchmarks [1908.10962].

In mirror descent and learned optimization, ICNNs serve as expressive, data-driven mirror potentials, enabling the acceleration of iterative optimization with learnable Bregman distances and provable regret bounds. Learned mirror maps on image denoising, inpainting, and large-scale classification outperform classical solvers by one order of magnitude in convergence speed and final accuracy [2206.06733].

In mathematical finance, variants that realize convexity as the supremum of affine forms (max-of-hyperplanes or log-sum-exp smoothing) are employed for option pricing and path-dependent payoffs, with provable approximation theorems (rates $O(n^{-2/d})$ in $d$ dimensions) and observed sub-percent relative errors against Monte Carlo estimators [2411.12854].

## 7. Advances, Extensions, and Theoretical Developments

Recent work has generalized the ICNN construction to spline-based (ICKAN) and Kolmogorov–Arnold architectures, achieving convexity via sums of convex 1D shape bases. Piecewise-linear and cubic-spline ICKANs enjoy universal approximation properties (where proven), and in practice, match or slightly outperform classical ICNNs in regression and learning transport potentials, at lower parameter counts in low to moderate dimensions [2505.21208].

There is ongoing research into the expressivity of general convex ReLU nets vis-à-vis ICNNs, with precise necessary and sufficient convexity characterizations—showing that as network depth increases, ICNNs become a vanishingly small subset of all convex ReLU functions [2501.03017]. Difference-of-convex architectures (CDiNNs) have been proposed to increase representational class, modeling arbitrary smooth functions as $f(x) = g(x) - h(x)$ with $g,h$ convex ReLU nets, while retaining subproblem tractability in optimization [2103.17231].

## References

- [1609.07152] Input Convex Neural Networks
- [2401.07121] Structure-preserving neural networks in data-driven rheological models
- [1805.11835] Optimal Control Via Neural Networks: A Convex Approach
- [2505.04037] Learning based convex approximation for constrained parametric optimization
- [2603.22095] Input Convex Encoder-Only Transformer for Fast and Gradient-Stable MPC in Building Demand Response
- [2410.00796] Fast and Reliable $N-k$ Contingency Screening with Input-Convex Neural Networks
- [2206.06733] Data-Driven Mirror Descent with Input-Convex Neural Networks
- [2312.12474] Principled Weight Initialisation for Input-Convex Neural Networks
- [2411.12854] A new Input Convex Neural Network with application to options pricing
- [2501.03017] Convexity in ReLU Neural Networks: beyond ICNNs?
- [2511.18241] A Convex-Inspired Neural Construction for Structured and Generalizable Nonlinear Model Reduction
- [2505.11047] User-centric Vehicle-to-Grid Optimization with an Input Convex Neural Network-based Battery Degradation Model
- [1908.10962] Optimal transport mapping via input convex neural networks
- [2505.21208] Input Convex Kolmogorov Arnold Networks
- [2103.17231] CDiNN - Convex Difference Neural Networks

Source: https://www.emergentmind.com/topics/input-convex-neural-networks-icnns