Input-Convex Neural Networks (ICNN)

• ICNNs are feed-forward networks designed to ensure the output is convex with respect to certain inputs by enforcing nonnegative hidden-to-hidden weights and using convex, non-decreasing activations.
• Their structure enables global optimization, allowing methods like gradient descent or linear programming to efficiently solve convex inference problems.
• ICNNs are applied in structured prediction, control, and physical modeling, and advancements include tailored initialization and extensions to handle representational limitations.

An Input-Convex Neural Network (ICNN) is a feed-forward neural network architecture in which the output is guaranteed to be a convex function of (some of) the inputs. The enforcement of convexity is achieved through specific architectural constraints: nonnegative hidden-to-hidden weight matrices and the use of convex, non-decreasing activations. ICNNs were formalized by Amos and Kolter (2017) and enable efficient, globally optimal inference procedures across diverse settings such as structured prediction, control, variational inverse problems, and physical modeling. Recent research has further developed their theory, initialization, numerical performance, and impact in scientific machine learning and optimization domains.

1. Formal Definition and Architectural Constraints

The canonical ICNN implements a deep scalar- or vector-valued mapping $f: \mathbb{R}^d \to \mathbb{R}^p$ such that for any input $x$, $f(x)$ is convex in $x$. The standard layered form for an ICNN with $K$ layers is: $$\begin{aligned} z_0 &= x,\ z_{\ell+1} &= g_\ell\left( W_\ell z_\ell + S_\ell x + b_\ell \right),\quad \ell=0,\ldots,K-1,\ f(x) &= z_K, \end{aligned}$$ where:

• Each activation $g_\ell: \mathbb{R} \to \mathbb{R}$ is convex and non-decreasing (typical choices include ReLU, leaky-ReLU, softplus).
• The hidden-to-hidden weight matrices $W_\ell$ satisfy $W_\ell \geq 0$ entrywise.
• Skip/input weights $S_\ell$ and biases $b_\ell$ are unconstrained.
• The base case $z_0 = x$ or $z_0 = 0$.

This architectural design ensures convexity in the input via the composition and weight constraints: at each layer, convex combinations and convex, non-decreasing activations preserve convexity from one layer to the next (Amos et al., 2016, Liu et al., 8 May 2025, Alvarez-Melis et al., 2021, Omidi et al., 11 Oct 2024, Hoedt et al., 2023).

2. Inference, Optimization, and Representational Properties

The defining property of an ICNN is that it enables global optimization with respect to convex inputs. For a mapping $f(x)$, inference (e.g., minimizing $f(x)$ over a convex set $X$) is a convex program: $\min_{x \in X} f(x)$ If activations are ReLU, exact inference can be reformulated as a linear program by introducing auxiliary variables for each layer (capturing the ReLU as $z \geq u,~z \geq 0$), yielding a tractable, scalable procedure (Liu et al., 8 May 2025, Amos et al., 2016, Omidi et al., 11 Oct 2024, Christianson et al., 1 Oct 2024).

Representationally, ICNNs are universal approximators for convex functions on compact domains: arbitrary convex, piecewise-linear functions can be exactly represented by two-layer ICNNs with sufficient hidden width. By stacking layers and increasing width, smooth convex functions can be approximated to arbitrary precision (Amos et al., 2016, Jones et al., 6 Jun 2025, Alvarez-Melis et al., 2021). However, ICNNs cannot represent nonconvex functions; the difference-of-convex (DC) extension, such as CDiNNs, addresses this representational limitation by learning $f(x) = f_+(x) - f_-(x)$ where both branches are ICNNs (Sankaranarayanan et al., 2021).

3. Initialization, Training Methodologies, and Practical Guidelines

Unique architectural constraints (nonnegative weights, non-decreasing activations) break the assumptions underlying standard initialization schemes such as Glorot/Xavier or He. ICNN-specific initialization generalizes signal propagation to the nonnegative mean/variance domain, setting positive means and appropriately negative biases to center pre-activations and stabilize depth scaling. Specifically, log-normal initialization for $W \geq 0$ and bias centering ensures controlled mean, variance, and correlation propagation, enabling deep ICNNs to train without skip connections (Hoedt et al., 2023).

Training is performed by standard stochastic gradient descent or Adam, with convexity enforced by projection (clipping negative entries to zero) or positive-softplus reparameterization after each step. Skip connections may improve convergence, but are not strictly necessary with principled initialization (Amos et al., 2016, Hoedt et al., 2023). Losses depend on task: supervised regression (MSE), adversarial objectives (learning convex regularizers), variational objectives (energy minimization), or task-specific structured losses. For tasks involving strong convexity, an explicit quadratic term can be added to the final output for analytical guarantees (Mukherjee et al., 2020, Alvarez-Melis et al., 2021).

4. Theoretical Guarantees and Convexity-Preserving Applications

ICNNs are architecturally hard-convex: under the stated constraints, convexity in input is preserved exactly (not merely in expectation or as a penalty). This enables their use in applications requiring global convergence or strong convexity, e.g.:

• Structured prediction with optimal inference and provable label-energy landscapes (Amos et al., 2016).
• Convex regularization in inverse problems, ensuring existence and uniqueness of reconstruction, robust error control, and convergence of subgradient methods (Mukherjee et al., 2020).
• Power systems security screening: the 0-sublevel set of an ICNN classifier forms a convex feasible set, which can be certified to be a subset of the true feasible region using convex optimization. Differentiable scaling layers embed these guarantees into end-to-end training, achieving zero false-negative (reliability) guarantees in real-world N–k contingency screening (Christianson et al., 1 Oct 2024).
• Differential equations and control: embedding ICNNs in governing equations guarantees the monotonicity and convexity properties needed for stability and existence/uniqueness results (e.g. in rheological laws for Stokes flows (Parolini et al., 13 Jan 2024), or in explict model-predictive control for nonlinear systems where convexity enables convex quadratic programming and guarantees global convergence (Wang et al., 13 Aug 2024)).

5. Advances and Domain Applications

ICNNs have seen a range of architectural, theoretical, and empirical advancements:

• Model reduction for nonlinear mechanics: Symmetry-augmented ICNN decoders with odd-symmetrization regularize latent energy landscapes for deformation simulation, achieving robust generalization and stable extrapolation beyond the training manifold (Huang et al., 23 Nov 2025).
• Stochastic programming: Exact surrogate inference in convex two-stage stochastic programming via LP-embedded ICNNs drastically reduces solution time versus mixed-integer NN approaches, with no combinatorial scaling (Liu et al., 8 May 2025).
• Multimodal and physical potentials: Extension to log–sum–exp mixtures (LSE-ICNN) yields differentiable multi-well landscapes with adaptive mode sparsification, supporting physical modeling, variational inference, and microstructural phase transitions (Jones et al., 6 Jun 2025).
• Plasticity and thermodynamic consistency: Hybrid and permutation-invariant ICNNs encode structure necessary for thermodynamically consistent anisotropic yield criteria, outperforming classical and deep-learning baselines on generalization with minimal data (Jadoon et al., 21 Aug 2025).
• Energy systems optimization: Epigraphic ICNN surrogates for nonlinear battery efficiency curves yield convex ERM formulations supporting fast optimization with feasibility guarantees, outperforming linear and mixed-integer models in practical BESS control (Omidi et al., 11 Oct 2024).
• Probabilistic and optimal transport: ICNNs serve as convex potential parametrizations for Brenier and Kantorovich duals, providing unique, robust optimal transport maps that can model sharp, discontinuous or disconnected support relationships (Makkuva et al., 2019, Alvarez-Melis et al., 2021).

6. Limitations, Extensions, and Future Directions

While ICNNs provide principled convex surrogates for a wide class of problems, they are subject to intrinsic limitations:

• Convexity constraint: ICNNs cannot represent or approximate nonconvex targets; for inherently nonconvex mappings, either mixture/ensemble approaches, LSE-ICNNs, or difference-of-convex (DCNN/CDiNN) architectures must be used (Sankaranarayanan et al., 2021, Jones et al., 6 Jun 2025).
• Expressiveness: Highly complex convex functions may require deeper and wider ICNNs, potentially leading to computational bottlenecks in LP- or QP-based inference, or in scenario-rich optimization settings (Liu et al., 8 May 2025).
• Initialization sensitivity: Without principled initialization tailored to nonnegative weights, ICNNs are prone to vanishing or exploding means/variances, leading to poor convergence, especially in deep architectures (Hoedt et al., 2023).
• Application-specific tuning: Model complexity, input dimensionality reduction (e.g. permutation-invariant pooling, principal stress decompositions), and regularization must be carefully aligned to data regime and underlying physical structure (Jadoon et al., 21 Aug 2025, Huang et al., 23 Nov 2025).

Active research directions encompass hybrid architectures mixing partial convexity with standard deep nets, improved large-scale optimization schemes, theoretical bounds for expressive power, convergence, and regularization, and further domain-driven applications in physics, engineering, and computational optimization.

7. Summary Table: ICNN Core Characteristics

Aspect	Implementation	Theoretical Guarantee
Weight constraints	Hidden-to-hidden $W \geq 0$; skip connections allowed	Global convexity in input
Activations	Convex, non-decreasing (e.g., ReLU, softplus)	Inductive convexity proof
Inference	Gradient descent / LP / QP / bundle methods	Global optima (convex programs)
Applications	Prediction, control, optimization, regularization	Existence/uniqueness/stability
Extensions	LSE-ICNN, CDiNN, hybrid/PI-ICNN	Multi-well, DC, symmetries, hybrid
Initialization	Log-normal mean/variance tuning	Stable signal propagation

ICNNs constitute a mature and rapidly developing framework for guaranteeing convexity in neural surrogates, supporting both rigorous theory and practical, high-impact applications across scientific, engineering, and optimization domains (Amos et al., 2016, Liu et al., 8 May 2025, Hoedt et al., 2023, Jones et al., 6 Jun 2025, Huang et al., 23 Nov 2025).