SuMo-net: Monotonic Neural Networks

• Monotonic Neural Networks (SuMo-net) are model architectures that ensure outputs increase monotonically with respect to designated input features, enhancing interpretability and fairness.
• They enforce monotonicity using rigorous mechanisms such as weight constraints, residual constructions, and smooth min-max operators that guarantee Lipschitz continuity and universal approximation.
• SuMo-net models find applications in finance, medicine, physics, and survival analysis, offering robust empirical performance alongside theoretical guarantees and efficient training procedures.

Monotonic Neural Networks (SuMo-net) are a family of neural network architectures that guarantee monotonic dependence of the output on designated subsets of the input features. Monotonicity is an essential inductive bias for numerous domains requiring interpretable, fair, or physically plausible models, including finance, medicine, physics, control, and scientific modeling. SuMo-net architectures enforce monotonicity via weight constraints, residual constructions, or specialized aggregation modules, providing theoretical guarantees, practical robustness, and competitive empirical performance. The field encompasses several related structural paradigms, notably exact Lipschitz-constrained MLPs, lattice-based architectures, smooth min–max modules, and monotonic survival and dynamical networks.

1. Theoretical Foundations and Architectural Design

Monotonic neural networks are constructed to ensure some or all output coordinates are non-decreasing functions with respect to specified input coordinates. The most rigorous approaches, such as the Lipschitz Monotonic Network (LMN), enforce monotonicity and Lipschitz bounds by architectural design rather than heuristic penalties.

Given a function $f:\mathbb{R}^d\to\mathbb{R}$, and a subset of monotonic coordinates $M\subseteq\{1,\dots,d\}$, monotonicity is defined as

$$\frac{\partial f}{\partial x_i}(x) \geq 0,\quad \forall i\in M,\, \text{a.e. }x.$$

The LMN design composes

$f(x) = g(x) + \lambda \sum_{i\in M} x_i$

where $g$ is a $D$-layer MLP implemented using gradient-norm-preserving activations (e.g., GroupSort), with all weight matrices normalized to enforce $\ell_1$-Lipschitz constant $\lambda$; the residual term ensures strict monotonicity in the selected coordinates. No sign constraint is placed on the unconstrained subset $U$ of features. Because the partial derivatives of $g$ are bounded, the monotonicity of $f$ in $M$ is exact by construction (Kitouni et al., 2023). Sorting-based activations and $\ell_1$ weight constraints enable universal approximation of all $\lambda$-Lipschitz monotonic functions.

Smooth Min-Max Monotonic Networks (SuMo-net modules) extend the classical Sill min–max architecture by replacing nonsmooth min and max operations with smooth and strictly increasing soft-min and soft-max operators—specifically scaled log-sum-exp functions. Each internal weight is parameterized to be strictly positive, ensuring monotonicity is preserved across module compositions (Igel, 2023). This approach is both expressive (retaining universal approximation capability for bounded monotonic functions) and efficient, as it avoids the optimization pathologies associated with nonsmooth combinators.

Other monotonic architectures, such as Deep Lattice Networks (DLNs), employ alternating layers of non-negative linear embeddings, piecewise-linear (calibrator) transforms, and ensemble-of-lattices layers, with explicit projections into constraint sets during training to enforce monotonicity even at fine granularity (You et al., 2017).

2. Monotonicity Enforcement Mechanisms

Weight non-negativity is the standard mechanism for monotonicity in MLPs. For an affine map $l(x)=Wx+b$, $W \geq 0$ ensures outputs are non-decreasing functions of their inputs. For deeper architectures, repeat compositions with non-decreasing activations (e.g., ReLU, softplus, or tanh) and non-negative weights guarantee the network is globally monotonic (Sartor et al., 5 May 2025).

Advanced strategies have been developed to circumvent the brittleness and constrained initialization of traditional non-negative weight approaches. For example, activation-switch splits each weight matrix into positive and negative parts $(W^+, W^-)$ and combines activations using

$\hat f(x) = W^+ \sigma(x) + W^- \sigma(-x) + b$

This formulation avoids explicit weight parameter projection and works for a broader class of activations, including one-sided saturating nonlinearities (e.g., ReLU, CELU, ELU), and allows for generic initializers (Sartor et al., 5 May 2025).

DLNs alternately constrain weights, calibrator knots, and lattice parameters through explicit projection steps: positive-weight clipping for linear layers, isotonic regression for calibrator nodes (enforcing $b[1] \leq \ldots \leq b[K]$), and partial-order isotonic regression for lattice vertex parameters (You et al., 2017).

Smooth min–max networks guarantee strict monotonicity by parameterizing internal weights as exponentials of unconstrained reals and aggregating via log-sum-exp functions with strictly positive scaling parameters (Igel, 2023). The resulting module is end-to-end differentiable, and standard optimizers suffice.

3. Expressiveness and Universal Approximation

The expressiveness of monotonic neural networks is well-characterized. LMNs with GroupSort activations and $\ell_1$ norm constraints are universal approximators for $\lambda$-Lipschitz monotonic functions over $\mathbb{R}^d$ (Kitouni et al., 2023). Similarly, DLNs, through calibration and lattice layers, construct universal approximators for arbitrary continuous monotonic functions over compact spaces (You et al., 2017).

Crucially, recent work shows that even classical MLPs with non-negative weights and alternating one-sided saturating activations suffice for universal approximation of monotonic functions on finite domains. The necessity of bounded-in-both-directions activations is obviated: ReLU, ELU, and related one-sided saturating nonlinearities are sufficient if alternated properly across layers (Sartor et al., 5 May 2025). The equivalence between non-negative and non-positive weight constraints (with appropriate activation flipping) broadens the architectural design space, permitting convex monotone activations and non-positive weight matrices without loss of universality.

Smooth Min-Max modules inherit the universal approximation property of the original min–max network, converging to arbitrary continuous monotonic functions as group and neuron counts—and scaling parameters—become large (Igel, 2023).

4. Training Procedures and Optimization

Training monotonic networks typically involves alternating unconstrained optimization steps with exact or approximate projections into the feasible parameter set. In LMNs, weight matrices are normalized after each gradient update to maintain the global $\ell_1$-Lipschitz constraint, ensuring exact monotonicity and robustness at all times (Kitouni et al., 2023). The training loop consists of standard mini-batch sampling, forward and backward passes, and projection steps—potentially per-column or full-matrix normalization.

DLNs perform projection on weights, calibrator knots (via coordinate-wise isotonic regression), and lattice parameters (full partial-order isotonic regression) after each ADAM update to sustain end-to-end monotonicity and feasibility (You et al., 2017).

Smooth Min-Max modules, because they employ unconstrained internal parameters (e.g., $z$ for weights via $w=\exp(z)$ and log-scale $\beta$), eliminate the need for projections. All training proceeds via standard gradient descent variants. Early stopping on a small held-out validation set is effective for regularization (Igel, 2023).

Empirically, activation-switch architectures that decouple parameter sign constrains from the forward graph (by positive/negative splitting) yield higher and more stable gradients—up to two orders of magnitude greater—than classical $|W|$ parameterizations, improving both convergence and solution quality (Sartor et al., 5 May 2025).

5. Applications in Scientific, Physical, and Fairness-Constrained Modeling

Monotonic neural networks have demonstrated efficacy in domains requiring theoretical guarantees on model monotonicity. Applications include:

• Particle physics: Real-time triggering in high-rate LHCb data acquisition, where monotonicity with respect to features like particle displacement is a domain axiom; LMNs yield smooth, $\ell_1$-robust decision boundaries with uniform efficiency (Kitouni et al., 2023).
• Survival analysis: SuMo-net survival models use monotonic neural nets to enforce the requirement that survival probability decreases with time. These models achieve state-of-the-art held-out log-likelihood and up to 100× inference speedup over ODE-based baselines on right-censored datasets. Enforced monotonicity enables proper scoring rules and scalability to millions of samples (Rindt et al., 2021).
• Dynamical systems: In learning monotone, stable physical or biological dynamics, architectures like SuMo-net enforce both monotonicity via weight constraints and stability via the joint training of a Lyapunov certificate. This yields substantial error reduction and avoids physically implausible extrapolations (Wang et al., 2020).
• Constitutive modeling in mechanics: Parametric hyperelastic PANNs, constructed as monotonic neural networks, encode relaxed ellipticity conditions for rubber-like materials. By constraining the derivatives of network outputs with respect to physically meaningful invariants, these models achieve robust numerical performance and excellent extrapolation (Klein et al., 5 Jan 2025).

Monotonic neural networks are also natural regularizers where interpretability, ceteris paribus reasoning, or algorithmic fairness are critical, as monotonicity can be mapped directly to real-world expectations or policy-mandated requirements.

6. Empirical Performance and Benchmarks

Experimental evaluations consistently demonstrate that monotonic neural networks match or exceed the empirical performance of unconstrained or heuristic-penalty models across a wide spectrum of tasks. LMNs achieve state-of-the-art accuracy or RMSE on tabular tasks such as COMPAS, BlogFeedback, LoanDefaulter, ChestXRay, AutoMPG, and HeartDisease, often with an order-of-magnitude parameter reduction compared to baselines (Kitouni et al., 2023, Igel, 2023).

Deep Lattice Networks maintain competitive classification and regression accuracy on large-scale benchmarks (Adult income, User Intent, monotonic regression on rater scores) and enjoy provable global monotonicity (You et al., 2017).

Smooth Min-Max modules demonstrate consistently lower test MSE and variance compared to classical min–max, XGBoost, isotonic regression, and hierarchical lattice layers, in both synthetic and real-world monotonic regression scenarios (Igel, 2023).

SuMo-net survival models achieve the best or near-best right-censored log-likelihoods on FLCHAIN, GBSG, KKBOX, METABRIC, and SUPPORT datasets, while offering 20–100$\times$ faster inference compared to Cox-Time or SODEN (Rindt et al., 2021).

7. Practical Considerations and Implementation Guidelines

Key practical insights for constructing and training SuMo-net-style monotonic neural networks include:

• Use smooth, strictly increasing nonlinearities with unconstrained parameterizations to sidestep vanishing gradient issues and dead units (as in MM architectures).
• Employ norm-preserving activations (e.g., GroupSort, Householder) and enforce Lipschitz bounds for robustness when required.
• For partial monotonicity, partition input features and augment architectures with small unconstrained auxiliary modules—hybrid designs are supported in SMM and DLN frameworks (Igel, 2023, You et al., 2017).
• Optimization is effective via standard ADAM, SGD+momentum, or Rprop; batch normalization may be included with empirical monitoring of monotonicity violations.
• For SMM modules, default group counts $K$ and size $h_k$ of 6 are robust across applications, and the scaling parameter $\beta$ can be trained end-to-end.
• Initialization can use standard routines (Kaiming, Glorot); positive weights can be encoded via exponentials or squaring of unconstrained reals.
• Early stopping on validation sets and mild parameter regularization (e.g., patience 50 or progress strip with $k=5$, $\tau=10^{-3}$) are effective for generalization and convergence.

In summary, SuMo-net architectures enable neural function approximation with provable, exact monotonicity guarantees, broadening the scope of neural models applicable in domains where such structural properties are mandated or strongly beneficial. Their structural simplicity, algorithmic transparency, theoretical expressiveness, and competitive empirical profile position them as foundational elements for interpretable, robust, and fairness-aware machine learning pipelines.