Non-Monotonic Activations in Deep Learning

• Non-monotonic activation functions are nonlinearities in neural networks characterized by local negative slopes that create multiple decision hyperplanes and richer expressivity.
• They improve optimization and generalization by enabling single-layer networks to solve non-linearly separable problems and reducing network depth and training epochs.
• Empirical benchmarks show that variants like Sqish, Mish, and GELU significantly boost performance in vision and classification tasks, often enhancing adversarial robustness.

Non-monotonic activation functions are nonlinearities in artificial neural networks whose input–output mapping is not globally monotonic. Unlike monotonic functions, which preserve the relative ordering of their inputs, non-monotonic activations display one or more regions where the derivative is negative, resulting in local “bumps,” oscillations, or symmetries. This property fundamentally changes a neuron’s expressivity, affects optimization and generalization, and underlies several recent advances in deep learning and theoretical neuroscience.

1. Mathematical Definitions and Key Variants

Non-monotonic activation functions are defined by the existence of finite intervals where their derivative becomes negative. Formally, for activation $f:\mathbb{R}\to\mathbb{R}$, $f$ is non-monotonic if $\exists\,a < b$ such that $f'(x) < 0$ for $x\in(a,b)$. Important classes include:

• Single-"bump" functions:
  • Swish: $f(x) = x \sigma(\beta x)$, with $\sigma$ the logistic sigmoid, is non-monotonic on $(-\infty,0)$ for $\beta > 0$ (Ramachandran et al., 2017).
  • Mish: $f(x) = x \tanh(\ln(1 + e^x))$; negative dips occur due to the bounded, smooth “self-gating” factor (Misra, 2019).
  • GELU: $f(x) = x \Phi(x)$, with $\Phi$ the Gaussian CDF, is non-monotonic near $x\approx-1$.
• Piecewise non-monotonic functions:
  • Sqish: $$f(x;a,\beta,\gamma) = a x + (1-a)x / \sqrt{1 + \beta e^{-2\gamma(1-a)x}}$$; tuning parameters produces a non-monotonic region (Biswas et al., 2023).
  • SGELU, SSiLU, SMish: Combine ReLU for $x\geq0$ with non-monotonic gates for $x<0$ (Chen et al., 2023).
• Oscillating/multizero functions:
  • Shifted Quadratic Unit (SQU): $f(z) = z^2 + z$ has two zeros and a parabolic profile.
  • Non-Monotonic Cubic Unit (NCU): $f(z) = z - z^3$ has three zeros and alternating slope (Noel et al., 2021).
  • Shifted Sinc Unit and Decaying Sine Unit: $f(z)$ constructed via $\sinc$ or combinations for repeated zero-crossings and oscillations.
• Even and symmetric activations:
  • Seagull: $f(x)=\log(1+x^2)$, an even, non-monotonic function (Gao et al., 2020).
• Parametric/trainable non-monotonic activations:
  • ErfAct / Pserf: $f(x) = x\,\text{erf}(\alpha e^{\beta x})$ or $x\,\text{erf}(\gamma \log(1 + e^{\delta x}))$; shape controlled by learned parameters (Biswas et al., 2021).
  • Unified Linear Unit (ULU): Piecewise, with $f(x;\alpha_1,\alpha_2)$ determined by distinct parameters for $x<0$ and $x\geq 0$ (Huo, 7 Aug 2025).

2. Geometric and Functional Properties

Non-monotonicity introduces essential geometric differences compared to monotonic activations:

• Multiple decision hyperplanes: Every real zero of $f(z)$ induces a corresponding affine hyperplane in input space where the neuron’s output switches sign. Monotonic $f$ (e.g., ReLU, sigmoid) provide a single hyperplane; oscillators such as SQU or NCU yield two or three parallel lines, and periodic activations like $\sinc$ create infinitely many (Noel et al., 2021).
• Symmetry and exchangeability induction: Even activations (e.g., Seagull) embed algebraic invariances directly, which is theoretically necessary for networks targeting partially-exchangeable functions (Gao et al., 2020).
• “Bump” regions and local gating: In Swish, Mish, and similar designs, a local negative slope around $x<0$ introduces a soft gating effect, leading to richer neuron responses (Ramachandran et al., 2017, Misra, 2019).

3. Theoretical Analysis: Learning, Optimization, and Capacity

The introduction of non-monotonicity impacts learnability, optimization trajectories, representational power, and network capacity:

• Learnability by Gradient Descent: Polynomial-time learnability for single neurons with non-monotonic activation is guaranteed whenever a “dominating linear part” is present, i.e., the positive-slope region is strong enough to override oscillations. Activations such as SiLU/Swish and GELU satisfy all required conditions; highly oscillatory, periodic activations do not and are typically unlearnable by GD in polynomial time (Wu, 2022).
• Associative memory models: Non-monotonic transfer functions in Hopfield-type networks reduce spurious attractors, enlarge retrieval basins, and can nearly triple storage capacity, with $\alpha_c$ rise from $\approx0.138$ (tanh) to $0.36$ (non-monotonic) (Kabashima et al., 22 Oct 2025).
• Gradient flow and saturation: Many non-monotonic activations are non-saturating: their derivative does not vanish as $|x|\to\infty$. For example, SQU and NCU have unbounded derivatives, while conventional sigmoid or tanh saturate and “kill” gradients; Mish and Swish keep nonvanishing gradients for large negative $x$, avoiding the “dead neuron” regime (Noel et al., 2021, Misra, 2019).
• Expressiveness: Non-monotonicity enables single-layer networks to solve problems such as XOR, which monotonic activations cannot achieve without stacked layers. Oscillating activations with multiple zeros yield multiple or piecewise decision boundaries, e.g., SQU or NCU for single-neuron XOR (Noel et al., 2021).

4. Experimental Evidence and Benchmarking

Comprehensive empirical evaluations consistently show that non-monotonic activations outperform their monotonic counterparts across vision, detection, segmentation, and translation tasks:

Activation	Mean Top-1 Gain (CIFAR-100)	Task Domains
Sqish	+5.87% (ShuffleNet V2)	Classification/Robustness (Biswas et al., 2023)
SGELU	+4.3% (MobileNet)	Image Classification (Chen et al., 2023)
ErfAct	+5.68% (ShuffleNet V2)	Classification, Detection, Segmentation (Biswas et al., 2021)
ULU	up to +5 pts (CIFAR-100)	Classification, Detection (Huo, 7 Aug 2025)
Oscillator SQU	+2–5 pts (CIFAR-10/100)	Efficient Vision (Noel et al., 2021)

• Convergence: Oscillating and non-monotonic activations reduce required network depth and training epochs—SQU and NCU reach threshold accuracy 10-15 epochs sooner than ReLU (Noel et al., 2021).
• Adversarial robustness: Sqish exhibits marked improvements for adversarial settings: e.g., +8.21% on CIFAR-100 under FGSM (Biswas et al., 2023).
• Trainable parametrization: ErfAct, Pserf, Sqish, and AULU leverage learnable shape parameters, achieving consistent further gains with negligible computation overhead (Biswas et al., 2023, Biswas et al., 2021, Huo, 7 Aug 2025).
• Symmetric targets: The Seagull activation delivers 2–4× MAE reductions for regression with exchangeable symmetries by directly hardwiring the required functional prior (Gao et al., 2020).

5. Biological Motivation and Neural Analogy

Non-monotonicity is not merely a functional engineering innovation but central in neurobiological models:

• Biological inspiration: Human neocortical pyramidal neurons exhibit non-monotonic response curves—output increases to a peak with increasing input and then decreases, yielding the ability to solve linearly inseparable relations (e.g., XOR) at the single-cell level (Noel et al., 2021).
• Associative memory: Non-monotonic transfer functions in models of neural memory storage dramatically enhance capacity and correct spurious recall artifacts, as shown by Morita (1993) and rigorously quantified via dynamical mean field theory (Kabashima et al., 22 Oct 2025).

6. Design Methodologies and Implementation Considerations

Several principles emerge for effective incorporation of non-monotonic activations:

• Parameterized flexibility: Most recent non-monotonic activations enable trainable shape parameters, e.g., Sqish $(a,\beta,\gamma)$, ErfAct $(\alpha,\beta)$, AULU $(\beta_1, \beta_2)$, allowing per-layer or per-channel adaptation (Biswas et al., 2023, Biswas et al., 2021, Huo, 7 Aug 2025).
• Piecewise/hybrid construction: Saturated non-monotonic activations (SGELU, SSiLU, SMish) explicitly separate the behavior on $x\geq0$ and $x<0$ to preserve gradient flow for positive activations and introduce non-monotonicity only where it benefits optimization (Chen et al., 2023).
• Symmetry and structure induction: Even activation functions directly induce required functional symmetries, a strategy particularly effective when target functions have known algebraic invariances (Gao et al., 2020).

Recommendations consistently favor non-monotonic variants (e.g., ULU, Sqish, ErfAct) as drop-in replacements, given their compatibility and minimal computational overhead relative to ReLU/Swish/Mish baselines (Biswas et al., 2021, Biswas et al., 2023, Huo, 7 Aug 2025).

7. Limitations, Open Questions, and Future Directions

• Pathological oscillations: Highly oscillatory activations (multiple sign changes over the input domain, such as $\sin(d x)$) are unlearnable by practical gradient descent due to Hermite-energy decay, echoing negative learnability results (Wu, 2022).
• Smoothness vs. computation: The trade-off between smooth, highly parametrized activations and computational efficiency is nontrivial. While ErfAct and Sqish induce only moderate overhead and maintain $C^\infty$ smoothness, piecewise constructions like SGELU incur derivative discontinuities at $x=0$.
• Transferability: Empirical gains are predominantly shown in vision tasks; extension and systematic evaluation in large-scale language or RL settings remain open (Biswas et al., 2023, Chen et al., 2023).
• Inductive bias quantification: AULU’s LIB metric provides a quantitative measure of activation asymmetry, connecting model architecture with functional bias (Huo, 7 Aug 2025). A plausible implication is that such meta-metrics may guide network hyperparameter selection or search in data-sparse regimes.
• Theoretical limits: Rigorous theory guarantees efficient learnability only when non-monotonicity is accompanied by a dominating positive-slope region. Further work is needed to precisely delineate which forms of non-monotonicity yield optimization and generalization gains (Wu, 2022).

Non-monotonic activation functions thus form a versatile and theoretically rich class of nonlinearities, spanning smooth, trainable gates to oscillatory, multi-hyperplane separators. Their detailed characterization, functional flexibility, and consistently favorable empirical performance underpin their growing adoption in contemporary deep learning research.