---
title: modReLU Activation Functions
url: https://www.emergentmind.com/topics/modrelu-activation
type: topic
---

# modReLU Activation Functions

The modReLU activation function is a family of modified or generalized rectified linear unit functions central to both real- and complex-valued neural networks, with forms and parameterizations that enhance smoothness, adaptability, expressivity, and robustness. modReLU functions appear as smooth ReLU variants, phase-preserving complex-valued nonlinearities, parametric cubic enhancements, normalized and matrix-valued extensions, and capped activations for adversarial robustness.

## 1. Mathematical Definitions and Variants

modReLU refers to a collection of activation functions that build upon the canonical ReLU, $f(x) = \max(0, x)$, by introducing parameters and structure for extended behavior. The central forms include:

- **Smooth modReLU:** Expressed as $f_\alpha(x) = x \cdot \Delta_\alpha(x)$, where $\Delta_\alpha(x)$ is a smooth cumulative distribution function (CDF) such as $(1 - e^{-\alpha x})$ for $x > 0$ and $\alpha > 0$. In the limit $\alpha \to \infty$, the function recovers the hard-step ReLU [1901.09849].

- **Complex-valued modReLU:** For $z \in \mathbb{C}$, $\sigma(z) = \mathrm{ReLU}(|z| + b) \cdot \mathrm{sgn}(z)$, where $b$ is a bias and $\mathrm{sgn}(z) = z/|z|$ for $z \ne 0$ [1910.02333, 2102.13092]. This formulation thresholds the magnitude but preserves the phase—an essential property for applications with complex data.

- **Parametric modReLU:** Enhanced by higher-order terms, $f^{(l)}(x; c_1^{(l)}, c_2^{(l)}) = \theta(x)[c_1^{(l)} x + y c_2^{(l)} x^3]$, where $c_1^{(l)}$ and $c_2^{(l)}$ are layer-dependent parameters and $y$ is a global scale [2403.19896].

- **Capped modReLU:**  $a(z, C) = \max(0, \min(z, C))$, introducing an upper bound $C$ to limit activation output and impede adversarial amplification [2405.03777].

- **Matrix-valued modReLU:**  Generalizes ReLU to matrix-operator activation, where each output can depend on trainable piecewise constant functions of the input, leading to richer cross-neuron adaptivity [2109.09948].

## 2. Smoothness, Adaptivity, and Normalization

Smooth modReLU variants replace the non-differentiable Heaviside step with a smooth CDF, such as the exponential or logistic CDF, leading to continuous derivatives and enhanced gradient flow in deep networks [1901.09849]. Adaptive versions further employ trainable smoothness or shape parameters (e.g., $\alpha$) for each neuron, which are updated during training:

- Back-propagation update for smoothing parameter:
  $$
  \alpha \leftarrow \alpha - \gamma \frac{\partial \mathrm{loss}}{\partial \alpha}
  $$
  with possible reparameterization to enforce positivity.

Static activation normalization, as with the "tilted ReLU" ($f_h(x) = |x| - \sqrt{2/\pi}$), ensures that activation outputs possess zero mean and unit variance under Gaussian input, preserving dynamical isometry and supporting robust convergence, notably permitting deeper architectures to be reliably trained [1905.01369].

## 3. modReLU in Complex-Valued Neural Networks

modReLU is the activation of choice in complex-valued neural networks (CVNNs) due to its phase equivariance, defined by $\sigma(e^{i\theta}z) = e^{i\theta}\sigma(z)$—ensuring rotation compatibility in the complex plane [2102.13092]. This property is central in domains where phase carries semantic information (e.g., MRI fingerprinting).

Theoretical analysis confirms that modReLU-equipped CVNNs approximate any $C^n$-regularity function on compact subsets of $\mathbb{C}^d$ with optimal rates (up to logarithmic factors):
$$
\text{Approximation error} \leq C\ \varepsilon \sim O(\varepsilon^{-2d/n})
$$
for error tolerance $\varepsilon$ and input dimension $d$. The doubling in exponent ("$-2d/n$") compared to real networks follows from the identification $\mathbb{C}^d \cong \mathbb{R}^{2d}$ [2102.13092, 2303.16813]. The optimality of these rates depends on the activations being non-polyharmonic and sufficiently smooth [2303.16813].

## 4. Adaptation, Parametric, and Matrix Extensions

Evolutionary search and gradient descent can be used to optimize modReLU parameters (e.g., bias $b$ or scaling $\alpha$) across architectures or tasks, yielding robust, adaptive activations and improved test accuracy over standard fixed functions [2006.03179]. In applications, custom modReLU forms as parametric modules:

$$
f(z; \theta) = \mathrm{ReLU}(\alpha |z| + b) \cdot (z/|z|)
$$

This parameterization supports architectural and dataset-specific tuning, enhancing performance. Matrix-valued modReLU generalizes fixed activations by encoding the activation as a trainable operator (possibly non-diagonal) over preactivations [2109.09948]:

- Diagonal, tri-diagonal, or general piecewise constant matrix forms
- Parameters trained jointly with network weights and biases
- Empirical accuracy often exceeds that of standard ReLU networks in classification and function approximation tasks

## 5. Regularization, Expressivity, and Spline Theory

Theoretical frameworks connect modReLU to linear spline representations in Banach spaces, with regularization terms such as the $\ell^1$ path-norm:

$$
R(\theta) = \sum_k |v_k| |w_k|
$$
and quadratic weight-decay equivalents. modReLU, similar to ReLU and leaky ReLU (both (0,1,2)-power activations), induces optimal spline fits, controlling function class complexity via underlying operator smoothness [1910.02333]. Complex modReLU fulfills generalized admissibility and scaling properties, preserving theoretical guarantees.

Skip connections—implemented as low-degree polynomials or residual bias terms—carry over from the ReLU activation landscape to modReLU settings, supporting the learning of low-frequency (or affine) components essential for stable and expressive representation [1910.02333].

## 6. Practical Impact: Accuracy, Robustness, and Applications

modReLU variants have demonstrated empirical gains in accuracy, representation richness, and training stability:

- Adaptive cubic extensions yield improved MNIST test accuracy (0.982–0.986), exceeding standard ReLU and swish, with tradeoffs in convergence stability parameter space [2403.19896].
- Smooth modReLU mitigates "dead neuron" effects and enhances learning in early layers via greater variability and curvature adaptivity [1901.09849].
- Matrix-value modReLU achieves lower approximation errors for oscillatory functions and higher classification accuracy on benchmarks such as CIFAR-10 relative to canonical ReLU [2109.09948].
- Capped modReLU, $a(z, C) = \max(0, \min(z, C))$, restricts adversarial perturbation amplification, yielding substantial improvements in adversarial robustness; training with adversarial examples further enhances this effect without major loss of standard accuracy. Sensitivity maps confirm reduced vulnerability at lower activation caps [2405.03777].

CVNNs using modReLU activation have proven effective for applications with natural complex-valued data such as MRI, offering phase-equivariant learning and optimal approximation rates [2102.13092, 2303.16813].

## 7. Limitations, Tradeoffs, and Future Directions

modReLU's key limitations center on parameter tuning, tradeoffs between accuracy and convergence (especially with strong nonlinear augmentations), and the curse of dimensionality in high-dimensional approximations. While the introduction of smoothness, adaptive parameters, and caps mitigate issues of non-differentiability and adversarial vulnerability, excessively tight constraints can lead to underfitting or vanishing gradients [2405.03777]. The optimality of modReLU for CVNN expressivity may be compromised if smoothness or non-polyharmonicity properties are lost [2303.16813].

Matrix and parametric generalizations introduce new layers of trainable parameters, increasing model complexity and computational requirements, although empirical results consistently suggest favorable efficiency/accuracy tradeoffs. Future work is expected to explore further extensions to activation function architecture, especially for complex-valued and adversarially robust models.

---

modReLU activation functions comprise a versatile family instrumental for advancing the expressivity, robustness, and adaptability of both real- and complex-valued neural architectures. Key developments include smooth and normalized variants for stable, deep training; parametric cubic enhancements for increased accuracy; phase-preserving forms for complex data; matrix-valued generalizations for learnable nonlinearity; and capped forms for adversarial defense. The theoretical and empirical body confirms modReLU’s central role in the modern landscape of adaptive activation design, with ongoing research directed at resolving tradeoffs inherent in parameterization, convergence, and approximation complexity.

Source: https://www.emergentmind.com/topics/modrelu-activation