Complex-Valued Activation Functions

Updated 18 November 2025

Complex-valued activation functions are nonlinear mappings in CVNNs that preserve amplitude and phase, crucial for processing complex signals.
They are designed by navigating holomorphic constraints, leading to either unbounded analytic or bounded non-holomorphic lifts such as modReLU and cardioid.
Advanced implementations like kernel-based and rational-polynomial activations offer universal approximation and enhanced training stability in practical applications.

Complex-valued activation functions (CVAFs) are the nonlinear mapping functions essential to Complex-Valued Neural Networks (CVNNs), mediating learned representations between layers in domains where both amplitude and phase convey information. Unlike real-valued activations, the design and analysis of CVAFs are fundamentally shaped by the interplay between holomorphicity, boundedness, and phase preservation, as no non-constant function can be both entire (holomorphic everywhere) and bounded due to Liouville’s theorem, and phase–magnitude symmetry is often critical in physical and signal processing applications. Rigorous classification, expressive power, universality, computational properties, and engineering trade-offs are now well-characterized, and a rapidly expanding repertoire of CVAFs and nonparametric families enables deep, high-performing CVNNs across scientific domains.

1. Mathematical Foundations and Classification

The theoretical space of complex-valued activation functions is delimited by fundamental constraints from complex analysis:

Holomorphicity and the Cauchy–Riemann Conditions: A function $f\!:\mathbb{C}\to\mathbb{C}$ is holomorphic if it is complex-differentiable everywhere, i.e., it satisfies $\frac{\partial f}{\partial \bar{z}} = 0$ . Classic activations such as the complex logistic sigmoid $\sigma(z)=1/(1 + e^{-z})$ and hyperbolic tangent $\tanh(z)$ are holomorphic except at isolated poles, but must be unbounded (Hammad, 27 Jul 2024, Abdalla, 2023, Bassey et al., 2021, Corte et al., 2014).
Non-Holomorphic (“Split” or “Componentwise”) CVAFs: These apply real activation functions independently to either Cartesian components, $f_A(z) = f_\Re(\Re z) + i f_\Im(\Im z)$ (“rectangular split”), or polar form, $f_B(z) = a(|z|) e^{i\phi(\arg z)}$ (“polar split”) (Abdalla, 2023, Barrachina et al., 2023). Most widely used CVAFs, including modReLU, zReLU, cardioid, and various amplitude-phase functions, are non-holomorphic but bounded, or saturate at large $|z|$ .
Phase-Amplitude Preserving Activations: These modulate only $|z|$ and leave $\arg z$ intact (e.g., $f(z) = h(|z|)e^{i\arg z}$ ; cardioid, modReLU, APTF/ASPF, CAP-family). They are non-holomorphic but retain essential physical structure in signal domains (Hammad, 27 Jul 2024, Virtue et al., 2017).
Parametric and Non-Parametric Families: Recent work proposes trainable CVAFs by kernel expansion (KAFs, WL–KAFs) or hybrid parameterized forms generalizing ReLU, Tanh, Softplus, etc., for efficient and expressive phase-sensitive modelling (Young et al., 4 Apr 2025, Scardapane et al., 2018, Scardapane et al., 2019).

The coexistence and trade-off between these types are dictated by the need for universal approximation, training stability, and domain-specific constraints.

2. Expressivity, Universality, and Approximation Theory

The universal approximation property and quantitative error rates critically depend on the analytic structure of CVAFs:

Universal Approximation—Characterization: For CVNNs with two or more hidden layers, universality is achieved if the activation is neither a polynomial in $z,\bar{z}$ , holomorphic, nor antiholomorphic (i.e., not analytic in $z$ or $\bar{z}$ alone), which excludes “degenerate” (e.g., only phase or modulus) activations and captures virtually all non-trivial phase–amplitude nonlinearities (modReLU, cardioid, split activations, etc.) (Geuchen et al., 2023, Voigtlaender, 2020).
Sharp Complexity Bounds: For smooth, non-polyharmonic CVAFs (generalizing “non-polynomial” in the real case), the best achievable $C^k$ function approximation by a CVNN with $m$ hidden units is $O(m^{-k/(2n)})$ where $n$ is the complex input dimension (Geuchen et al., 2023). This matches the classical real-valued optimal rates up to constants and logarithmic factors (Caragea et al., 2021).
Width and Depth Results: Deep CVNNs achieve universality with width $2n+2m+5$ for input-output dimensionalities $n, m$ , with potential improvement to $n+m+3$ for certain CVAFs (those with only one vanishing Wirtinger derivative at some point) (Geuchen et al., 2023).
Shallow CVNNs: Universality for one-hidden-layer CVNNs occurs if and only if the CVAF is not almost polyharmonic (i.e., there does not exist $m$ with $\Delta^m f \equiv 0$ ) (Voigtlaender, 2020).

3. Principal Families and Examples of Complex-Valued Activation Functions

A wide variety of CVAFs is currently in active use. Representative families and their properties:

Name / Notation	Formula / Main Property	Analyticity / Boundedness
Complex sigmoid	$\sigma(z) = 1/(1+e^{-z})$	Holomorphic except poles, unbounded
Tanh	$\tanh(z) = (e^z - e^{-z})/(e^z + e^{-z})$	Holomorphic except poles, unbounded
modReLU	$\mathrm{modReLU}(z) = \mathrm{ReLU}(\|z\|+b)\, z/\|z\|$	Non-holomorphic, phase-preserving, bounded below by $b$
Cardioid	$f(z) = \frac{1}{2}(1+\cos(\arg z))z$	Non-holomorphic, phase-preserving, bounded $[0,1]$
zReLU	$z$ if $0\le\arg z\le\pi/2$ , $0$ else	Non-holomorphic, quadrant gating
Split-Sigmoid/Tanh	$\sigma(\Re z) + i \sigma(\Im z)$ / $\tanh(\Re z) + i \tanh(\Im z)$	Non-holomorphic, bounded
Amplitude–Phase (APTF, APSF)	$h(\|z\|)e^{i\arg z}$ / $z(\alpha+\|z\|)/(\beta+\|z\|)$	Non-holomorphic, bounded
CAP-family (PROPOSED)	See (Hammad, 27 Jul 2024): capping or saturating amplitude	Non-holomorphic, saturating, phase-preserving
Kernel Activation (KAF, WL–KAF)	$g(z) = \sum_n \alpha_n \kappa(z, d_n)$ (+ conjugate term)	Universal, non-holomorphic
Rational-polynomial	$Z(z)=\alpha z + \frac{\sum_{n=0}^p k_n z^n}{\|z\|^q+\epsilon}$	Non-holomorphic, bounded by denominator (Young et al., 4 Apr 2025)

Notably, parametric and kernel-based activation families enable nonparametric adaptation and richer expressivity than either split or analytic scalar functions alone (Scardapane et al., 2018, Scardapane et al., 2019). Novel families generalizing reciprocal-polynomials and smooth switches offer high parameter-efficiency and continuous phase modulation (Young et al., 4 Apr 2025).

4. Backpropagation and Computational Implications

Gradient Formalisms: Holomorphic functions admit classic complex gradients (one Wirtinger derivative, $\partial/\partial z$ ), but most practical CVAFs (especially split-type) require full Wirtinger calculus, involving both $\partial/\partial z$ and $\partial/\partial \bar{z}$ (Abdalla, 2023, Corte et al., 2014). Typical frameworks (TensorFlow, PyTorch) now support complex automatic differentiation, transparently handling the necessary partials for both types (Barrachina et al., 2023).
Branch-Cut and Discontinuity Handling: CVAFs such as zReLU and quadrant-gated switches (e.g., cReImLU, cRecipMax) can exhibit zero derivative regions or discontinuities. Smooth variants (e.g., reciprocal-polynomial, soft-plus parametric forms) are often preferred for training stability (Young et al., 4 Apr 2025).
Phase Preservation and Numerical Stability: activations that maintain $\arg(z)$ (modReLU, cardioid, APTF, CAP-PLS) are vital for signal processing tasks; others (CReLU, split-Cartesian) destroy phase information and may cause unphysical feature maps in such domains (Virtue et al., 2017, Abdalla, 2023). Boundedness or smooth saturation is essential to avoid exploding gradients in deep architectures (Hammad, 27 Jul 2024).

5. Empirical Evidence and Application Domains

Empirical studies consistently validate theoretical expectations:

Performance Gains: modReLU and cardioid activations improve accuracy and stability for MRI fingerprinting, audio, and communications tasks. For example, in MRI reconstruction, complex-cardioid net halved $T_1$ RMSE compared to both real ReLU and split-separable activations; modReLU yielded state-of-the-art RNN stability (Virtue et al., 2017, Bassey et al., 2021, Caragea et al., 2021).
Kernel/Nonparametric Activations: Both split-KAF and fully-complex KAF outperform fixed-shape and real baselines on channel identification, wind prediction, and complex-MNIST (e.g., split-KAF achieves $97.2\%$ test accuracy on complex MNIST vs. $95.9\%$ for modReLU) (Scardapane et al., 2018, Scardapane et al., 2019).
Hybrid Real–Complex Architectures: Introduction of generalized reciprocal- and switch-type CVAFs in hybrid neural networks yields lower parameter count and test loss compared to pure real-valued models on tasks with inherently complex structure (Young et al., 4 Apr 2025).

6. Advanced Design Methodologies and Best Practices

Activation Selection Criteria: For depth-narrow universality, a CVAF must not be holomorphic, antiholomorphic, or $\mathbb{R}$ -affine, and should have non-vanishing Wirtinger derivative at some point (Geuchen et al., 2023). Explicit phase preservation is recommended for physical signals; if analytic coupling is required (for mathematical convenience in backpropagation), fully-complex holomorphic functions may be justified (with care to avoid poles) (Corte et al., 2014, Hammad, 27 Jul 2024).
Weight Initialization and Regularization: For bounded CVAFs (modReLU, cardioid, CAP-PLS), standard complex Glorot/He initialization adapted by a factor $1/\sqrt{2}$ is recommended. Entire functions (tanh, sigmoid) require scaling input/weights to avoid amplification near poles (Barrachina et al., 2023).
Parametric/Nonparametric Approaches: Kernel-based and widely linear KAFs offer universal approximation, adaptability and faster convergence, particularly where real–imaginary coupling is important (Scardapane et al., 2019). Rational-polynomial and switch-type activations can be tuned by architecture search (Young et al., 4 Apr 2025).

7. Open Problems and Future Directions

Bounded Holomorphic CVAFs: No non-constant bounded entire analytic CVAF exists. Progress relies on functions analytic except at engineered isolated singularities, but controlling gradient explosion near these poles remains challenging (Abdalla, 2023, Hammad, 27 Jul 2024).
Phase–Magnitude Tradeoffs: The design of adaptive CVAFs that can learn when to suppress/retain phase or amplitude remains an open engineering and modelling problem, especially as deep CVNNs are scaled to vision and speech domains (Bassey et al., 2021).
Interpretability and Modularization: Understanding how CVAF choice encodes task-specific features (e.g., phase-coded information in electromagnetic or audio signals), and building interpretable CVNN toolkits in major frameworks is an ongoing concern (Bassey et al., 2021, Barrachina et al., 2023).
Scaling and Architectures: Extension to large-scale hybrid real–complex and attention-based architectures will require further innovations in numerically stable, expressive, and computationally efficient CVAFs (Young et al., 4 Apr 2025).

Complex-valued activation functions now constitute a rich, theoretically grounded, and empirically validated foundation for neural architectures in domains with complex structure. Rigorous universality, optimal approximation rates, phase-dynamics, and flexible kernel-based nonlinearities underpin their adoption in modern CVNN frameworks (Geuchen et al., 2023, Caragea et al., 2021, Scardapane et al., 2018, Young et al., 4 Apr 2025, Abdalla, 2023, Hammad, 27 Jul 2024).