--- title: Complex-Valued Activation Functions url: https://www.emergentmind.com/topics/complex-valued-activation-functions type: topic --- # Complex-Valued Activation Functions Complex-valued activation functions (CVAFs) are the nonlinear mapping functions essential to Complex-Valued Neural Networks (CVNNs), mediating learned representations between layers in domains where both amplitude and phase convey information. Unlike real-valued activations, the design and analysis of CVAFs are fundamentally shaped by the interplay between holomorphicity, boundedness, and phase preservation, as no non-constant function can be both entire (holomorphic everywhere) and bounded due to Liouville’s theorem, and phase–magnitude symmetry is often critical in physical and signal processing applications. Rigorous classification, expressive power, universality, computational properties, and engineering trade-offs are now well-characterized, and a rapidly expanding repertoire of CVAFs and nonparametric families enables deep, high-performing CVNNs across scientific domains. ## 1. Mathematical Foundations and Classification The theoretical space of complex-valued activation functions is delimited by fundamental constraints from complex analysis: - **Holomorphicity and the Cauchy–Riemann Conditions:** A function $f\!:\mathbb{C}\to\mathbb{C}$ is holomorphic if it is complex-differentiable everywhere, i.e., it satisfies $\frac{\partial f}{\partial \bar{z}} = 0$. Classic activations such as the complex logistic sigmoid $\sigma(z)=1/(1 + e^{-z})$ and hyperbolic tangent $\tanh(z)$ are holomorphic except at isolated poles, but must be unbounded [2407.19258, 2312.06087, 2101.12249, 1406.5254]. - **Non-Holomorphic (“Split” or “Componentwise”) CVAFs:** These apply real activation functions independently to either Cartesian components, $f_A(z) = f_\Re(\Re z) + i f_\Im(\Im z)$ (“rectangular split”), or polar form, $f_B(z) = a(|z|) e^{i\phi(\arg z)}$ (“polar split”) [2312.06087, 2302.08286]. Most widely used CVAFs, including modReLU, zReLU, cardioid, and various amplitude-phase functions, are non-holomorphic but bounded, or saturate at large $|z|$. - **Phase-Amplitude Preserving Activations:** These modulate only $|z|$ and leave $\arg z$ intact (e.g., $f(z) = h(|z|)e^{i\arg z}$; cardioid, modReLU, APTF/ASPF, CAP-family). They are non-holomorphic but retain essential physical structure in signal domains [2407.19258, 1707.00070]. - **Parametric and Non-Parametric Families:** Recent work proposes trainable CVAFs by kernel expansion (KAFs, WL–KAFs) or hybrid parameterized forms generalizing ReLU, Tanh, Softplus, etc., for efficient and expressive phase-sensitive modelling [2504.03497, 1802.08026, 1902.02085]. The coexistence and trade-off between these types are dictated by the need for universal approximation, training stability, and domain-specific constraints. ## 2. Expressivity, Universality, and Approximation Theory The universal approximation property and quantitative error rates critically depend on the analytic structure of CVAFs: - **Universal Approximation—Characterization:** For CVNNs with two or more hidden layers, universality is achieved if the activation is neither a polynomial in $z,\bar{z}$, holomorphic, nor antiholomorphic (i.e., not analytic in $z$ or $\bar{z}$ alone), which excludes “degenerate” (e.g., only phase or modulus) activations and captures virtually all non-trivial phase–amplitude nonlinearities (modReLU, cardioid, split activations, etc.) [2305.16910, 2012.03351]. - **Sharp Complexity Bounds:** For smooth, non-polyharmonic CVAFs (generalizing “non-polynomial” in the real case), the best achievable $C^k$ function approximation by a CVNN with $m$ hidden units is $O(m^{-k/(2n)})$ where $n$ is the complex input dimension [2303.16813]. This matches the classical real-valued optimal rates up to constants and logarithmic factors [2102.13092]. - **Width and Depth Results:** Deep CVNNs achieve universality with width $2n+2m+5$ for input-output dimensionalities $n, m$, with potential improvement to $n+m+3$ for certain CVAFs (those with only one vanishing Wirtinger derivative at some point) [2305.16910]. - **Shallow CVNNs:** Universality for one-hidden-layer CVNNs occurs if and only if the CVAF is not almost polyharmonic (i.e., there does not exist $m$ with $\Delta^m f \equiv 0$) [2012.03351]. ## 3. Principal Families and Examples of Complex-Valued Activation Functions A wide variety of CVAFs is currently in active use. Representative families and their properties: | Name / Notation | Formula / Main Property | Analyticity / Boundedness | |---------------------------|------------------------------------------|------------------------------------------| | **Complex sigmoid** | $\sigma(z) = 1/(1+e^{-z})$ | Holomorphic except poles, unbounded | | **Tanh** | $\tanh(z) = (e^z - e^{-z})/(e^z + e^{-z})$ | Holomorphic except poles, unbounded | | **modReLU** | $\mathrm{modReLU}(z) = \mathrm{ReLU}(|z|+b)\, z/|z|$ | Non-holomorphic, phase-preserving, bounded below by $b$ | | **Cardioid** | $f(z) = \frac{1}{2}(1+\cos(\arg z))z$ | Non-holomorphic, phase-preserving, bounded $[0,1]$ | | **zReLU** | $z$ if $0\le\arg z\le\pi/2$, $0$ else | Non-holomorphic, quadrant gating | | **Split-Sigmoid/Tanh** | $\sigma(\Re z) + i \sigma(\Im z)$ / $\tanh(\Re z) + i \tanh(\Im z)$ | Non-holomorphic, bounded | | **Amplitude–Phase (APTF, APSF)** | $h(|z|)e^{i\arg z}$ / $z(\alpha+|z|)/(\beta+|z|)$ | Non-holomorphic, bounded | | **CAP-family (PROPOSED)** | See [2407.19258]: capping or saturating amplitude | Non-holomorphic, saturating, phase-preserving | | **Kernel Activation (KAF, WL–KAF)** | $g(z) = \sum_n \alpha_n \kappa(z, d_n)$ (+ conjugate term) | Universal, non-holomorphic | | **Rational-polynomial** | $Z(z)=\alpha z + \frac{\sum_{n=0}^p k_n z^n}{|z|^q+\epsilon}$ | Non-holomorphic, bounded by denominator [2504.03497] | Notably, parametric and kernel-based activation families enable nonparametric adaptation and richer expressivity than either split or analytic scalar functions alone [1802.08026, 1902.02085]. Novel families generalizing reciprocal-polynomials and smooth switches offer high parameter-efficiency and continuous phase modulation [2504.03497]. ## 4. Backpropagation and Computational Implications - **Gradient Formalisms:** Holomorphic functions admit classic complex gradients (one Wirtinger derivative, $\partial/\partial z$), but most practical CVAFs (especially split-type) require full Wirtinger calculus, involving both $\partial/\partial z$ and $\partial/\partial \bar{z}$ [2312.06087, 1406.5254]. Typical frameworks (TensorFlow, PyTorch) now support complex automatic differentiation, transparently handling the necessary partials for both types [2302.08286]. - **Branch-Cut and Discontinuity Handling:** CVAFs such as zReLU and quadrant-gated switches (e.g., cReImLU, cRecipMax) can exhibit zero derivative regions or discontinuities. Smooth variants (e.g., reciprocal-polynomial, soft-plus parametric forms) are often preferred for training stability [2504.03497]. - **Phase Preservation and Numerical Stability:** activations that maintain $\arg(z)$ (modReLU, cardioid, APTF, CAP-PLS) are vital for signal processing tasks; others (CReLU, split-Cartesian) destroy phase information and may cause unphysical feature maps in such domains [1707.00070, 2312.06087]. Boundedness or smooth saturation is essential to avoid exploding gradients in deep architectures [2407.19258]. ## 5. Empirical Evidence and Application Domains Empirical studies consistently validate theoretical expectations: - **Performance Gains:** modReLU and cardioid activations improve accuracy and stability for MRI fingerprinting, audio, and communications tasks. For example, in MRI reconstruction, complex-cardioid net halved $T_1$ RMSE compared to both real ReLU and split-separable activations; modReLU yielded state-of-the-art RNN stability [1707.00070, 2101.12249, 2102.13092]. - **Kernel/Nonparametric Activations:** Both split-KAF and fully-complex KAF outperform fixed-shape and real baselines on channel identification, wind prediction, and complex-MNIST (e.g., split-KAF achieves $97.2\%$ test accuracy on complex MNIST vs. $95.9\%$ for modReLU) [1802.08026, 1902.02085]. - **Hybrid Real–Complex Architectures:** Introduction of generalized reciprocal- and switch-type CVAFs in hybrid neural networks yields lower parameter count and test loss compared to pure real-valued models on tasks with inherently complex structure [2504.03497]. ## 6. Advanced Design Methodologies and Best Practices - **Activation Selection Criteria:** For depth-narrow universality, a CVAF must not be holomorphic, antiholomorphic, or $\mathbb{R}$-affine, and should have non-vanishing Wirtinger derivative at some point [2305.16910]. Explicit phase preservation is recommended for physical signals; if analytic coupling is required (for mathematical convenience in backpropagation), fully-complex holomorphic functions may be justified (with care to avoid poles) [1406.5254, 2407.19258]. - **Weight Initialization and Regularization:** For bounded CVAFs (modReLU, cardioid, CAP-PLS), standard complex Glorot/He initialization adapted by a factor $1/\sqrt{2}$ is recommended. Entire functions (tanh, sigmoid) require scaling input/weights to avoid amplification near poles [2302.08286]. - **Parametric/Nonparametric Approaches:** Kernel-based and widely linear KAFs offer universal approximation, adaptability and faster convergence, particularly where real–imaginary coupling is important [1902.02085]. Rational-polynomial and switch-type activations can be tuned by architecture search [2504.03497]. ## 7. Open Problems and Future Directions - **Bounded Holomorphic CVAFs:** No non-constant bounded entire analytic CVAF exists. Progress relies on functions analytic except at engineered isolated singularities, but controlling gradient explosion near these poles remains challenging [2312.06087, 2407.19258]. - **Phase–Magnitude Tradeoffs:** The design of adaptive CVAFs that can learn when to suppress/retain phase or amplitude remains an open engineering and modelling problem, especially as deep CVNNs are scaled to vision and speech domains [2101.12249]. - **Interpretability and Modularization:** Understanding how CVAF choice encodes task-specific features (e.g., phase-coded information in electromagnetic or audio signals), and building interpretable CVNN toolkits in major frameworks is an ongoing concern [2101.12249, 2302.08286]. - **Scaling and Architectures:** Extension to large-scale hybrid real–complex and attention-based architectures will require further innovations in numerically stable, expressive, and computationally efficient CVAFs [2504.03497]. --- Complex-valued activation functions now constitute a rich, theoretically grounded, and empirically validated foundation for neural architectures in domains with complex structure. Rigorous universality, optimal approximation rates, phase-dynamics, and flexible kernel-based nonlinearities underpin their adoption in modern CVNN frameworks [2305.16910, 2102.13092, 1802.08026, 2504.03497, 2312.06087, 2407.19258]. Source: https://www.emergentmind.com/topics/complex-valued-activation-functions