Universal Approximation for Complex-Valued NNs

• Universal Approximation Theorem for Complex-Valued Neural Networks demonstrates that CVNNs can approximate any continuous complex function on compact sets under non-degenerate algebra and tailored activation functions.
• The framework details critical architectural requirements, emphasizing the need for split and non-holomorphic activations in shallow networks and broader permissible functions in deeper models.
• Quantitative insights reveal efficient approximation rates with optimized depth, width, and parameter growth, highlighting CVNNs' advantages over traditional real-valued networks.

A universal approximation theorem (UAT) for complex-valued neural networks (CVNNs) establishes that — under appropriate architectural and activation-function conditions — such networks can approximate any continuous complex-valued function on compact subsets of $\mathbb{C}^n$ to arbitrary precision. The complex-valued setting demands fundamentally distinct algebraic, analytic, and constructive considerations compared to the real case, most notably in the structure of admissible activations, layer width and depth requirements, and the algebraic richness needed for dense approximation.

1. Algebraic Foundations and the Role of Non-degenerate Structure

CVNNs operate over the field of complex numbers, which forms a two-dimensional real algebra with basis $(1, i)$ and multiplication determined by $i^2 = -1$. Many results on UAT for CVNNs extend to broader families of hypercomplex algebras, with universality contingent on algebraic non-degeneracy. Specifically, a real algebra with basis $\tau = \{1, i_1, ..., i_n\}$ and multiplication defined by bilinear forms $B_j$ is non-degenerate if all associated matrices $[B_j]_\tau$ are invertible, implying that the bilinear product structure admits no nontrivial null space (Vital et al., 2022, Valle et al., 2024). For $\mathbb{C}$, the matrices $$[B_0] = \begin{bmatrix} 1 & 0 \ 0 & -1 \end{bmatrix}$$ and $$[B_1]=\begin{bmatrix} 0 & 1 \ 1 & 0 \end{bmatrix}$$ are both invertible, confirming that $\mathbb{C}$ is non-degenerate and thus suitable for UAT constructions in the complex domain.

2. Statement of the Universal Approximation Theorem for CVNNs

The standard form of the UAT for a single-hidden-layer CVNN, specialized to non-degenerate algebras and $\mathbb{C}$ in particular, is as follows (Vital et al., 2022, Valle et al., 2024, Voigtlaender, 2020):

• Network structure: $f(z) = \sum_{j=1}^m \alpha_j \sigma(w_j^T z + b_j)$ for $z \in \mathbb{C}^n$, with $w_j, b_j, \alpha_j \in \mathbb{C}$ and activation $\sigma:\mathbb{C} \rightarrow \mathbb{C}$.
• Activation: $\sigma$ is a "split" version of a real continuous function: $\sigma(x_0 + i x_1) = \phi(x_0) + i \phi(x_1)$, where $\phi:\mathbb{R} \to \mathbb{R}$ is continuous, non-constant, with $\lim_{t\to -\infty} \phi(t)=0$ (e.g., ReLU, logistic).
• Domain and metric: On any compact subset $K \subset \mathbb{C}^n$, with uniform (supremum) norm.
• Theorem: For every continuous $f:K\to\mathbb{C}$ and every $\varepsilon > 0$, there exists $m\in\mathbb{N}$, weights $w_j, b_j, \alpha_j \in \mathbb{C}$ so that $$\sup_{z\in K} |f(z) - \sum_{j=1}^m \alpha_j \sigma(w_j^T z + b_j)| < \varepsilon$$.

This existential result shows that such CVNNs can uniformly approximate any continuous complex function on compacts, providing a foundational theoretical guarantee for their use in function approximation, regression, and classification in complex domains.

3. Characterization of Universal Activation Functions

A hallmark feature of UAT for CVNNs is the nuanced dependence on activation function analyticity and polyharmonicity. Unlike the real case, where any continuous non-polynomial $\sigma:\mathbb{R}\to\mathbb{R}$ suffices, CVNNs encounter subtler obstructions (Voigtlaender, 2020, Geuchen et al., 2023):

• Shallow (1-hidden-layer) networks: Universality holds if and only if $\sigma$ is not almost polyharmonic; that is, neither the real nor imaginary part of $\sigma$ is polyharmonic of finite order. For instance, $\sigma(z) = \mathrm{Re}(z)^3 + \mathrm{Im}(z)$ is admissible, but holomorphic functions (e.g., $\sigma(z)=z^k$, $\sigma(z)=e^z$) or real-affine functions are not.
• Deep CVNNs (depth $\geq 2$): Universality is broader, requiring only that $\sigma$ is not holomorphic, antiholomorphic, or (real-)affine; any non-(anti)holomorphic, non-real-affine, and non-polynomial activation suffices (Voigtlaender, 2020, Geuchen et al., 2023). This inclusion notably admits non-holomorphic activations such as split ReLU, modReLU, or phase-magnitude nonlinearities.
• Width and depth requirements: For networks of minimal width, universality demands $W \geq 2n+2m$ ($n$ input, $m$ output) in general, with improved bounds for specific non-polyharmonic activations (Geuchen et al., 2023).

These results fundamentally distinguish CVNNs from their real-valued counterparts and from the special case of holomorphic neural networks, where universality fails for the full space $C(K,\mathbb{C})$.

4. Proof Techniques and Density Arguments

UAT for CVNNs relies on complexified versions of the Stone–Weierstrass theorem and sophisticated algebraic constructions (Vital et al., 2022, Valle et al., 2024, Voigtlaender, 2020). The key proof steps are:

1. Decomposition: Any continuous $f:K\to\mathbb{C}$ is written as $u(z) + i v(z)$, with $u, v \in C(K;\mathbb{R})$.
2. Real UAT Application: The classical Cybenko/Hornik theorem provides real MLPs that independently approximate $u$ and $v$ arbitrarily well on $K$.
3. Complex Synthesis via Split Activation: The real approximants are "assembled" into a complex-valued network using the split activation, ensuring that each complex neuron independently approximates real and imaginary parts.
4. Alternative Direct Approach: For deep CVNNs, explicit construction of polynomial generators (e.g., $z^k$, $\bar{z}^l$) and conjugation via networks using non-(anti)holomorphic activations; subsequent application of Stone–Weierstrass guarantees uniform density.
5. Algebraic Non-degeneracy: Ensures sufficient richness in the set of linear functionals to match real and imaginary parts through projections, critical for both representational completeness and proof closure.

5. Quantitative Approximation Rates and Efficiency

Beyond existential density, quantitative error estimates have been obtained for CVNNs with specific activations. For instance, with modReLU activation, any $C^n$ function on a compact $K \subset \mathbb{C}^d$ can be approximated to error $\varepsilon$ using networks of depth $O(\log(1/\varepsilon))$, size $O(\varepsilon^{-2d/n} (\log(1/\varepsilon))^2)$, and modest coefficient growth (Caragea et al., 2021). The approximation rate matches, up to log factors, the best-known rates for real-valued ReLU networks over $\mathbb{R}^{2d}$, reflecting the effective doubling of input dimension.

For certain target classes, notably radial functions $f(x)=g(\|x\|)$, complex-reaction networks with zReLU activation achieve polynomial parameter counts (width $O(d(2d)^{3.75})$), whereas real networks require exponential parameter growth to match accuracy (Zhang et al., 2021). This separation is enabled by the phase-magnitude coupling in zReLU together with complex-linear parameterization.

6. Architectural Implications and Landscape Properties

Several critical architectural and optimization-theoretic distinctions arise:

• Activation choice: Admissible activations must not be holomorphic, antiholomorphic, or real-affine; split ReLU, modReLU, and zReLU are theoretically justified, while standard holomorphic choices collapse the functional closure (Voigtlaender, 2020, Geuchen et al., 2023).
• Width and depth: A width of $2n+2m+5$ always suffices for universality in deep CVNNs for all input/output dimensions; this can be reduced to $n+m+3$ for activations with certain Wirtinger nonvanishing derivatives (Geuchen et al., 2023).
• Optimization landscape: The critical set for empirical risk in complex-reaction networks is strictly smaller than for real-valued analogues, due to additional holomorphic constraints restricting degeneracy directions. All global minima of the complex model are stationary for the real model, but the converse fails, implying enhanced "sharpness" in the complex landscape (Zhang et al., 2021).

7. Comparison with Real-valued Networks and Holomorphic Models

The UAT for CVNNs extends and departs from the classical real case in several core respects:

Setting	Admissible Activations	Key Obstructions	Input Domain
Real, shallow	Non-polynomial, continuous	Polynomials only	$K \subset \mathbb{R}^n$
Complex, shallow	Not almost polyharmonic	Polyharmonicity	$K \subset \mathbb{C}^n$
Complex, deep ($L \geq 2$)	Not (anti)holomorphic/affine	(Anti)holomorphic, real-affine	$K \subset \mathbb{C}^n$
Holomorphic NNs	Holomorphic	Only holomorphic targets possible	$K \subset \mathbb{C}^n$

A salient distinction is the essential role of non-holomorphicity for universality in the complex-valued setting (Voigtlaender, 2020). Furthermore, certain function classes (notably those with high radial Fourier content) can be learned efficiently by CVNNs but not by real networks, evidencing expressivity advantages in the presence of complex-structured data (Zhang et al., 2021).

• "Extending the Universal Approximation Theorem for a Broad Class of Hypercomplex-Valued Neural Networks" (Vital et al., 2022)
• "Universal Approximation Theorem for Vector- and Hypercomplex-Valued Neural Networks" (Valle et al., 2024)
• "The universal approximation theorem for complex-valued neural networks" (Voigtlaender, 2020)
• "Universal approximation with complex-valued deep narrow neural networks" (Geuchen et al., 2023)
• "Quantitative approximation results for complex-valued neural networks" (Caragea et al., 2021)
• "Towards Understanding Theoretical Advantages of Complex-Reaction Networks" (Zhang et al., 2021)