Surjectivity in Neural Networks

• Surjectivity in neural networks is defined as the property where every point in the codomain is attainable by some input, ensuring comprehensive expressivity and potential invertibility.
• Theoretical frameworks analyze surjectivity across models—from piecewise affine (ReLU) networks and smooth mappings on manifolds to infinite-dimensional neural operators—using distinct mathematical criteria.
• Verifying surjectivity is computationally challenging (NP-hard/coNP-hard), with implications for robust generative modeling, architectural safety, and the reliable inversion of network outputs.

Surjectivity in neural networks concerns whether every point in a network's codomain is attainable by some input. This property is critical for understanding both the expressivity and the safety of neural function classes, and is central to theoretical foundations of generative modeling, invertibility of neural operators, and verification of network behavior. The following sections outline key mathematical frameworks, principal results, and their implications for diverse neural architectures, as established in the literature.

1. Mathematical Formulations of Surjectivity in Neural Architectures

For a function $F: \mathbb{R}^n \to \mathbb{R}^n$ computed by a neural network, $F$ is surjective if for every $y \in \mathbb{R}^n$ there exists $x \in \mathbb{R}^n$ such that $F(x) = y$. The specific criteria for surjectivity depend strongly on the underlying network architecture and activation scheme:

• Piecewise Affine Networks: For feedforward ReLU networks, $F$ is piecewise affine—formed by collections of affine ‘selection functions’ defined on polyhedral regions. The function $F$ is surjective if all selection functions active on the unbounded polyhedra of the domain have Jacobian determinants with the same nonzero sign, i.e.,

$$\text{sign}\left[\det(D_x F)\right] = \text{constant} \neq 0$$

for all $x$ in unbounded polyhedra of $P(F)$ (Radons, 2017).

• Smooth Networks: For $C^1$ maps $f: M \rightarrow N$ between manifolds, $f$ is surjective if $f(M)$ is closed, $f$ has at least one regular point, and $\dim(\text{Crit}(f)) < n - 1$ (Shi et al., 2018).
• Neural Operators: In infinite-dimensional spaces (e.g., $L^2(D)$), operators of the form $F(u) = W u + K(u)$, with $W$ bijective, $K$ continuous and compact, and $\sigma$ a surjective activation, are surjective if the ‘shifted’ operator $u \mapsto \alpha u + W^{-1}K(u)$ is coercive for some $\alpha \in (0,1)$ (Furuya et al., 2023).
• Pre-LayerNorm and Modern Blocks: Neural mappings incorporating pre-layer normalization (LN) and residual connections, such as $g(x) = f(\text{LN}(x)) + x$, are surjective for every continuous $f$ via Brouwer or degree-theoretic arguments (Jiang et al., 26 Aug 2025).

2. Surjectivity in Piecewise Affine Neural Networks

ReLU networks partition their domain into polyhedral regions (the so-called polyhedral fan), on each of which the network acts as an affine map. The surjectivity of such networks is governed not by all local affine regions, but by the coherent orientation of those selection functions acting on unbounded regions of the input space (Radons, 2017):

Property	Sufficient Surjectivity Condition	Consequence
Piecewise affine $F$	$\text{sign}[\det D_xF]=\text{const}\neq 0$ on unbounded polyhedra	Network covers full codomain (no "dead zones")
Injectivity	Coherent orientation on all regions	Ensures unique preimages

A practical implication is that, despite the exponential number of affine pieces in deep ReLU networks, verifying surjectivity reduces to checking orientation on a tractably small subset (extreme rays/unbounded regions). This condition also impacts invertibility: a coherently oriented network behaves as a branched covering, governing the multiplicity of preimages and aiding robust inversion.

3. Surjectivity in Smooth and Infinite-Dimensional Neural Mappings

For networks with smooth activations or architectures acting on infinite-dimensional function spaces (neural operators), surjectivity is characterized differently:

• Critical Set Analysis: If a smooth neural network (with differentiable activations) has at least one regular point and the critical set—where the Jacobian is not full rank—has Hausdorff dimension less than $n-1$, then the network map is automatically surjective provided its image is closed (Shi et al., 2018). This provides a rigorous link between architecture-induced degeneracies and collapse of expressivity.
• Fredholm and Degree Theory for Neural Operators: For neural operators acting between function spaces, surjectivity relies on global analytic properties such as coercivity and fixed-point existence. The presence of bijective activations (e.g., leaky ReLU), together with compact perturbations and the Fredholm property of the linear part, enable the use of degree theory (Leray–Schauder) to guarantee surjectivity (Furuya et al., 2023). These conditions ensure invertibility and universal approximation for neural operators in inverse problems and function space generative models.

4. Algorithmic and Complexity-Theoretic Aspects

The verification of surjectivity, particularly for piecewise linear networks, presents significant computational hardness:

• Two-Layer ReLU Networks: For networks with one hidden ReLU layer and scalar output, surjectivity is equivalent to the existence of both positive and negative output values, which can be reframed as a dual problem of zonotope containment. This surjectivity verification is NP-complete, and the corresponding verification counterpart (guaranteeing output boundedness over an input set) is coNP-hard (Froese et al., 30 May 2024).

Problem	Complexity Class	Equivalent Formulation
$\varphi(x)>0$ ?	NP-complete	$\exists x$ such that $\varphi(x)>0$
Surjectivity	NP-hard	$\exists$ extreme rays $r^+,\ r^-$: $\varphi(r^+)>0$, $\varphi(r^-)<0$
Verification	coNP-hard	$\forall x$ in $S$, $\varphi(x)\leq t$?

For higher-dimensional and more complex architectures, the intractability calls for parameterized or heuristic approaches, especially when certifying global output properties in safety-critical systems.

5. Surjectivity in Modern Architectures: Transformers, Attention, and Generative Models

Recent work demonstrates that the fundamental blocks of contemporary neural models—particularly pre-layer normalization (Pre-LN) and linear attention/retention modules—are almost always surjective (Jiang et al., 26 Aug 2025):

• Pre-LayerNorm Residual Blocks: If $g(x) = f(\text{LN}(x)) + x$, $g$ is surjective for any continuous $f$. The proof utilizes the boundedness of $\text{LN}(x)$ and the Brouwer fixed point theorem to guarantee that every output is attainable.
• Linear Attention/Retention Layers: For mappings $f(x) = Mx + (x^\top N x)x$, surjectivity holds generically (except for a zero-measure set of $M, N$), by appealing to topological degree theory and homotopy invariance of the degree.
• Architectural Closure Under Composition: Surjectivity is preserved under composition, so that entire transformer stacks or multi-stage diffusion pipelines inherit the property.

Implications: Generative models built from these blocks (e.g., GPT, diffusion models) are structurally surjective, implying that any possible output can (in principle) be generated by some input vector. This has direct consequences for adversarial vulnerability and model safety: there is always some (adversarially constructed) prompt or noise vector that will yield any specified output, including harmful ones.

6. Feature Mapping Surjectivity in Physics-Informed Neural Networks

Surjectivity in the context of feature mapping layers—in particular, the use of Fourier features versus radial basis functions (RBFs) in PINNs—represents a trade-off between expressivity and generalizability (Zeng et al., 10 Feb 2024):

• Fourier Features: Maps of the form $\varphi(x)=\sin(2\pi \mathcal{B}x)$ are highly surjective with high probability, but prone to mapping many distinct $x$ to nearly identical outputs, resulting in overlap ("collapse") of the feature space. This impairs gradient diversity and can induce failure modes such as the Gibbs phenomenon or poor training convergence due to loss of input distinguishability.
• RBF Features: RBF mappings with tunable bandwidth produce more localized, less-overlapping feature representations ("more injective"), which alleviates these pathologies. Composing RBFs with Fourier features permits fine control of surjectivity and bandwidth, providing a mechanism to balance local expressivity and global generalization.
• Kernel Bandwidth Tuning: The kernel width ($\sigma$ or $\xi$) and composition hyperparameters (e.g., periodicity $\gamma$ in $RBF\cdot\cos(2\pi x/\gamma)$) control the degree of surjectivity and overlap in the induced feature space, with practical effects on the NTK spectrum and convergence rates.

Feature Mapping	Surjectivity Level	Training Implication
Fourier (sine)	High	Overlap, poor gradient spread
RBF (Gaussian)	Moderate/Low	Distinct features, better convergence
RBF × Fourier	Tunable	Trade-off; custom regularity

7. Safety, Invertibility, and Theoretical Implications

Surjectivity enables both the existence of inverses for generative/inverse-problem solvers and poses challenges for safety:

• Invertibility: For a network to be invertible, surjectivity is a necessary condition (alongside injectivity). The examination of coherent orientation, degree conditions, and critical set dimension provides constructive or diagnostic approaches for achieving invertible maps in practice (Radons, 2017, Furuya et al., 2023).
• AI Safety and Jailbreak Vulnerability: Modern architectures’ structural surjectivity precludes architectural constraints that prevent arbitrary outputs. Thus, even with extensive safety fine-tuning or output filtering, the ‘train-for-safety’ paradigm is vulnerable: adversarially designed inputs can always elicit target behaviors, as is empirically demonstrated in modern generative models (Jiang et al., 26 Aug 2025).
• Verification and Computational Barriers: The computational intractability of surjectivity verification in practical ReLU networks (NP- and coNP-hardness) underscores that theoretical guarantees are, in some cases, unattainable for real-world architectures, necessitating reliance on empirically motivated heuristics or restriction to parameterized subcases (Froese et al., 30 May 2024).

---

Surjectivity in neural networks is thus a multidisciplinary concept, intertwining nonlinear analysis, differential topology, convex geometry, operator theory, architectural analysis, and computational complexity. Contemporary architectures—especially those based on Pre-LN/residual stacks and attention modules—are structurally and generically surjective in both finite- and infinite-dimensional settings, enabling maximal expressivity and, simultaneously, presenting inherent safety and verification challenges.