Expressive Power of Floating-Point Neural Networks with Arbitrary Reduction Orders and Inexact Activation Implementations

Abstract: Most existing expressivity theories for neural networks assume exact real arithmetic, whereas practical neural networks are executed under finite-precision floating-point arithmetic with implementation-dependent execution semantics. Recent works have begun studying the expressive power of floating-point neural networks, but existing results are limited to highly restricted activation functions and idealized assumptions such as fixed left-to-right reduction orders and correctly rounded activation implementations. In this work, we study the expressive power of floating-point neural networks under generalized floating-point execution semantics, including arbitrary reduction orders and inexact activation implementations with bounded ulp errors. We investigate when floating-point neural networks can represent arbitrary functions between floating-point domains exactly. To this end, we introduce a general distinguishability framework and show that the ability to distinguish every pair of distinct inputs in the first layer is necessary for universal representability. This characterization yields broad classes of activation implementations that are not universal representators, extending previous isolated counterexamples such as the correctly rounded cosine activation. We further prove that a suitable form of distinguishability is also sufficient for universal representability under mild conditions on the activation implementation. Using this framework, we establish universal representability results for a broad class of practical activation functions, including implementations of $\mathrm{Sigmoid}$, $\tanh$, $\mathrm{ReLU}$, $\mathrm{ELU}$, $\mathrm{SeLU}$, $\mathrm{GeLU}$, $\mathrm{Swish}$, $\mathrm{Mish}$, and $\sin$, under significantly more realistic floating-point execution models than previously known.

• The paper introduces a necessary and sufficient distinguishability criterion for universal representability of FP neural networks under arbitrary reduction orders.
• It demonstrates that carefully constructed separating affine maps enable practical activations to preserve expressivity despite bounded-ULP rounding errors.
• The findings highlight that even with finite precision, neural networks can achieve nontrivial universal function representation, influencing hardware and design choices.

Expressive Power of Floating-Point Neural Networks under Arbitrary Reduction Order and Inexact Activations

Introduction and Context

This paper ("Expressive Power of Floating-Point Neural Networks with Arbitrary Reduction Orders and Inexact Activation Implementations" (2605.28704)) addresses foundational questions concerning the expressive power of neural networks executed using finite-precision floating-point (FP) arithmetic, focusing specifically on: (i) arbitrary reduction order in affine transformations and (ii) non-ideal activation implementations (including bounded-ULP error). Classical universal approximation theorems for neural networks with non-polynomial activations rely on real arithmetic. However, practical NNs operate over a finite and nonlinear set $\mathbb{F}_{p,q}$ of machine representable FP numbers, where every arithmetic operation is susceptible to rounding errors, and network execution semantics are highly implementation dependent.

Previous analyses of FP neural networks either assume exact activation rounding and/or a particular (e.g., left-to-right) reduction order for summations, and have been largely restricted to piecewise-linear activations such as ReLU. The present work removes these constraints, delivering a unified characterization of universal representability for FP neural networks across a broad class of activation implementations and execution semantics.

Main Theoretical Contributions

The central technical result is a necessary and sufficient (under mild conditions) distinguishability criterion for universal representability of FP NNs, where “universal representability” means the capacity to exactly realize every mapping $\mathcal{D} \to \overline{\mathbb{F}}_{p,q}$ for subsets $\mathcal{D} \subset \mathbb{F}_{p,q}^d$.

1. General Distinguishability Condition: Universal representability for FP $\sigma$-networks requires that for any $x \neq x'$ in $\mathcal{D}$, there exists a FP affine map (under the given reduction order $S$) such that $\sigma(\phi(x)) \neq \sigma(\phi(x'))$. This condition significantly generalizes earlier work and is reduction-order agnostic.
2. Sufficiency under Realistic Constraints: The paper proves that, provided the implemented activation admits a form of “range coverage” (a small set of target values, e.g., via well-chosen codesigns as in Condition 1 of the manuscript), this distinguishability is also sufficient. Thus, for a broad class of practical activations (including inexact, non-piecewise-linear, and smooth ones), and for all reduction orders, FP NNs can represent all possible FP-valued functions on a correspondingly large domain.

   Figure 1: Visualization of the conditions in Lemma 4.13, the main distinguishability criterion for universal representability with real-valued activations.

3. Inexact Activation Analysis: The authors extend the framework beyond the (often impractical) assumption of correct rounding, allowing bounded-ULP deviations between the activation implementation and the real-valued function, and provide sufficient conditions for expressivity preservation under inexact activations.
4. Counterexamples and Negative Results: As a byproduct, they systematically cover classes of activations (e.g., correctly-rounded $\cos$, $1 + x^2$, $\mathcal{D} \to \overline{\mathbb{F}}_{p,q}$0, etc.) which, even under any reduction order, cannot satisfy the distinguishability criterion for small enough input separations; universal representability fails for these networks.

Methodological Framework

The proof approach exploits explicit construction of separating affine maps that, after floating-point reduction and activation, can differentiate any two FP input points even in the presence of arbitrary reduction semantics. The key insight is that, because all layer computations (affine and nonlinear) are strictly over $\mathcal{D} \to \overline{\mathbb{F}}_{p,q}$1 with non-associative, non-distributive operations, carefully controlling bit-level output dynamics under composition is crucial.

The sufficiency proofs hinge on an induction over the FP representation structure, leveraging the fact that the set of values output by a single-layer affine transformation (with arbitrary reduction order and floating-point rounding) is itself a finite subset of $\mathcal{D} \to \overline{\mathbb{F}}_{p,q}$2 whose properties can be analyzed combinatorially. The existence of separating points (cf. Definition 4.14) is shown for activation implementations with moderate monotonicity and covering properties.

Figure 2: Visualization of the conditions in Lemma 4.19, demonstrating handling of bounded-ULP inexact activations and separation construction.

Implications and Theoretical Significance

On the Fundamental Gap between Theory and Practice

The main results bridge the gap between classical universal approximation heuristics (which ignore computational precision and reduction ordering) and the realities of deployed FP NNs. They show that, for all widely used activation implementations in deep learning (including $\mathcal{D} \to \overline{\mathbb{F}}_{p,q}$3, $\mathcal{D} \to \overline{\mathbb{F}}_{p,q}$4, $\mathcal{D} \to \overline{\mathbb{F}}_{p,q}$5, $\mathcal{D} \to \overline{\mathbb{F}}_{p,q}$6, $\mathcal{D} \to \overline{\mathbb{F}}_{p,q}$7, $\mathcal{D} \to \overline{\mathbb{F}}_{p,q}$8, $\mathcal{D} \to \overline{\mathbb{F}}_{p,q}$9, $\mathcal{D} \subset \mathbb{F}_{p,q}^d$0, and $\mathcal{D} \subset \mathbb{F}_{p,q}^d$1) and for all practical FP formats (float16, bfloat16, float32, etc.), exact function representation is achievable under realistic execution assumptions.

• Arbitrary Reduction Order: Expressive power is not contingent on associativity or a fixed summation convention (in direct contrast to claims in earlier literature), which is particularly relevant for highly parallel hardware.
• Activation Implementation Robustness: Expressivity is robust to moderate inexactness (ULP-level errors), explaining the empirical success of high-level NN architectures across diverse software and hardware stacks with only partial IEEE-754 compliance.

On Practical Neural Network Design

The distinguishability criterion provides a diagnostic tool for practitioners to assess whether a given composition of architecture, FP format, reduction order, and activation implementation meets the requirements for full mapping universality. For instance, using correctly-rounded activations with sufficiently large output ranges and monotonic intervals guarantees universal representability, while using non-distinguishing activations (e.g., FP $\mathcal{D} \subset \mathbb{F}_{p,q}^d$2 or similar) strictly prevents universality.

On the Nature of Expressivity under Finite Precision

The finding that even identity-activated FP NNs can be universal function representors (in contrast to their real-activation counterparts, which can only represent affine transformations) highlights the nontrivial, inherently nonlinear nature of floating-point arithmetic as a computational substrate.

Quantitative and Numerical Claims

• Layer Depth: The paper constructs networks of depth at most 4 layers for sufficiency, i.e., every function $\mathcal{D} \subset \mathbb{F}_{p,q}^d$3 for $\mathcal{D} \subset \mathbb{F}_{p,q}^d$4 large can be realized by a 4-layer FP NN with practical activations.
• Activation Coverage: Explicit verification for all activations listed in Corollary 4.22, with domain intervals dependent on activation monotonicity, output covering, and floating-point granularity.
• Negative Case Construction: Systematic exclusion of expressivity for activating functions meeting the unusual flatness or small-derivative requirements in the relevant (small-gap) intervals.

Future Research Directions

• Width Constraints: The current framework establishes representability, not notionally minimal architectural resources; analyzing the minimal width and depth for given $\mathcal{D} \subset \mathbb{F}_{p,q}^d$5 remains open.
• Training Implications: While function representability is assured, optimization under finite-precision is nontrivial—future work could integrate expressivity with learnability and robustness analyses.
• Low-Precision/Custom Formats: As hardware continues to drive toward 8-bit and adaptive-precision arithmetic, further exploration of expressivity boundaries in extremely quantized/limited-exponent domains is warranted.
• Generalization to Stochastic/Probabilistic Rounding and Other Hardware-induced Errors: The analysis could be extended to scenarios beyond deterministic ULP-bound deviations or fully specified reduction orders.

Conclusion

This work establishes a rigorous, reduction-order- and rounding-robust theory for the expressive power of finite-precision floating-point neural networks with practical activation implementations. It offers not only a precise characterization of universality for FP NNs, but also a framework extensible to emerging hardware settings and nonstandard activation design. The proposed distinguishability criterion and its constructive realization mark a significant advance in the theoretical understanding of neural computation under realistic semantics.