On Explicit Super-Expressive Approximation for Neural Networks

Published 7 Jul 2026 in cs.LG | (2607.06781v1)

Abstract: In this work, we investigate the fixed-architecture neural network approximation with explicit parameter bounds and elementary activations. While prior work demonstrated super-expressive approximation using fixed-size networks, they lack quantitative and non-asymptotic characterizations of parameter magnitude with respect to the approximation error. We resolve this issue by introducing the Chinese Remainder Theorem as a constructive encoding mechanism. For Lipschitz continuous functions on $[0,1]<sup>D$ , we construct a width- $\max{D,4}$ , depth-$5$ network with explicit parameter-error trade-offs. For Hölder-smooth functions in $C<sup>{r,γ}_A\left([0,1]<sup>D\right)$ , our fixed network of width $\max{2D,\ D+5N+1}$ and depth $r + 9$ achieves the parameter magnitude $\mathcal{P}$ bounded by $\log_2 \mathcal{P}=\mathcal{O}\bigl(\varepsilon<sup>{-2D/(r+γ)}\log(1/\varepsilon)\bigr)$ . This is the dual result compared to those in the parameter-bounded and architecture-unbounded paradigm.

Abstract PDF Upgrade to Chat

Authors (4)

Summary

The paper introduces an explicit, non-asymptotic scaling law linking neural network parameter magnitudes to approximation error using super-expressive activations.
It employs the Chinese Remainder Theorem for encoding grid partitions, enabling fixed-depth and fixed-width networks to approximate Lipschitz and Hölder-smooth functions with quantifiable precision.
The work bridges classical approximation theory with practical network design by detailing precise parameter trade-offs and novel activation engineering for ultra-high expressivity.

Explicit Super-Expressive Approximation in Fixed-Architecture Neural Networks

Introduction

The paper "On Explicit Super-Expressive Approximation for Neural Networks" (2607.06781) addresses the fundamental theory of function approximation by neural networks, focusing on explicit quantitative characterizations of parameter magnitudes in fixed-architecture networks deploying super-expressive activations. Classical results in neural network approximation theory demonstrate universality—that is, the capacity of neural architectures to approximate continuous functions to arbitrary accuracy given sufficient parameterization. However, the transition from asymptotic or existential results to explicit, non-asymptotic bounds on parameter growth as a function of approximation error has been elusive—especially in the context of super-expressive approximation with fixed network architecture and elementary activation functions.

This work pioneers a constructive approach for the explicit relationship between parameter magnitude and approximation error in such fixed-width, fixed-depth networks. Central to the authors' construction is the use of the Chinese Remainder Theorem (CRT), enabling direct and efficient encoding/decoding schemes for partitioned domains and quantized targets. These results fill a clear gap in the landscape by providing, for the first time, direct scaling laws quantifying the trade-off between parameter magnitudes and target accuracy for super-expressive architectures.

Background and Motivation

Traditional universal approximation theorems affirm that neural networks with increasing capacity (width or depth) can densely approximate broad classes of functions [yarotsky2018Optimal, hornik1989universal]. However, the required network size often scales poorly with the inverse of the target error, especially for higher-dimensional, less regular functions. The paradigm of super-expressive approximation, utilizing exotic or composite activation functions, upends this by allowing the architecture to remain fixed—as the approximation error vanishes, all expressivity is transferred to the parameters.

While earlier results (e.g., [maiorov1999lower, yarotsky2021elementary]) established the existence of super-expressive networks with fixed architectures, these typically either (1) provided only asymptotic or non-explicit constructions, (2) required complex or non-elementary activations, or (3) supplied parameter-magnitude-expressiveness trade-offs not quantified as a function of error with explicitly computable constants. This left unresolved whether one could construct fixed-architecture, elementary-activation networks whose parameter magnitude increases in a precisely controlled and non-asymptotic manner as the required accuracy increases.

Main Results and Theoretical Contributions

Super-Expressive Approximation for Lipschitz Functions

The primary result for Lipschitz targets asserts that for any $f \in \mathrm{Lip}_A([0,1]^D)$ , a depth-5, width- $\max\{D,4\}$ neural network with activations $\{\rho_{floor}, \rho_1, \rho_2, \rho_{inv}\}$ achieves approximation accuracy $\varepsilon$ with parameter magnitudes explicitly bounded by

$\log_2 \mathcal{P} \leq O\big(\varepsilon^{-2D} \log(1/\varepsilon)\big).$

This construction relies on partitioning the input domain into quantization grids, discretizing the target, and assigning each cell a unique residue code using CRT. The congruence system is realized within the network by composing floor, reciprocal, and rectified power unit activations, yielding an explicit encoding/decoding map between grid cells and quantization values.

Figure 1: Visualization of the floor activation $\rho_{floor}$ —a key primitive in the integer extraction modules of the network.

Figure 2: Schematic pipeline for the CRT-based encoding and network architecture that realizes Theorem 1, showing grid partitioning, quantization, and mapping via residue systems.

Key claims:

This is the first explicit, non-asymptotic parameter scaling law for super-expressive fixed networks approximating Lipschitz functions.
The parameter magnitude bound is exponential in dimension and polynomial in $1/\varepsilon$ , consistent with the "curse of dimensionality" but now fully quantified for fixed-architecture super-expressive regimes.

Super-Expressive Approximation for Hölder-Smooth Functions

Extending the approach to smoother classes, the network approximates any $f \in C_A^{r,\gamma}([0,1]^D)$ using a width $\max\{2D, D+5N+1\}$ , depth $r+9$ fixed architecture ( $\max\{D,4\}$ 0), with the parameter magnitude scaling as

$\max\{D,4\}$ 1

The construction exploits gridwise polynomial Taylor approximations for local accuracy, rationalizing coefficients with finite bit-length (quantified using algorithmic complexity theory), and uses CRT to encode the rational coefficients of all grid-local polynomials into the network's parameters.

Figure 3: Scaling law for the parameter magnitude versus target accuracy, highlighting the improved exponent for smoother function classes.

Figure 4: Architecture of the CRT-enabled fixed network realizing general gridwise polynomial surrogates.

Salient features:

The scaling law's exponent improves as the regularity $\max\{D,4\}$ 2 increases, demonstrating an explicit benefit from target function smoothness.
The result is fully explicit: all parameter sizes, layer counts, and activation choices are specified, with each bit-cost and coefficient rationalization computed.
Duality to classical paradigm: Instead of trading network size/architecture for error, this result completely fixes the architecture and trades parameter magnitude, thereby "completing the picture" in super-expressive approximation.

Technical Approach

The methodology is distinguished by its constructive, stepwise CRT-encoding process:

Partition and quantization: The domain $\max\{D,4\}$ 3 is partitioned into $\max\{D,4\}$ 4 uniform grids; the target is quantized into $\max\{D,4\}$ 5 integer bins.
Residue encoding via CRT: Pairwise coprime moduli are assigned to each grid cell; the unique congruence solution associates each cell with its quantized value.
Network realization: Linear-affine transformations compute grid indices, floor activations extract integer grid coordinates, $\max\{D,4\}$ 6 computes reciprocals, and $\max\{D,4\}$ 7 enables multiplication.
Parameter-bound analysis: Every parameter is explicitly tracked throughout the construction. For instance, the maximum CRT modulus and its effects on overall parameter magnitude are calculated.
Gridwise polynomial encoding: For Hölder-smooth functions, rational Taylor surrogates are constructed on each grid cell, then simultaneously encoded into the parameter vector—allowing recovery of all polynomial coefficients with a single fixed-size network.

Numerical and Qualitative Implications

Scaling behavior: The derived scaling laws (see Figure 3) concretely demonstrate that while fixed-architecture, super-expressive networks require parameter magnitudes that grow rapidly with error, higher smoothness manifests as a reduced exponent, reflecting known upper bounds from non-super-expressive approximation theory but now realized in the fixed-architecture regime.
Activation engineering: The work utilizes only elementary nonlinearity primitives (e.g., floor, RePU, reciprocal) rather than highly engineered, exotic, or nonconstructive activations, situating it closer to potential implementability.
Explicit structure: Modular, layered construction allows for direct analysis and possible extensions or optimizations in magnitude scaling.

Theoretical and Practical Implications

Theoretical impact:

Forms the explicit dual to architecture-unbounded, parameter-bounded approximation schemes, providing a full quantifiable spectrum for trade-offs between error, architecture, and parameter magnitude.
Supplies a constructive alternative to Diophantine/Kolmogorov-based existence proofs, enabling the analysis of capacity, memorization, and expressivity in resource-constrained or hardware-limited regimes.
The CRT-based encoding scheme creates a robust bridge from classical integer coding theory and computational complexity into neural approximation analysis.

Potential practical implications:

Although parameter magnitudes can be extremely large (potentially super-exponential in $\max\{D,4\}$ 8 in high dimension), the method establishes what is theoretically possible in fixed-size, ultra-high-capacity parameterizations—suggestive of regimes with quantized, integer, or low-width/high-depth applications (e.g., on specific ASIC hardware).
The explicit compositional structure could inform energy-efficient or quantization-aware network design, where only parameter scaling—not reallocation of architectural resources—is permitted.

Limitations and Future Work

Redundant or extreme parameterization: The absolute size of parameter magnitudes required may make direct computational implementation impractical outside certain theoretical or hardware-constrained scenarios.
Activation dependence: The necessity for the floor and reciprocal functions as activation primitives restricts the approach from immediate transfer to standard architectures with common activations (e.g., ReLU or GELU).
Potential for constructional efficiency: There may be room to sharpen the modularity of the CRT encoding, reduce redundancy, or exploit alternative integer encoding theories to ease parameter growth.

Future directions include:

Tightening parameter magnitude bounds by optimizing the CRT construction or exploring more efficient multi-valued residue systems.
Investigating the expressivity–trainability trade-off: establishing whether these networks can be efficiently trained in practice rather than merely constructed.
Examining the extension to other activation families, extending results to more complex function spaces (e.g., Sobolev, Besov), and integrating with neural operator frameworks.

Conclusion

This paper makes a significant advance in the explicit, quantitative theory of neural network approximation by constructing fixed-architecture, elementary-activation networks whose parameter magnitudes are precisely characterized as a function of approximation error. The use of CRT as a parameter-encoding mechanism enables the establishment of scaling laws for parameter magnitude in both Lipschitz and Hölder-smooth target regimes. The duality to architecture-unbounded schemes completes the expressivity landscape for neural networks, providing a blueprint for future theoretical exploration and informing the design of networks in parameter-constrained environments.

References

[maiorov1999lower]: Lower Bounds for Approximation by MLP Neural Networks
[yarotsky2021elementary]: Elementary Superexpressive Activations
[shen2022fixed]: Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons
[beknazaryan2022neural]: Neural networks with superexpressive activations and integer weights
(2607.06781): On Explicit Super-Expressive Approximation for Neural Networks