Algorithmic Foundations of Deep Learning: Complexity-Theoretic Rates and a Characterization of Universal Approximation

Abstract: Feedforward neural network (NN) expressivity is typically studied by emulating optimal basis-expansion schemes. While powerful, this perspective is incomplete: it primarily captures complexity through regularity, and therefore does not distinguish intuitively simple and complicated objects with comparable regularity, such as the square-root function and a typical Brownian path. The guiding message is that neural networks should be viewed not only as flexible basis functions, but also as models of computation. If a function is computable by a real-valued circuit over a prescribed elementary gate language, then it can be computed to comparable accuracy by an NN with explicit depth, width, and non-zero-parameter bounds controlled by the depth, width, gate count, and gate structure. Thus, neural-network complexity is not governed by regularity alone, but also by algorithmic complexity. We then show that any definable NN model satisfying a natural parallelization condition, allowing possibly multivariate non-linearities such as attention or layer normalization, is a universal approximator if and only if it contains a non-affine nonlinearity. The scope of our theory is illustrated by deducing universal approximation guarantees for continuous functions, minimax-optimal approximation guarantees for Besov classes, logarithmic-error complexity for holomorphic functions, and by showing that NNs can emulate numerical algorithms such as Newton-Raphson root finding and power iteration without architecture-specific arguments. Its precision is illustrated by shortest-path computation on $k$-vertex graphs: compiling the tropical dynamic-programming circuit yields NNs with O(log(1/ε)) non-zero parameters, exponentially improving in 1/ε over the generic $O(ε^{-c k<sup>2})$ Lipschitz-approximation scale, for a constant c>0.

• The paper introduces a circuit-to-neural-network compilation theorem that quantifies approximation complexity in terms of circuit metrics.
• It demonstrates that any definable non-affine feedforward architecture achieves universal approximation and minimax-optimal rates.
• The results imply that neural networks can emulate classical algorithms with exponentially reduced complexity compared to Lipschitz-based bounds.

Algorithmic Complexity and Deep Learning: A Complexity-Theoretic Perspective on Universal Approximation

Introduction

The paper "Algorithmic Foundations of Deep Learning: Complexity-Theoretic Rates and a Characterization of Universal Approximation" (2606.26705) develops a rigorous foundation for neural network approximation theory grounded in circuit and computational complexity rather than traditional regularity-based analysis. Rather than viewing neural networks (NNs) as mere function approximation tools relying on smoothness or regularity of the target, the paper advances the perspective that expressivity and complexity should also be measured in terms of the algorithmic complexity required to compute the target function. This approach enables a more nuanced understanding of why certain functions are "hard" or "easy" for NNs to approximate, resolves key limitations of classical approximation theory, and leads to quantitative, architecture-agnostic results on NN complexity.

Limitations of Regularity-Based Approximation

Classical neural approximation theory typically evaluates the efficiency of NNs through the lens of function regularity: Hölder, Sobolev, Besov, or holomorphic smoothness, often using minimax approximation rates for entire regularity classes. For example, both $\sqrt{x}$ and a sample path of Brownian motion have the same Hölder exponent, and their NN approximation rates are predicted to be identical under regularity-based theory.

However, as illustrated in (Figure 1), the square-root function is algorithmically simple—teachable at the high-school level—whereas a Brownian path is algorithmically complex. While both fall within the same functional class, the compositional or circuit complexity is fundamentally different.

Figure 1: Two functions with identical $\tfrac{1}{2}$-Hölder regularity ($\sqrt{x}$ in green and a Brownian path in red) are vastly different in algorithmic complexity. NN approximation rates under classical regularity theory are nearly identical, but circuit-based analysis predicts $\mathcal{O}(\varepsilon^{-1})$ neurons for $\sqrt{x}$ versus $\mathcal{O}(\varepsilon^{-2})$ for the Brownian path.

Classical theory thus fails to distinguish "structurally easy" from "structurally hard" targets within the same regularity class. By introducing algorithmic complexity—for example, via the size and depth of a Boolean or real-valued circuit required to compute a function—the authors provide a more granular measure of hardness.

From Circuits to Neural Networks: Compilation and Complexity

The central technical advance is a quantitative circuit-to-neural-network compilation theorem: every function computable (to error $\varepsilon$) by a bounded-depth circuit over a defined gate set (arithmetic, tropical, radicals, etc.) can be approximated (to comparable accuracy) by a neural network, where the depth, width, and parameter count are controlled by circuit metrics.

Figure 3: A function constructed by a real-valued circuit (left, over a gate set $\mathbb{G}$) can be "compiled" into a neural network emulator (right, of architecture $\mathcal{A}$NN) with explicit depth/width bounds that reflect the underlying circuit's size and composition.

This shift leads to a "grey-box" (circuit-complexity-driven) phase between the "black-box" (pure regularity, no algorithmic insight) and "white-box" (full algorithmic knowledge) settings:

Figure 2: The black-box setting sees only the regularity of input/output, the white-box knows the entire computation, and the grey-box leverages complexity information at the circuit level. This gap is bridged by the compilation theorem.

Main Quantitative Result:

Let $f$ be computed to accuracy $\tfrac{1}{2}$0 by a size-$\tfrac{1}{2}$1, depth-$\tfrac{1}{2}$2 circuit over language $\tfrac{1}{2}$3. Then there exists a neural network of depth $\tfrac{1}{2}$4 and parameter count $\tfrac{1}{2}$5 (for $\tfrac{1}{2}$6-ary gates) achieving comparable error, with more elaborate bounds for tropical, rational, or radical gate classes. Notable implications are:

• All non-affine feedforward architectures are universal approximators: The classical barrier of "nonpolynomial activation" is replaced by "non-affine nonlinearity" for general architectures, including those with multivariate nonlinearities and operations such as attention or normalization.
• Minimax-optimal rates are generically achieved: Any architecture conforming to the o-minimal, non-affine condition automatically achieves minimax rates for classical regularity classes (e.g., Besov, holomorphic).
• Algorithmic structure is leveraged: Problems such as All-Pairs Shortest Paths (APSP), which admit $\tfrac{1}{2}$7-size circuits, yield exponentially smaller NNs than Lipschitz-based approximation would predict. See (Figure 4).

  Figure 4: APSP neural compilation. The classical Lipschitz-based upper bound for the APSP map is exponential in $\tfrac{1}{2}$8, but the compilation theorem gives $\tfrac{1}{2}$9 size, exploiting tropical circuit structure.

Universality and o-Minimal Definability

A central theoretical contribution is a precise, if and only if characterization of universality:

Any definable (in an o-minimal structure) feedforward architecture is universal in $\sqrt{x}$0 if and only if it contains a non-affine nonlinearity.

This result generalizes and improves the shallow MLP characterization (requiring nonpolynomial univariate activation), and extends it to deep and modern architectures (transformers, ResNet, etc.), as long as their nonlinearities are definable (excluding, for instance, the use of pathological activations or stochastic processes).

The proof uses model theory and real algebraic geometry, leveraging the structural regularity preserved by o-minimality to guarantee the ability to "synthesize" multiplications and thus, by extension, simulate arbitrary circuits.

Technical Methodology: Abstract Surgery

The compilation theorem is enabled by an "abstract surgery" method, which recursively:

1. Emulates each circuit gate by a small NN ("expert gate emulation").
2. Aligns the computational graph with a uniform layered NN architecture.
3. Composes these approximations using network parallelization and identity padding to ensure depth alignment.

Figure 6: Demonstrating abstract surgery, where each gate in a canonicalized circuit is replaced by a neural network emulator, yielding a new NN that mirrors the circuit's structure.

The method provides explicit counts for depth, width, and parameter complexity that precisely track the circuit's structure and the approximation scale $\sqrt{x}$1.

Implications and Numerical Results

The implications are both theoretical and practical:

• Unified phase diagram: Network complexity scales according to both regularity and circuit complexity, yielding a matrix of scaling laws covering both well-structured algorithms and minimax regularity classes (as exhibited in provided tables).
• Algorithm emulation: NNs can efficiently emulate the Newton-Raphson method, power iteration, ODE solvers, and tropical dynamic programming algorithms, at demonstrably optimal complexities, without requiring architecture-specific proofs.
• No architectural bias for approximation: As soon as non-affinity is present, all definable architectures are universal and minimax-optimal; further architectural modifications should be justified by inductive bias, geometry, or optimization traits, not by approximation ability.
• Practical scaling: In realistic algorithmic tasks, neural networks can be exponentially smaller than uniform-Lipschitz-based theory would predict, as in APSP or sparse polynomials.

Conclusion and Future Directions

The complexity-theoretic paradigm advanced in this paper broadens the theoretical foundation of deep learning and connects expressivity to classical models of computation and circuit complexity, offering both a unification and generalization of prior regularity-based approaches.

Possible future work includes:

• Complexity Separations: Investigating architectures or gate sets for which neural networks are strictly more efficient than any circuit model with a restricted gate set, using, e.g., Lipschitz widths or manifold widths.
• Neural Compilation Algorithms: Automated "NN compilers" that convert pseudocode or explicit circuits into concrete neural architectures, making the theory constructive for engineering practice.

The framework developed provides robust, general, and quantitative theoretical underpinnings for the practice of deep learning as a form of algorithmic synthesis, precisely quantifying the role of compositional and algorithmic structure in neural expressivity.

• "Algorithmic Foundations of Deep Learning: Complexity-Theoretic Rates and a Characterization of Universal Approximation" (2606.26705)
• Model-theoretic and o-minimal background [vanDenDries1998TameTopology]
• Prior results in circuit complexity [jukna2023tropical], minimax rates [devore1993besov], and universality [PinkusOGpaper_1999]

---

This summary is intended for technical experts interested in neural approximation theory, real computation, and complexity, and omits introductory explanations and generalist context in favor of technical detail and explicitness.