On efficiently computable functions, deep networks and sparse compositionality (2510.11942v1)

Abstract: We show that \emph{efficient Turing computability} at any fixed input/output precision implies the existence of \emph{compositionally sparse} (bounded-fan-in, polynomial-size) DAG representations and of corresponding neural approximants achieving the target precision. Concretely: if $f:[0,1]^d\to\R^m$ is computable in time polynomial in the bit-depths, then for every pair of precisions $(n,m_{\mathrm{out}})$ there exists a bounded-fan-in Boolean circuit of size and depth $\poly(n+m_{\mathrm{out}})$ computing the discretized map; replacing each gate by a constant-size neural emulator yields a deep network of size/depth $\poly(n+m_{\mathrm{out}})$ that achieves accuracy $\varepsilon=2^{-m_{\mathrm{out}}}$. We also relate these constructions to compositional approximation rates \cite{MhaskarPoggio2016b,poggio_deep_shallow_2017,Poggio2017,Poggio2023HowDS} and to optimization viewed as hierarchical search over sparse structures.

• The paper’s main contribution is showing that efficiently computable functions can be represented as compositionally sparse DAGs, enabling effective neural approximations.
• It details an algorithmic transformation from polynomial-time Turing computations to bounded-fan-in Boolean circuits and neural subnetworks.
• The study underscores practical optimization advantages by reducing exponential search complexity to polynomial via a sparse compositional structure.

Efficient Turing Computability and Compositional Networks

Introduction

The paper "On efficiently computable functions, deep networks, and sparse compositionality" addresses the interplay between efficient computability and compositional sparsity in the context of deep learning. It establishes that efficiently computable functions can be represented as compositionally sparse DAGs, leading to efficient neural network approximations with bounded local complexity.

Setup: Precision, Discretization, and Families

To relate continuous functions to discrete computational models, the paper introduces quantization methods for input/output precision. A key concept is the discrete map $F_{n,m_{\mathrm{out}}$, which acts on quantized inputs to approximate real-valued functions. Efficient Turing computability is defined in terms of polynomial-time computability at any precision $(n, m_{\mathrm{out}})$.

Discrete Family and Sparse Representation

Efficiently computable functions possess discrete representation families $\{F_{n,m_{\mathrm{out}}}$ that are compositionally sparse. Such families are characterized by DAGs with bounded local arity, scaling polynomially with the precision parameters, thereby avoiding the curse of dimensionality.

From Polynomial-Time Computability to Bounded-Fan-In Circuits

Efficiently computable functions can be simulated by polynomial-size Boolean circuits. The construction involves transforming Turing machine computations into circuits with bounded fan-in, reflecting the local update rules. This transformation is algorithmic, providing P-uniform circuit families crucial for structured representation.

Bounded-Fan-In Circuits

For any efficiently computable function, corresponding Boolean circuits with bounded fan-in can be constructed. These circuits have depth and size constraints proportional to the input/output bit-depths, ensuring compositionally sparse DAG representations.

Neural Emulation at Fixed Precision

The paper demonstrates how Boolean circuits can be emulated by neural networks at fixed precision. Each Boolean gate is replaced by a neural subnetwork, maintaining the compositional sparsity and achieving the desired precision through careful error propagation control.

Neural Network Construction and Error Management

Neural subnetworks are used to emulate logic gates with standard activations, ensuring compositional sparsity is preserved. The resultant networks have size and depth that grow polynomially with precision, supporting efficient approximation within specified accuracy bounds.

Relation to Compositional Approximation and Autoregressive Universality

Efficient Turing computability implies compositional sparsity, aligning with established results on approximation theory. The compositional structure enables efficient neural approximations, enhancing training dynamics. Autoregressive universality complements this by showing dataset existences where token predictors achieve universal approximation.

Compositional Advantages in Optimization

Compositional sparsity offers significant optimization advantages: hierarchical procedures reduce search complexity from exponential to polynomial. This brings practical benefits in terms of efficient model training and deployment in real-world applications.

Boolean vs. Real: From Discrete to Smooth Networks

The transition from discrete computation to real-valued functions involves smooth lifting of Boolean circuits, maintaining the sparse structure. This allows efficiently computable functions to leverage deep networks for smooth approximation, bridging the gap between logical and continuous models.

Conclusion

The paper provides a robust framework linking efficient computability to structured, sparse neural representations. This connection supports efficient approximation, optimization, and potential universality, aligning computational and learning theories with practical deep learning applications.