Recurrent neural networks approximate continuous functions

Abstract: Classical approximation theorems ask for a new neural network whenever the target accuracy is improved. This paper studies the opposite possibility: can the network be chosen once and for all, and can accuracy be bought only by letting it run longer? We prove that this is possible for every continuous function on [-1,1]. More precisely, each such function is uniformly approximated by the time evolution of a single ReLU recurrent neural network with fixed weights and fixed hidden dimension. The mechanism behind the construction is a new intermediate model, the Turing machine with neural units (TMNU). This model retains the algorithmic freedom needed to implement polynomial approximation schemes, while remaining rigid enough to be simulated by RNNs with explicit bounds on hidden dimension and weight magnitude. The resulting convergence rates reflect the underlying polynomial approximation rates. We complement the construction with minimax lower bounds showing that runtime is not merely a proof artifact, but an unavoidable resource in this fixed-network approximation paradigm.

• The paper demonstrates that fixed-weight ReLU RNNs, with finite width, can approximate any continuous function by increasing runtime.
• It introduces the TMNU model to construct and simulate piecewise-linear arithmetic routines, yielding explicit convergence rates based on function smoothness.
• The work reveals runtime—not network size—is the key resource for accuracy, influencing the design of efficient, computation-centered neural architectures.

Recurrent Neural Networks as Universal Continuous Function Approximators

Introduction

This paper analyzes the functional approximation paradigm for Recurrent Neural Networks (RNNs), recasting classical neural network approximation theorems through a computational lens whereby the network architecture and parameters are fixed independently of the target accuracy. Instead, the degree of approximation is controlled by runtime—namely, the number of recurrent steps. The core result establishes that for every continuous function $f:[-1,1]\to\mathbb{R}$, there exists a single fixed-weight ReLU RNN, of finite width and weight magnitude, whose iterates uniformly approximate $f$ as $t\to\infty$, i.e., the only variable resource is computation time. This reverses the standard quantifier order in classical results, situating time, rather than network size or complexity, as the fundamental asymptotic resource.

Approximation Paradigms and Theoretical Framework

Classical network approximation theory generally takes the form: for every target function $f$ and prescribed tolerance $\epsilon > 0$, there exists a parameterized network $\mathcal{N}_\epsilon$ such that $\|\mathcal{N}_\epsilon - f\|_\infty \leq \epsilon$. Thus, stringency in accuracy necessitates a new architecture and set of weights.

Following the quantifier reversal paradigm introduced in [14], the present work studies whether, for every $f$, one can select a single network $\mathcal{N}_f$ (with fixed architecture and parameter magnitudes, independent of $\epsilon$) such that, for any desired error, increasing the computational runtime $f$0 yields an iterate $f$1 satisfying $f$2, once $f$3 exceeds a threshold depending on $f$4. This paradigm is not merely an immediate consequence of density theorems, as naive constructions would require infinite-dimensional state to accommodate all target functions. Instead, the key technical obstacle lies in achieving uniform boundedness of the approximation mechanism within finite resources.

Turing Machines With Neural Units: TMNU Model

A central conceptual and proof innovation is the introduction of the Turing Machine with Neural Units (TMNU) as an intermediate computational model, synthesizing algorithmic flexibility with real-valued computation. TMNUs comprise:

• A classical finite-state Turing machine (discrete state and tape)
• A finite-dimensional real-valued "neural" register, manipulated only via affine and ReLU operations

TMNUs act as readily-sequentialized, programmatically specifiable devices manipulating real variables through piecewise-linear (specifically, ReLU) mapping sequences dictated by the tape/controller logic. They are chosen for their equivalence (under boundedness) to RNN computations, while retaining the advantageous compositional structure of algorithmic models. The authors prove that every uniformly bounded TMNU can be simulated efficiently by an RNN with explicit bounds on hidden state and parameter magnitude. This equivalence is crucial: to prove the existence and boundedness of universal approximators implemented as RNNs, one can instead construct explicit TMNUs for the approximation task.

Main Results

Universal Approximation with Fixed RNNs

The authors rigorously prove that for any $f$5, there exists a ReLU RNN $f$6 with:

• Hidden state dimension $f$7
• Weight/bias magnitude $f$8 controlled explicitly by properties of $f$9

such that:

$t\to\infty$0

where $t\to\infty$1 denotes the prescribed (essentially Dirac) sequence input mechanism. This construction is fundamentally reliant on the Weierstrass theorem, sequentially encoding increasingly precise polynomial (Chebyshev) approximation instructions onto the TMNU tape, implemented through a modular subroutine hierarchy in the TMNU architecture and then transferred to the RNN.

Explicit Convergence Rates

The convergence of the RNN iterates to $t\to\infty$2 is quantitatively controlled via the smoothness properties of $t\to\infty$3, as analyzed through its Chebyshev expansion. For polynomials, the error decays at an exponential rate in computation time. More generally, for Dini-Lipschitz continuous functions (including all Lipschitz/Hölder/smooth functions), the rate of approximation with respect to runtime $t\to\infty$4 is matched to the Chebyshev partial sum convergence rates. For example:

• For $t\to\infty$5: approximation error $t\to\infty$6
• For $t\to\infty$7 with $t\to\infty$8 continuous derivatives: $t\to\infty$9
• For analytic $f$0: error decays exponentially in $f$1

The results provide explicit choices of degree/precision tradeoff schedules to optimize the rate for any function in these classes.

Minimax Lower Bounds

The authors establish nearly tight lower bounds (up to log-factors) on the decay of the worst-case approximation error for any RNN of fixed size and parameter bounds, via information-theoretic (metric entropy) arguments. Hence, runtime emerges as the unique asymptotically tunable resource in this "fixed hardware, arbitrary accuracy" paradigm. For many function classes, the exponential rates achieved are shown to be optimal.

Construction and Simulation Details

The proof techniques combine explicit construction of piecewise-linear (ReLU) arithmetic routines, algorithmic subroutine chaining (by exploiting the modular structure of the TMNU), and careful analysis of input/output encodings to maintain uniform boundedness throughout all intermediate computations. The tape of the TMNU encodes (in finite blocks) polynomial coefficients and degree/precision parameters in prefix-free binary code, and the neural register carries the input and intermediate values. By organizing the TMNU evaluation as a nested subroutine hierarchy, all requisite arithmetic (affine, multiplication, accumulator logic) is built from scratch, with each level's invariants (accuracy, boundedness, resource usage) propagated analytically.

Simulation of the TMNU by an RNN is effected via a systematic procedure: encoding all tape/state variables into a vector, realizing each transition (reading, writing, shifting, command) as a composed sequence of affine plus ReLU transformations, and iterating these via the recurrent operator. Explicit bounds on required dimension and parameter magnitude flow from subroutine counts and from global magnitude calculations.

Practical and Theoretical Implications

This work has direct relevance to the study of RNN expressivity, learning-to-compute paradigms, and neural algorithmic reasoning research. It repositions runtime—rather than width or depth—as the primary axis along which functional expressivity is modulated, thus providing a formal justification for fixed-weight, fixed-architecture computation-centric approaches (relevant for resource-constrained, on-device, or hardware-optimized deployment). The technical apparatus introduced (TMNU model, careful encoding schemes, modular construction proofs) is likely transferable to other settings: e.g., different domains, input/output modalities, or activation functions. The lower-bound results inform architectural and algorithmic limitations for time-unfolded computation in RNNs.

Theoretically, this paradigm invites renewed analysis of the role of time in neural function approximation, bridging universal approximation and Turing completeness, but under explicit resource constraints and in the function space setting (rather than classic language recognition). The constructive correspondences imply that algorithmic "programming"—in the form of explicit instruction sequences—can be realized efficiently in recurrent architectures with finite memory, given unbounded computation time.

Future Directions

Several interesting avenues open from this work: extending results to broader domains or multidimensional settings; identifying minimal hidden dimension for the universal property; generalizing to different activations (e.g., rational, analytic, or smooth); or examining the statistical learnability of such universal machines. Further, the TMNU model offers a structured bridge between symbolic computation and neural architectures, inviting richer study into hybrid computation and arithmetic reasoning in neural models.

Conclusion

This paper formally establishes that fixed-size, fixed-parameter ReLU RNNs are universal uniform approximators of continuous functions on $f$2, with arbitrary accuracy achievable through increased runtime alone. The constructive argument leverages a new hybrid computational model—the TMNU—that enables explicit algorithmic realization of any continuous function via manageable, compositional arithmetic subroutines. The implications of this result extend throughout the theory of deep learning, recurrent computation, and neural program synthesis, setting runtime as the core resource and providing a foundation for future theoretical and practical advancements in computation-limited neural architectures.

---

Reference: "Recurrent neural networks approximate continuous functions" (2606.20325)