Approximation Error Upper and Lower Bounds for Hölder Class with Transformers

Abstract: We explore the expressive power of Transformers by establishing precise approximation error upper and lower bounds for Hölder class. Specifically, a new approximation upper bound is derived for the standard Transformer architecture equipped with Softmax operators, ReLU activation functions, and residual connections. We prove that a Transformer network composed of at most $\mathcal{O}(\varepsilon^{-{d_{0}}/α})$ blocks can approximate any bounded Hölder function with $d_{0}$-dimensional input and smoothness $α\in(0,1]$ under any accuracy $\varepsilon>0$. In the case of approximation lower bounds, leveraging the VC-dimension upper bound, we are the first to rigorously prove that Transformers demand for at least $Ω(\varepsilon^{-{d_{0}}/({4α})})$ blocks to achieve the $\varepsilon$ approximation accuracy. As a final step, we extend the derived results for standard Transformers to a general regression task and establish the corresponding excess risk rates demonstrating Transformers' empirical effectiveness in real-world settings.

• The paper introduces explicit non-asymptotic L2 error bounds for standard Transformers approximating Hölder functions.
• It employs a constructive approach and VC-dimension analysis to derive tight upper and lower bounds on the approximation error.
• The study highlights practical depth-width trade-offs and finite-sample regression rates, guiding the design of scalable Transformer architectures.

Approximation Error Bounds for the Hölder Class with Standard Transformers

Overview

This work establishes strong non-asymptotic upper and lower bounds for the $L_2$ approximation error of standard Transformer architectures when modeling the Hölder function class. Unlike previous literature, these results cover standard Transformers—that is, architectures with canonical Softmax-based self-attention, ReLU-activated feed-forward layers, and residual connections—without modification or variant (e.g., no replacement with Hardmax, omission of residuals, or non-standard modules). The results precisely characterize the expressivity and limitations of Transformers in approximating functions with bounded smoothness, demonstrating the delicate dependence of required network depth on the input dimension $d_0$ and the Hölder smoothness $\alpha$. Both the upper and lower bounds are substantiated via explicit construction and VC-dimension-based argumentation, respectively.

Transformer Function Class and Hölder Setting

The analysis rigorously formalizes the function spaces and Transformer architectures under consideration. For an input dimensionality $d_0$ and smoothness $\alpha\in(0,1]$, the target space is the class $\mathcal{H}^{\alpha}_{d_0}([0,1]^{d_0}, K)$ of $d_0$-dimensional vector-valued Hölder functions defined on the unit cube, with norm and difference bounds on partial derivatives up to order $r = \lfloor\alpha\rfloor$. Importantly, the analysis is intrinsically high-dimensional, imposing no structural constraints such as coordinate separability.

The Transformer class in this analysis (pipeline illustrated in Figure 1) receives input vectors, reshapes them into $d\times L$ matrices (where $d_0 = dL$), appends positional encodings, and processes them through a composition of $d_0$0 identical Transformer blocks. Each block consists of a Softmax-based multi-head self-attention layer, a two-layer ReLU feed-forward subnetwork, and residual connections wrapping both modules, mirroring practical architectures in foundation models.

Figure 1: Pipeline of the standard Transformer function class: bridging the vector form and sequence form.

Main Upper Bound: Constructive Expressivity Rate

A core contribution of the work is an explicit constructive result: for any $d_0$1, any function in the Hölder class $d_0$2 can be approximated in $d_0$3 distance by a standard Transformer with at most $d_0$4 blocks. Notably, the architecture is of bounded width (dependent only on $d_0$5), and the rate matches the best-known for deep ReLU feed-forward networks (except for a potential log factor, see Tightness discussion below).

The construction is technically subtle—the challenge is to map high-dimensional vectors to sequences, perform fine-grained quantization, and guarantee token separation via Softmax self-attention. The pipeline is as follows:

• Quantization: The input is mapped to a discrete grid via stacked feed-forward-ReLU blocks, ensuring most of the continuous space is effectively reduced to a tractable piecewise-constant structure.
• Contextual separation: A single self-attention layer is shown to suffices for distinguishing all quantized grid entries, even in presence of duplicate tokens, by producing unique contextual identifiers per grid cell using designed weight matrices.
• Value assignment: Further feed-forward blocks associate each unique contextual token with its corresponding target output value.

The error is shown to be tightly controlled as long as $d_0$6; otherwise, the quantization grid separation and approximation error cannot be simultaneously achieved.

Lower Bound: VC-Dimension Argument and Limits

A rigorous lower bound complements the upper bound, which is derived via a careful estimation of the VC-dimension of the standard Transformer class (based on the operational complexity of the architecture including Softmax nonlinearity, following [anthony2009neural]). The argument shows that any standard Transformer achieving $d_0$7-approximation of a generic Hölder function must possess at least $d_0$8 blocks. This is the first such lower bound matching the theoretical framework and structure of widely-used Transformers.

The VC-dimension calculation (summarized in Figure 2) is based on an explicit enumeration of the arithmetic, comparison, and nonlinearity operations in the constructed network. The lower bound guarantees that network capacity (block depth) cannot be lowered below this threshold without loss in approximation power, providing a minimal scaling requirement for model size as a function of dimension and smoothness.

Figure 2: Error decomposition of the excess risk.

Depth-Width Trade-offs

The analysis also extends to depth-width trade-offs. The upper bound can be achieved either with high-depth, bounded-width Transformers or, equivalently, with shallower, extremely wide architectures, as long as the product $d_0$9 is maintained. This flexibility is realized by carefully constructing block-diagonal feed-forward modules corresponding to 'parallelization' of quantization or value-assignment subunits, confirming the theoretical potential for parallel scaling.

Application: Statistical Regression and Excess Risk

The theoretical results directly imply generalization bounds for regression tasks with Hölder-smooth ground truth. By decomposing the estimator's excess risk into approximation, statistical, and optimization components (as shown diagrammatically in Figure 2), the paper provides explicit rates for the excess population risk as a function of the number of samples $\alpha$0:

$\alpha$1

when the block number $\alpha$2 is chosen optimally in terms of $\alpha$3. The rate is suboptimal compared to the information-theoretic minimax rate (for ReLU networks), primarily due to looseness in the VC-dimension upper bound for architectures with exponential nonlinearity (Softmax); partially closing this gap remains an open direction.

Technical Innovations and Implications

The work introduces novel theoretical techniques:

• Upper bounds are obtained via explicit, non-asymptotic construction, leveraging all architectural components (layer normalization is omitted but otherwise the construction closely mirrors practical implementations).
• The attention module is shown to achieve unique contextual encoding with just a single Softmax head by explicit design, sidestepping problematic norm growth inherent in approximations using artificially sharpened Softmax or Hardmax replacement.
• Lower bounds do not rely on the feed-forward network approximation theory alone (as in previous works), but intrinsically handle the compositional nature and exponential nonlinearity of self-attention.

The results have the following implications:

• Practical guidance for model scaling: The scaling laws for required depth are explicit in both target smoothness and input dimension, strongly suggesting practical design heuristics for Transformer-based regressors and function approximators in high-dimensional structured-data tasks.
• Statistical learning guidance: The bounds provide finite-sample guarantees and clarify the bias-variance trade-off, notably capturing the optimal block number for minimal excess risk.

Limitations and Research Directions

While the separation between the upper ($\alpha$4) and lower ($\alpha$5) bounds is tight up to a polynomial factor, there is still a nontrivial gap. This arises from both architectural constraints on value mapping (which do not fully exploit higher smoothness, i.e., for $\alpha$6) and the conservative nature of available VC-dimension bounds for networks with exponential activations. Tighter lower bounds—ideally matching the optimal rate—would require a more refined combinatorial argument or new structural insights into the parameter space of Softmax-based networks.

Future work directions include:

• Tighter VC-dimension analysis for exponential-operator networks, potentially leveraging symmetries or sparsity specific to attention architectures.
• Analysis for higher smoothness classes and piecewise-polynomial approximation using multi-head or deeper attention compositions.
• Investigation of the implicit regularization effects of normalization layers and practical, non-asymptotic optimization error bounds for gradient-based training in large-scale settings.

Conclusion

This paper provides a comprehensive theoretical characterization of the approximation limits and statistical performance of standard Transformer architectures for the Hölder class. It establishes, for the first time, both upper and lower bounds rooted in the precise operational structure of Transformers, and connects these to finite-sample regression error via covering number and VC-dimension analysis. These results clarify the model complexity requirements for accurate sequence-based regression and provide a robust theoretical foundation to guide future architectural and optimization developments for deep attention networks (2605.07463).