Truncated Polynomial Layer in Neural Networks

• TruncatedPolynomialLayer is a parameterized function layer that expands inputs into polynomial features up to a fixed degree using bases like Chebyshev or monomial.
• It leverages a learnable linear map and an optional nonlinearity to achieve efficient universal function approximation with fewer parameters than traditional deep networks.
• The layer supports efficient hardware inference and practical applications in generative modeling, classification, and regression, backed by theoretical approximation guarantees.

A TruncatedPolynomialLayer is a parameterized function layer that maps input vectors to polynomial feature expansions up to a fixed degree, applies a learnable linear map, and optionally a non-polynomial activation. These layers generalize linear layers by replacing affine transformations with multivariate polynomial bases—commonly Chebyshev or monomial—truncated at a specified order. They support efficient, theoretically justified universal function approximation with fewer parameters than deep neural networks for the same target fidelity. TruncatedPolynomialLayers are central in architectures such as SUPN and are also widely studied for efficient hardware inference, generative modeling, polynomial networks, and as piecewise or localized polynomial feature maps.

1. Mathematical Formulation and Feature Construction

Let $x \in [-1,1]^D$ be the input. A TruncatedPolynomialLayer first selects a downward-closed multi-index set $\Lambda \subset \mathbb{N}_0^D$, such as the total-degree lower set:

$$\Lambda_{\mathrm{TD}}(M) = \{ m \in \mathbb{N}_0^D : \sum_{d=1}^D m_d \leq M \}$$

or the hyperbolic cross:

$$\Lambda_{\mathrm{HC}}(M) = \{ m \in \mathbb{N}_0^D : \prod_{d=1}^D (m_d + 1) \leq M+1 \}$$

For each $m \in \Lambda$, define the polynomial feature

$T_m(x) = \prod_{d=1}^D T_{m_d}(x_d)$

where $T_k(\cdot)$ is the $k$th Chebyshev polynomial of the first kind.

The feature lift is the map

$$\Phi_\Lambda(x) = (T_m(x))_{m \in \Lambda} \in \mathbb{R}^{|\Lambda|}$$

The layer output for $N$ neurons is then

$\Lambda \subset \mathbb{N}_0^D$0

where $\Lambda \subset \mathbb{N}_0^D$1 is a learned weight matrix and $\Lambda \subset \mathbb{N}_0^D$2 an optional bias (commonly $\Lambda \subset \mathbb{N}_0^D$3 in SUPN). The final output is typically $\Lambda \subset \mathbb{N}_0^D$4 for some fixed nonlinearity, such as $\Lambda \subset \mathbb{N}_0^D$5 (Morrow et al., 26 Nov 2025).

2. Parameterization, Truncation, and Computational Tradeoffs

Truncation degree $\Lambda \subset \mathbb{N}_0^D$6 sets expressivity, with $\Lambda \subset \mathbb{N}_0^D$7 for the total-degree case. The polynomial layer has $\Lambda \subset \mathbb{N}_0^D$8 parameters, possibly plus $\Lambda \subset \mathbb{N}_0^D$9 biases. The practitioner selects the maximal $$\Lambda_{\mathrm{TD}}(M) = \{ m \in \mathbb{N}_0^D : \sum_{d=1}^D m_d \leq M \}$$0 such that $$\Lambda_{\mathrm{TD}}(M) = \{ m \in \mathbb{N}_0^D : \sum_{d=1}^D m_d \leq M \}$$1 matches a target budget; then $$\Lambda_{\mathrm{TD}}(M) = \{ m \in \mathbb{N}_0^D : \sum_{d=1}^D m_d \leq M \}$$2 calibrates the available "neuronal width" versus polynomial expressivity. Increasing $$\Lambda_{\mathrm{TD}}(M) = \{ m \in \mathbb{N}_0^D : \sum_{d=1}^D m_d \leq M \}$$3 allows higher-order or higher-frequency approximation but quadratically increases complexity.

For batch evaluation, the construction is

• Compute $$\Lambda_{\mathrm{TD}}(M) = \{ m \in \mathbb{N}_0^D : \sum_{d=1}^D m_d \leq M \}$$4 (batch polynomial features, shape $$\Lambda_{\mathrm{TD}}(M) = \{ m \in \mathbb{N}_0^D : \sum_{d=1}^D m_d \leq M \}$$5),
• $$\Lambda_{\mathrm{TD}}(M) = \{ m \in \mathbb{N}_0^D : \sum_{d=1}^D m_d \leq M \}$$6,
• $$\Lambda_{\mathrm{TD}}(M) = \{ m \in \mathbb{N}_0^D : \sum_{d=1}^D m_d \leq M \}$$7,
• Final output via a linear map on $$\Lambda_{\mathrm{TD}}(M) = \{ m \in \mathbb{N}_0^D : \sum_{d=1}^D m_d \leq M \}$$8.

Pseudocode for the forward pass is explicitly provided in (Morrow et al., 26 Nov 2025), facilitating practical implementation.

3. Approximation Guarantees and Expressivity

TruncatedPolynomialLayers inherit both classical polynomial approximation rates and the universal approximation property when composed with nonlinearities such as $$\Lambda_{\mathrm{TD}}(M) = \{ m \in \mathbb{N}_0^D : \sum_{d=1}^D m_d \leq M \}$$9. For univariate $$\Lambda_{\mathrm{HC}}(M) = \{ m \in \mathbb{N}_0^D : \prod_{d=1}^D (m_d + 1) \leq M+1 \}$$0, there exists a one-neuron TruncatedPolynomialLayer of degree $$\Lambda_{\mathrm{HC}}(M) = \{ m \in \mathbb{N}_0^D : \prod_{d=1}^D (m_d + 1) \leq M+1 \}$$1 achieving error

$$\Lambda_{\mathrm{HC}}(M) = \{ m \in \mathbb{N}_0^D : \prod_{d=1}^D (m_d + 1) \leq M+1 \}$$2

More generally, for $$\Lambda_{\mathrm{HC}}(M) = \{ m \in \mathbb{N}_0^D : \prod_{d=1}^D (m_d + 1) \leq M+1 \}$$3 in a multivariate function space, the best achievable error with index set $$\Lambda_{\mathrm{HC}}(M) = \{ m \in \mathbb{N}_0^D : \prod_{d=1}^D (m_d + 1) \leq M+1 \}$$4 is

$$\Lambda_{\mathrm{HC}}(M) = \{ m \in \mathbb{N}_0^D : \prod_{d=1}^D (m_d + 1) \leq M+1 \}$$5

and a single TruncatedPolynomialLayer followed by a final linear map achieves

$$\Lambda_{\mathrm{HC}}(M) = \{ m \in \mathbb{N}_0^D : \prod_{d=1}^D (m_d + 1) \leq M+1 \}$$6

where $$\Lambda_{\mathrm{HC}}(M) = \{ m \in \mathbb{N}_0^D : \prod_{d=1}^D (m_d + 1) \leq M+1 \}$$7 is a function of the network size and choice of $$\Lambda_{\mathrm{HC}}(M) = \{ m \in \mathbb{N}_0^D : \prod_{d=1}^D (m_d + 1) \leq M+1 \}$$8 (Morrow et al., 26 Nov 2025).

4. Implementation Variants and Adaptations

Polynomial Basis and Activation

• Most SUPN variants use Chebyshev polynomial features for numerical stability.
• Monomial basis is also used in practice without loss of optimization generality.
• Non-polynomial activations, e.g., $$\Lambda_{\mathrm{HC}}(M) = \{ m \in \mathbb{N}_0^D : \prod_{d=1}^D (m_d + 1) \leq M+1 \}$$9, ensure non-trivial approximation rates.

Piecewise and Localized Extensions

• Piecewise polynomial layers partition the input domain and define localized truncated polynomial maps on each subdomain, e.g., via intervals in discontinuous piecewise polynomial neural networks (Loverich, 2015).
• FPGA/ASIC deployments favor quantized, LUT-based polynomial layers, enabling ultra-low-inference-latency architectures (Andronic et al., 2023).

Compositions and Deep Polynomial Networks

• Composing multiple truncated polynomial layers yields "deep polynomial networks," where exponential approximation rates can be achieved on target functions such as $m \in \Lambda$0, outperforming classical polynomial and rational approaches for a fixed parameter count (Yeon, 2 Mar 2025).
• Ladder polynomial networks use chain products and low-order expansions to efficiently build all degree-$m \in \Lambda$1 multivariate monomials (Liu et al., 2021, Li et al., 2019).

5. Empirical Performance and Applications

Comparative studies over 13,000 models in various dimensions (1D, 2D, 10D) demonstrate that, for fixed parameter budgets, TruncatedPolynomialLayers in SUPN architectures yield:

• Order-of-magnitude reductions in $m \in \Lambda$2 approximation error and test error variance compared to equivalently sized DNNs and KANs.
• Comparable or better generalization than pure polynomial projections, even outperforming on non-smooth target functions.
• Spectral-like convergence rates on smooth targets, fast finite-order convergence on non-smooth ones (Morrow et al., 26 Nov 2025).

FPGA-inference with TruncatedPolynomialLayers (e.g., PolyLUT) achieves similar test accuracy to conventional networks but with up to $m \in \Lambda$3 lower latency and significant area reduction, owing to the expressivity of higher-degree per-neuron polynomials (Andronic et al., 2023).

Piecewise and non-local (truncated third-order) polynomial layers exhibit activation sparsity, computational scalability, and match or exceed deep learning baselines in classification and detection tasks (Loverich, 2015, Babiloni et al., 2021).

6. Design Principles and Practical Tuning

• Truncation Degree ($m \in \Lambda$4, $m \in \Lambda$5): Set as large as parameter or computational budgets allow. Larger values afford more expressive function classes at the cost of increased computation and storage.
• Activation Function: Employ fixed nonlinearity (e.g., $m \in \Lambda$6) to combine polynomial features non-trivially; in piecewise architectures the nonlinearity may instead be generated by the piecewise structure.
• Regularization: Regularize polynomial coefficients directly (e.g., via weight decay), especially for high-degree terms to mitigate overfitting.
• Basis Choice: Chebyshev polynomials are preferred for numerical reasons, especially at moderate or high degree.
• Optimization: Use batch normalization and dropout to stabilize and regularize training, particularly in ladder (product-activation) or deep polynomial networks (Liu et al., 2021).
• Hardware Integration: For secure inference (e.g., SMPC), truncation degree governs trade-off between accuracy and online computational/communication cost (Derbeko et al., 2021).

7. Theoretical and Algorithmic Implications

The truncated polynomial approach provides a principled route to controlling expressivity and generalization in neural networks:

• It yields explicit approximation rates and error bounds via Jackson-type theorems.
• Parameter-efficient realization of high-degree interactions without the combinatorial blow-up of dense DNNs.
• Enables random feature selection, low-rank factorization, or index-set tailoring for various data regimes (Chen et al., 2022).
• Connects classical spectral methods and function approximation theory directly to neural representation.

A prominent research direction is the integration of TruncatedPolynomialLayers as foundational elements for compact, robust, and hardware-aware networks, with theoretical guarantees and application to both regression and classification regimes across scientific and engineering domains (Morrow et al., 26 Nov 2025, Andronic et al., 2023, Yeon, 2 Mar 2025).