A Novel Tensor Product-Based Neural Network for Solving Partial Differential Equations

Abstract: This paper presents the Tensor Product Network (TPNet), a novel neural architecture for efficient and accurate function approximation and PDE solving. The core of the proposal involves constructing the solution explicitly as a linear combination of basis functions integrated into the network, with coefficients determined by a direct least-squares solve, thereby bypassing traditional gradient-based training. The key methodological contribution include: (1) an efficient tensor-product scheme that generates multi-dimensional basis functions from combinations of two sets of subnetwork outputs, significantly reducing model complexity and parameter count while maintaining expressivity; (2) a block time-marching strategy to improve computational efficiency in long-time simulations; and (3) a linear reformulation strategy for handling nonlinear PDEs by treating known nonlinear terms as sources. TPNet achieves superior accuracy and shorter training times than conventional neural network solvers. This performance gain stems from its structured design and deterministic least-squares fitting, which contrast with the iterative, often computationally intensive optimization required by mainstream methods like Physics-Informed Neural Networks (PINNs).

• The paper introduces a novel tensor product-based neural network (TPNet) that approximates PDE solutions via deterministic least-squares optimization.
• The methodology leverages dual subnetworks to construct a rich basis from tensor products, ensuring superior accuracy with reduced training complexity.
• Practical experiments demonstrate TPNet’s scalability and error minimization in various PDE contexts, outperforming traditional and neural solvers.

Tensor Product-Based Neural Network (TPNet): A Structured Approach to Efficient PDE Solving

Introduction

This paper presents TPNet, a tensor product-based neural network architecture specifically designed for high-efficiency and high-accuracy solution of partial differential equations (PDEs) (2605.29688). TPNet introduces a fundamentally different mechanism for function approximation and PDE solution by representing the solution as a deterministic linear combination of basis functions formed via tensor products of two subnetwork outputs, with coefficients determined analytically by least-squares optimization rather than gradient-based approaches. The methodology not only unifies mesh-free, high-dimensional, and time-dependent PDE solvers but also achieves significant computational gains over established neural approaches such as HLConcELM, PINNs, DRM, and random feature methods.

Methodological Innovations

Tensor Product Basis Construction

The core contribution of TPNet is the construction of basis functions through the tensor product of two stochastically parameterized neural subnetworks. For a $d$-dimensional input $\mathbf{x}$, two subnetworks (with architectures such as ELM, MLP, or ResNet) each produce $p$ outputs, forming basis function sets $\Phi_1$ and $\Phi_2$. The overall basis for the solution space is constructed as $\Phi = \Phi_1 \otimes \Phi_2$, yielding $M = p^2$ basis functions. This approach permits high expressivity with a manageable parameter count, ensures ease of automatic differentiation, and circumvents the complexity increases typical of single large monolithic networks.

Figure 2: TPNet architecture—two subnetworks process the entire input and their outputs define tensor-product basis functions, which are linearly combined to approximate the solution.

Least Squares Analytical Solution and Training Efficiency

Unlike iterative, gradient-based optimization used in PINNs, DGM, and DRM, TPNet analytically determines the coefficients $w_{ij}$ in the linear combination of basis functions via a single least-squares solve. The basis function parameters are frozen after random initialization, so no further backpropagation or weight updates are required beyond the output layer. This strategic choice reduces both training time and solution variance, and mitigates spectral bias associated with first-order optimizers.

Block Time-Marching for Long-Time Simulations

For time-dependent PDEs, TPNet incorporates a block time-marching (BTM) approach. The time axis is divided into blocks, and the solution is computed sequentially for each time block using the state at the start of the block as the initial condition. This enables stable, accurate long-term simulation, which is not trivial for neural network solvers due to error accumulation.

Theoretical Properties

The paper proves that the tensor-product construction delivers universal approximation capability, retaining the denseness property in $\mathcal{C}(\Omega)$ for any continuous (sigmoidal) activation function, including $\tanh$. The extension of Cybenko’s theorem to tensor product bases guarantees that with sufficient $\mathbf{x}$0, the TPNet can approximate any continuous function arbitrarily closely.

Numerical Experiments

Function Approximation

TPNet variants (TP-ELM, TP-MLP, TP-ResNet) are evaluated on benchmark function approximation and showcase competitive or superior accuracy to HLConcELM, with TP-ResNet achieving $\mathbf{x}$1 error as low as $\mathbf{x}$2, notably outperforming HLConcELM at similar computational cost.

Figure 1: $\mathbf{x}$3 error and training time for function approximation comparing TPNet variants and HLConcELM.

Linear and Nonlinear PDEs

Helmholtz Equation

TPNet, especially the ResNet variant, achieves remarkable accuracy ($\mathbf{x}$4 error $\mathbf{x}$5), far surpassing both classical FEM (with $\mathbf{x}$6 degrees of freedom) and neural baselines with dramatically reduced training time.

Figure 5: $\mathbf{x}$7 error and training time for Helmholtz equation across models, highlighting TP-ResNet’s superiority at large basis size.

Heat and Wave Equations

For time-dependent PDEs, TPNet variants scale with basis size to maintain low errors, whereas HLConcELM degrades as dimensionality grows. TP-MLP attains the lowest errors in heat equation, while TP-ELM is fastest. TPNet’s compatibility with BTM is crucial for accurate, stable long-time solutions.

Figure 3: TPNet maintains low $\mathbf{x}$8 errors and shorter training times for the heat equation, outperforming HLConcELM.

High-Dimensional Poisson Equation

TPNet handles up to $\mathbf{x}$9 dimensions stably, with TP-MLP and TP-ResNet maintaining smaller errors than HLConcELM, demonstrating effective high-dimensional representation leveraging the tensor product structure.

Figure 4: TPNet’s accuracy and efficiency on the high-dimensional Poisson equation as dimension increases.

Nonlinear Burger’s Equation

For non-linear dynamics, the combination of tensor-product basis and Picard iteration yields rapid convergence, with TPNet attaining substantially lower errors and training times compared to HLConcELM.

Figure 6: $p$0 error and runtime comparison for nonlinear Burger’s equation, showing TPNet’s efficiency in nonlinear contexts.

Diffusion Equation and BTM

TPNet combined with block time-marching achieves machine-precision level errors for long-time diffusion, whereas HLConcELM without BTM fails to control error growth. BTM drastically reduces errors for all models; TP-ResNet achieves $p$1 error $p$2 in only $p$3 seconds.

Figure 10: BTM sharply reduces error and runtime for TPNet versus HLConcELM in long-time diffusion problems.

Figure 8: Heatmap of absolute error for TP-ResNet in long-time diffusion using BTM, with errors on the order of $p$4 across space-time.

Practical and Theoretical Implications

The TPNet methodology provides an efficient, scalable, and highly accurate mesh-free solver for both linear and nonlinear PDEs in arbitrary dimensions, decoupling model complexity from basis size through tensor-product construction. It eliminates the need for domain decomposition and complex architectural modifications required by recent high-precision neural solvers. The deterministic least-squares analytical solution substantially improves training efficiency and interpretability.

Strong empirical results: TPNet achieves or surpasses state-of-the-art accuracies (sometimes by several orders of magnitude) while reducing model size and training duration. TP-ResNet in particular consistently produces the lowest $p$5 and $p$6 errors—unmatched by both deep single-network models and traditional mesh-based methods.

Contradictory findings to intuition: The results highlight that more complex single-network architectures (HLConcELM, single deep MLPs) are not strictly necessary for high-accuracy PDE solution — a carefully structured, lower-parameter tensor product design offers superior efficiency and error control.

Implications for PDE solvers: The TPNet’s design is especially pertinent in scenarios where high-dimensionality or long-time integration renders classical approaches infeasible. Its fast differentiation and deployment pathways are conducive to broad adoption in scientific computing and engineering applications.

Limitations and future directions: The method's accuracy plateaus beyond a certain basis function threshold, indicating a potential for further architectural and initialization improvements; integration of adaptive or data-driven subnetwork parameterization strategies may yield further gains. Next steps include extending the framework to irregular domains, integrating enhanced regularization techniques, and exploring its performance in stochastic, multi-physics, and real-world parameterized PDE settings.

Conclusion

TPNet redefines neural PDE solvers by leveraging tensor product-based basis construction and deterministic least-squares coefficient solution. This enables superior approximation accuracy, efficient automatic differentiation, and mesh-free scalability, as validated by rigorous numerical experiments. TPNet’s design principles and findings set a new standard for efficient scientific machine learning methods, with wide-ranging ramifications for computational mathematics and engineering simulation.