Fourier Neural Operators with rank-1 lattice points and hyperbolic cross

Abstract: The \emph{Fourier neural operator} (FNO) is a neural network architecture that learns mappings between function spaces. Its efficient implementation is based on the multi-dimensional Fourier transform. By deriving general regularity bounds for the FNO with respect to both the spatial and parametric variables, we prove that the generalization error of the FNO can be improved by replacing spatial tensor product grids with purpose-built rank-1 lattice points, and by using a second lattice carefully constructed as training points in the parametric space. We achieve more accurate and efficient approximations from fewer network parameters, fewer spatial points, and fewer training samples. In addition, the architecture is simplified, because the high-dimensional Fourier transform on rank-1 lattices requires only a \emph{one-dimensional fast Fourier transform}, and we can use a \emph{hyperbolic cross} frequency index set with lattice points. We demonstrate the benefits of our \emph{lattice-based hyperbolic-cross FNOs} for an elliptic PDE on the torus.

• The paper shows that augmenting FNOs with rank-1 lattice sampling in both spatial and parametric domains drastically reduces sample complexity and improves generalization bounds.
• The methodology employs hyperbolic cross frequency truncation to cut down the number of modes and achieve up to 90% reduction in trainable parameters without losing accuracy.
• Empirical results on high-dimensional PDEs demonstrate significant training speedup and robust approximation performance, backed by rigorous theoretical error estimates.

Fourier Neural Operators with Rank-1 Lattice Points and Hyperbolic Cross: An Expert Perspective

Introduction and Motivation

This paper presents a rigorous theoretical and empirical investigation of Fourier Neural Operators (FNOs) augmented by rank-1 lattice sampling and hyperbolic cross frequency truncation. The central thesis is that spatial and parametric sampling using rank-1 lattices, coupled with efficient frequency domain truncations, significantly improves generalization bounds and empirical efficiency for operator learning tasks, particularly those arising from parametric PDEs.

The FNO architecture is examined in depth, with the analysis extending beyond empirical observations to precise regularity bounds in both spatial and parametric domains. The authors leverage advances in quasi-Monte Carlo integration and weighted Korobov/Sobolev spaces to formalize convergence rates and error bounds, thus bridging the gap between neural operator theory and computational practice.

Core Contributions

The paper delineates three primary innovations:

1. Rank-1 Lattice Sampling in Physical Space: Spatial data points on the domain $D$ are selected using rank-1 lattices instead of traditional tensor grids. This reduces the sample complexity in high dimensions while maintaining approximation fidelity.
2. Rank-1 Lattice Sampling in Parameter Space: The parametric domain, representing random field expansions (e.g., affine or Chebyshev models), is sampled using a second, carefully constructed rank-1 lattice. This mitigates generalization error with provable convergence rates dependent on regularity.
3. Hyperbolic Cross Frequency Truncation: Frequency truncation in Fourier space is effected via hyperbolic crosses rather than axis-aligned boxes. For functions with prescribed smoothness, this mechanism ensures a drastic reduction in the number of modes for an equivalent error threshold.

Figure 2: Real part of the forcing term $f()$ displaying its structure on the physical domain.

Theoretical Foundations

The FNO architecture is explicitly formulated, with kernel integral operators implemented as convolutions in the Fourier domain. The derivation rigorously details the kernel $\kappa_\ell$ and the truncation set $\mathcal{A}$, addressing both practical computation via FFT and theoretical approximation rates.

A universal approximation result is announced for a subsequent paper, but the present manuscript rigorously builds regularity bounds for both parametric and spatial variables. These bounds, formalized via mixed derivatives and function space norms (weighted Sobolev/Korobov spaces), are exploited to construct a combined generalization error bound. Notably, the bounds are independent of data, relying only on network and activation parameters and the prescribed smoothness of the target operator.

The use of hyperbolic cross index sets in the frequency domain is motivated both theoretically (decay of Fourier coefficients) and computationally (reduction in mode count), and the construction of spatial and parametric lattices is explained with reference to fast component-by-component QMC methods.

Figure 4: Real part of a single realization of the affine random input field $a(,y^*)$, illustrating spatial variability induced by parametric sampling.

Numerical Experiments

Experiments address a high-dimensional elliptic PDE on the torus, with both input fields and solutions parameterized via random fields (affine expansion over Fourier modes). The explicit form of the forcing term and solution regularity allows for controlled evaluation of the FNO variants.

Architectures compared include standard FNO (box truncation, tensor grid), FNO-LAT (box truncation, rank-1 lattice), FNO-HC (hyperbolic cross, tensor grid), and FNO-HC-LAT (hyperbolic cross, rank-1 lattice). Empirical results demonstrate that FNO-HC-LAT achieves comparable or superior generalization error to conventional FNO with dramatically reduced trainable parameter counts—up to a 90% reduction—without loss of accuracy.

Figure 1: Real part of output fields generated by FNO-HC-LAT for the same input realization, evidencing model accuracy and robustness with reduced parameterization.

Additional analyses confirm that the speedup in training time and reduction in parameter count scale with the frequency truncation level $M$, suggesting practical viability for high-dimensional operator learning. Importantly, matched-parameter experiments show that FNO-HC-LAT does not compromise approximation despite the aggressive reduction in parameter count.

Practical and Theoretical Implications

The theoretical advances provide a clear pathway for reductions in spatial and parametric discretization cost for operator learning problems in uncertainty quantification, optimal control, and forward modeling for PDEs. The generalization bounds derived are independent of the dimensionality of the parametric space, a strong claim with direct implications for scalability.

The empirical results support the assertion that rank-1 lattice and hyperbolic cross FNOs are not only theoretically justified but also practically advantageous, with observed reductions in training time and model complexity even for moderate-dimensional test problems.

The simplification in FFT computation (from $d$-dimensional to 1-dimensional transforms) is a substantial algorithmic improvement, enabling deployment of FNOs in applications previously limited by computational constraints.

Future Directions

The universal approximation theorem for lattice-based hyperbolic cross FNOs is deferred to a follow-up manuscript. Anticipated developments include application to non-periodic domains, extension to nonlinear lifting/projecting operators, and further empirical tests across a broader class of operator regression tasks.

There is substantial scope for integrating these constructions into neural operator frameworks for scientific machine learning, including DeepONets and other operator-based architectures, potentially yielding new theoretical insights into expressiveness and efficiency in infinite-dimensional machine learning settings.

Conclusion

By combining QMC-inspired rank-1 lattices in both spatial and parametric domains with hyperbolic cross frequency truncations, the paper establishes Fourier Neural Operators as efficient, theoretically robust models for learning complex operators between function spaces. The main numerical results show that such FNO variants achieve equivalent accuracy to conventional FNOs with less than 10% of the parameter count and significantly reduced computational cost. The framework delivers strong theoretical guarantees for generalization error bounds and sets the stage for future research in scalable operator learning and scientific machine learning (2606.08871).