IPOT: Inducing Point Operator Transformer

• IPOT is an attention-based neural operator that leverages a latent bottleneck of learnable inducing points to map input functions to outputs over irregular domains.
• It employs an encoder–processor–decoder architecture with cross- and self-attention mechanisms to achieve efficient linear scaling with respect to input and output sizes.
• Empirical evaluations demonstrate competitive performance and reduced computational complexity compared to established operator learning methods across diverse PDE benchmarks.

The Inducing Point Operator Transformer (IPOT) is an attention-based neural operator architecture designed for flexible and scalable solution operator learning for partial differential equations (PDEs) defined on irregular and high-resolution domains. IPOT introduces a latent bottleneck of learnable inducing points to decouple the discretizations of input and output function samples from the computational processor, enabling efficient linear complexity and mesh-invariant generalization across variable domains, resolutions, and geometries (Lee et al., 2023).

1. Problem Formalism and Mesh-Invariant Operator Learning

IPOT addresses the operator learning problem of mapping an input function $a \in \mathcal{A}$ to an output function $u = G(a) \in \mathcal{U}$, where $\mathcal{A}, \mathcal{U}$ are infinite-dimensional function spaces (e.g., Sobolev or $L^2$ spaces) defined on a domain $\Omega$. In practice, only finite samples of $a$ are available at input locations $X = \{x_1, \ldots, x_n\} \subset \Omega_x$ with $a|_X \in \mathbb{R}^{n \times d_a}$, while the desired output is a set of values $u|_Y \in \mathbb{R}^{m \times d_u}$ at arbitrary query points $Y = \{y_1, \ldots, y_m\} \subset \Omega_u$. The system may be highly irregular: $u = G(a) \in \mathcal{U}$0 and $u = G(a) \in \mathcal{U}$1 need not coincide, and $u = G(a) \in \mathcal{U}$2 is permitted. The principal requirement is mesh-invariance, i.e., the ability to process or predict on any set of irregular points and variable discretizations.

2. Model Architecture: Encoder–Processor–Decoder with Inducing-Point Latents

IPOT is structured as an encoder–processor–decoder model with $u = G(a) \in \mathcal{U}$3 learnable inducing points forming a latent bottleneck.

• Inducing points: A set of $u = G(a) \in \mathcal{U}$4 latent locations $u = G(a) \in \mathcal{U}$5 that serve as global queries for summarizing and propagating information between inputs and outputs.
• Embedding maps:
  • $u = G(a) \in \mathcal{U}$6, concatenating spatial coordinates with input features.
  • $u = G(a) \in \mathcal{U}$7 for output/query features.

2.1 Encoder: Cross-Attention from $u = G(a) \in \mathcal{U}$8 to $u = G(a) \in \mathcal{U}$9

Given embedded inputs $\mathcal{A}, \mathcal{U}$0 and initial latent queries $\mathcal{A}, \mathcal{U}$1, cross-attention computes

$\mathcal{A}, \mathcal{U}$2

for $\mathcal{A}, \mathcal{U}$3, $\mathcal{A}, \mathcal{U}$4, $\mathcal{A}, \mathcal{U}$5. Post-attention latents $\mathcal{A}, \mathcal{U}$6 and subsequent layers employ LayerNorm, MLP, and residual connections.

2.2 Processor: Self-Attention on Inducing Points

For $\mathcal{A}, \mathcal{U}$7 latent blocks, standard Transformer-style self-attention is applied among the $\mathcal{A}, \mathcal{U}$8 inducing points,

$\mathcal{A}, \mathcal{U}$9

with $L^2$0.

2.3 Decoder: Cross-Attention from $L^2$1 to $L^2$2

Output embeddings $L^2$3 attend to final latents $L^2$4:

$L^2$5

where $L^2$6, $L^2$7, $L^2$8. A pointwise decoder MLP maps $L^2$9 to predictions $\Omega$0.

All attention blocks use LayerNorm, residuals, and GELU-activated feed-forward sublayers.

3. Computational Complexity and Scaling Characteristics

A central feature of IPOT is its linear scaling with respect to the number of input ($\Omega$1) and output ($\Omega$2) points, owing to the choice of $\Omega$3. The per-layer computational cost is:

Component	Complexity	Dependency
Encoder	$\Omega$4	Linear in $\Omega$5 and $\Omega$6
Processor	$\Omega$7	Quadratic in $\Omega$8, independent of $\Omega$9
Decoder	$a$0	Linear in $a$1 and $a$2

The total per-forward computation (for $a$3 latent processor layers) is $a$4, in contrast to the quadratic scaling $a$5 of a standard Transformer on joint input-output sets. A typical value of $a$6–$a$7 yields an advantageous trade-off between computational efficiency and model expressivity. The latent depth $a$8 is decoupled from the size of the input/output discretization, allowing arbitrarily deep and long-horizon architectures.

4. Training Methodology and Data Regimes

IPOT is trained to minimize the relative $a$9 error across a dataset:

$X = \{x_1, \ldots, x_n\} \subset \Omega_x$0

No explicit PDE residual or physics-informed loss was used in the reported experiments, although such losses can be incorporated. Regularization is performed using standard weight decay (AdamW optimizer).

Key datasets include:

• Regular grids: 1D Burgers (n=1024), 2D Darcy (n=85²), 2D Navier–Stokes (n=65², with time).
• Irregular grids: Airfoil meshes, point cloud elasticity, 3D plastic forging meshes, spherical shallow-water (8192 points).
• Real-world weather: ERA5 daily 2m temperature ($X = \{x_1, \ldots, x_n\} \subset \Omega_x$116,200 points, 7 time channels).

Inputs and outputs may be masked or vary in resolution. The model supports arbitrary masking and different spatial/temporal discretizations.

5. Empirical Results and Benchmark Comparisons

IPOT demonstrates competitive or superior performance relative to state-of-the-art operator-learning architectures, including Fourier Neural Operators (FNO), FFNO, and OFormer, across both regular and irregular domains.

Dataset	FNO Rel $X = \{x_1, \ldots, x_n\} \subset \Omega_x$2	FFNO / OFormer	IPOT Rel $X = \{x_1, \ldots, x_n\} \subset \Omega_x$3 (Params, Time, Mem)
Darcy (85²)	1.09e-2	7.70e-3 / 1.26e-2	1.73e-2 (0.15M, 2.70 s, 1.82 GB)
Navier–Stokes (65², time)	1.28e-2	— / 1.04e-2	8.85e-3 (0.12M, 21.05 s, 2.08 GB)
Airfoil (11,271 pts)	—	7.80e-3 / 1.83e-2	8.79e-3 (0.12M, 2.15 s, 2.10 GB)
Elasticity (972 pts)	—	2.63e-2 / 1.83e-2	1.56e-2 (0.12M, 1.99 s, 1.13 GB)
Plasticity (62,620 pts)	—	4.70e-3 / 1.83e-2	3.25e-3 (0.13M, 10.14 s, 5.35 GB)
ERA5 Temperature	—	7.25e-3 / 1.15e-2	6.64e-3 (0.51M, 9.83 s, 10.58 GB)

In multi-resolution (ERA5, 4°, 1°, 0.25°) and masked experiments, IPOT matches or outperforms FNO and OFormer. On long-term shallow water forecasting, IPOT achieves relative error 1.11e-3 at $X = \{x_1, \ldots, x_n\} \subset \Omega_x$4–40, versus DINO’s 1.52e-3.

6. Model Properties, Limitations, and Prospective Extensions

Key strengths include:

• Mesh-invariance: No dependency on regular grids or structural bias—supports arbitrary input and output locations.
• Linear complexity: Efficient handling of high-dimensional and large-scale inputs/outputs, enabled by the inducing-point bottleneck.
• Depth-agnostic scalability: Latent depth is independent of data discretization size.

Notable limitations:

• Trade-off in latent size ($X = \{x_1, \ldots, x_n\} \subset \Omega_x$5): Small $X = \{x_1, \ldots, x_n\} \subset \Omega_x$6 may underfit; larger $X = \{x_1, \ldots, x_n\} \subset \Omega_x$7 increases computational cost.
• Hyperparameter sensitivity: Performance depends on the choice of attention head count, latent dimensions, and block depth.
• Lack of explicit PDE bias: The architecture is purely data-driven in its reported form; physics-informed terms can be added but are not intrinsic.

Possible extensions include:

• Incorporation of physics-informed losses (e.g., PINN-style regularization).
• Adaptive or hierarchical selection of inducing points, possibly varying $X = \{x_1, \ldots, x_n\} \subset \Omega_x$8 per layer.
• Generalization to operator inversion, control, and inverse problem settings.
• Introducing continuous-time latent recurrence for forecasting tasks.
• Coupling multi-fidelity or multi-domain operators via shared latent structure.

7. Context and Significance within Neural Operator Methods

IPOT responds to the twin challenges of mesh flexibility and scalability in operator learning by introducing an explicit inducing-point mechanism, inspired by inducing point methods in kernel machines but realized in an attention-centric, end-to-end differentiable architecture. This approach enables rapid and memory-efficient processing of irregular, high-resolution PDE data, making it suitable for scientific computing applications such as high-resolution weather prediction, elasticity, fluid mechanics, and more. The architecture’s modularity enables potential integration with explicit physics-based constraints and flexible adaptation to novel operator learning regimes (Lee et al., 2023).