LLT: Local Linear Transformer for PDE Operator Learning

Abstract: Neural operators have become a common approach for learning PDE solution maps and accelerating numerical simulations. Transformer-based neural operators are of particular interest, since attention can learn long-range dependencies in the computational domain. However, standard attention has two major limitations when applied to PDEs: it scales quadratically with the number of computational nodes, and it lacks an explicit bias toward local interactions. To address these issues, we introduce Local Linear Transformer (LLT) for PDE operator learning. The architecture combines linear global attention with local spatial mixing, and incorporates coordinate and geometry information. We evaluate LLT on several PDE problems, including elasticity, plasticity, airfoil flow, pipe flow, and Darcy flow. The reference data for these problems span finite-element, finite-volume, and finite-difference discretizations on structured and unstructured meshes. Compared with other neural-operator and transformer baselines from prior studies, LLT achieves competitive or lower relative $L_2$ error across these problems. On matched structured discretizations, wall-clock time per training iteration is reduced by factors of 1.8 to 2.5 relative to Transolver. We also scale the approach and apply it to a three-dimensional car aerodynamics dataset with 32,186 unstructured mesh points per sample. Together, these results indicate that LLT provides an accurate and computationally efficient operator for PDE problems across discretizations, mesh types, and problem settings.

• The paper introduces LLT, which integrates linear global attention and local spatial mixing to overcome scalability issues in PDE operator learning.
• The architecture employs branch-specific encoders and transformer blocks to enable near-linear runtime and precise reconstruction of complex solution fields.
• Empirical results across five benchmark scenarios and a 3D car aerodynamics task demonstrate superior predictive accuracy and significant computational speed improvements.

LLT: Local Linear Transformer for Efficient and Accurate PDE Operator Learning

Introduction

The LLT (Local Linear Transformer) architecture, introduced in "LLT: Local Linear Transformer for PDE Operator Learning" (2607.07718), addresses foundational computational and architectural challenges in operator learning for partial differential equations (PDEs). Leveraging both linear global attention and explicit local spatial mixing within a unified Transformer-based framework, LLT achieves accurate and scalable surrogate modeling of complex PDE solution maps across structured and unstructured domains. This essay provides a technical synthesis of the paper's motivation, architecture, empirical results, and implications for scientific machine learning.

Motivation and Technical Challenges

Neural operators are increasingly deployed to approximate solution operators for PDEs, supplanting or augmenting classical solvers in domains such as mechanics, fluid dynamics, and materials science. Transformer-based models are a promising paradigm for operator learning due to their nonlocal attention mechanisms, but they present significant limitations:

• Quadratic complexity: Standard self-attention incurs $\mathcal{O}(N^2)$ cost with respect to the number of spatial points $N$, impeding scalability to practical grid sizes.
• Insufficient locality bias: Generic attention lacks explicit spatial locality, neglecting the pronounced local structure ubiquitous in PDE solutions, especially on unstructured meshes or for hyperbolic systems.

Addressing these deficiencies, LLT combines global communication (via linear attention) with learned local mixing in a computationally efficient architecture that is directly compatible with both grid-based and mesh-based discretizations.

Architectural Overview

LLT processes input coordinates and field values at all mesh points through branch-specific encoders comprising distance-based, Fourier, and convolutional channels. Following concatenation and projection by an MLP, a stack of $L$ Transformer-based encoder blocks refines these feature vectors. Each encoder block couples RMS normalization, a linear-attention module for global communication, a local mixing operator (convolution or masked neighborhood attention, depending on mesh structure), and a SwiGLU-based feedforward path. The decoder leverages skip connections to merge representations from the initial embedding and computes the predicted solution field via a final MLP.

Figure 1: Architecture of LLT, illustrating geometry-aware embedding, global/local attention in the encoder, and skip-connected decoding for PDE surrogate prediction.

The linear attention variant in LLT utilizes kernelized feature maps (ELU+1) to ensure near-linear runtime ($\mathcal{O}(N)$) for global mixing. For local mixing, LLT applies depthwise-separable convolutions on regular grids or masked softmax attention restricted to spatial neighborhoods on unstructured meshes.

Benchmark Suite and Experimental Setup

LLT was evaluated on five canonical PDE modeling problems reflecting a broad spectrum of domains, numerics, and mesh complexities:

• Elasticity (2D, unstructured FEM): Node-wise stress prediction under varying solid structures.
• Plasticity (2D+time, body-fitted structured): Time-conditioned prediction of deformation fields after die impact.
• Airfoil Flow (2D, body-fitted structured): Prediction of Mach field over deformed airfoils.
• Pipe Flow (2D, body-fitted structured): Incompressible Navier-Stokes solution (velocity) in curved pipes.
• Darcy Flow (2D, uniform grid): Scalar elliptic pressure prediction as a function of spatial coefficient field $a(x)$.

Additionally, LLT was scaled to a large-scale 3D car aerodynamics dataset with over $32,000$ points per sample, emphasizing its universality and scalability across mesh types and problem dimensions.

Figure 2: Representative computational meshes for all five considered PDE benchmarks, demonstrating the diversity of spatial discretizations addressed by LLT.

Numerical Results and Empirical Analysis

Predictive Accuracy

LLT produced highly competitive, and frequently the lowest, reported relative $L_2$ errors on standard test splits across the examined benchmarks. Notably, LLT achieved superior accuracy for Elasticity, Plasticity, Airfoil Flow, and Darcy Flow compared to all previously published neural operator and Transformer operator baselines. For Pipe Flow, LLT remained closely competitive with Transolver++ and Latent Neural Operator (LNO).

Qualitative Accuracy: Field Reconstruction

In all cases, LLT's predictions faithfully reproduce complex local and global solution features, including boundary layer phenomena, stress concentrations, and intricate deformation fields. The following figures showcase representative prediction/reference comparisons across benchmarks:

Figure 3: Elasticity — predicted stress $\sigma$ vs reference fields for five test geometries; accurate spatial localization of stress concentration by LLT.

Figure 4: Plasticity — predicted vs reference displacement magnitudes $\|\mathbf{u}\|$ at final time step, capturing fine-grained, boundary-driven plastic flow details.

Figure 5: Darcy Flow — reconstruction of pressure $u$ for diverse permeability fields; LLT demonstrates precise adherence to reference field structure.

Figure 6: Airfoil Flow — predicted Mach number $N$0 fields over varied airfoil geometries, accurately tracking the high-gradient regions and shocks.

Figure 7: Pipe Flow — horizontal velocity $N$1 predictions for distinct pipe centerlines; LLT achieves close agreement to finite-element reference.

Large-Scale 3D: Car Aerodynamics

LLT was further evaluated on the ShapeNet Car dataset, predicting surface pressure on unstructured 3D meshes, and effectively captured both stagnation and suction regions with strong qualitative similarity to CFD reference solutions.

Figure 8: Car design — surface pressure predictions for three car geometries, with LLT handling high-resolution unstructured 3D meshes.

Computational Efficiency

Empirical timing studies demonstrate that LLT realizes substantial training-step efficiency advantages over structure-adaptive Transformer baselines. On standardized structured grid benchmarks, LLT achieves wall-clock speedups ranging from 1.8$N$2 to over 4$N$3 for equivalent model and discretization settings, with a nearly linear cost in $N$4.

Figure 9: Scaling of wall-clock time and peak memory with resolution, showing linear computational scaling for LLT and clear runtime advantage vs Transolver.

Figure 10: Computational performance across all benchmarks, further quantifying LLT's efficiency relative to Transolver in training iteration cost.

Discussion and Implications

Architectural Implications

LLT demonstrates that explicit integration of spatial locality (via convolution or masked neighbor attention) is critical for accurate operator learning in PDE contexts, complementing domain-wide communication provided by attention. Linear attention suffices for nonlocal information exchange without the bottleneck of $N$5 cost, provided the architecture also incorporates strong geometric and positional encoding.

Practical Relevance

The reduction in per-iteration computational cost widens the scope for operator learning on high-resolution domains, irregular geometries, and three-dimensional problems, enhancing the viability of neural surrogate modeling in both computational physics workflows and inverse design. The architecture is inherently mesh-agnostic, allowing deployment across finite element, finite volume, and finite difference discretizations without modification.

Theoretical Considerations and Future Directions

The explicit local-global mixing strategy in LLT provides a template for further study into inductive biases required for surrogate PDE models, especially as operator learning is extended toward multimodal, multiscale, and time-dependent scenarios. Prospective research directions include:

• Integration with physics-informed objectives or hybrid training protocols to enforce conservation or monotonicity constraints.
• Extension to multi-physics and coupled multiphase systems.
• Development of scalable variants for even higher-dimensional or multi-resolution meshes.

Conclusion

LLT advances the state of neural PDE surrogate modeling by fusing fast, scalable linear attention with targeted local spatial mixing in a Transformer framework generic to both structured and unstructured meshes. Empirical results demonstrate leading predictive accuracy on four benchmarks, consistently strong performance overall, and substantial computational efficiency gains over strong operator learning baselines. LLT provides a robust, extensible architecture for large-scale data-driven solution operators and will inform ongoing advances in neural scientific computing architectures.