Transolver++/Transolver-3: Scalable Neural PDE Solvers

• The paper introduces scalable transformer architectures that employ slice-based physics-attention, reducing computational complexity and achieving up to 13% improvement in accuracy.
• Transolver++ processes large-scale meshes by aggregating local geometric features into global eidetic states, significantly lowering communication overhead and maintaining high prediction fidelity.
• Transolver-3 extends these innovations with geometry tiling and state caching to reduce memory footprint and inference latency by 60%, enabling efficient simulation on industrial-scale meshes.

Transolver++ and Transolver-3 are advanced neural partial differential equation (PDE) solvers engineered for scalable, high-fidelity simulation on complex meshes characteristic of industrial engineering problems. Both are descendants of the Transolver family of transformer-based solvers, targeting large-scale mesh discretization domains (up to $10^8$ points), with architectural, algorithmic, and parallelism techniques specifically designed for data-, memory-, and compute-efficient PDE field prediction in computational engineering (Luo et al., 4 Feb 2025, Elrefaie et al., 25 Nov 2025, Zhou et al., 4 Feb 2026).

1. Formulation and Design Philosophy

Transolver++ and Transolver-3 address the fundamental limitation of prior neural PDE solvers and graph neural networks, which typically scale poorly beyond $O(10^4)$ nodes, by introducing modular, parallel transformer architectures that efficiently aggregate, process, and propagate physical state information on unstructured and structured meshes at industrial-relevant scales. The central mechanism across the family is “physics-attention via eidetic states” (“slice-based attention”), where mesh points are softly assigned to a reduced set of global states, enabling scalable nonlocal information mixing without explicit $O(N^2)$ messaging (Luo et al., 4 Feb 2025, Zhou et al., 4 Feb 2026).

2. Architecture and Algorithmic Components

Transolver++

Transolver++ processes a mesh of $N$ points, each represented by geometric and boundary-condition features bundled in $X \in \mathbb{R}^{N \times C}$. The model comprises $L$ repeated “Transolver++ blocks”, each operating as:

• Feed-Forward Embedding: Linear + GELU to project per-point features.
• Rep-Slice Assignment: For each point $x_i$, compute soft weights $w_i \in \mathbb{R}^M$ assigning it to $M$ “eidetic physical states”. The assignment employs local adaptive temperature: $\tau_i = \tau_0 + \mathrm{Linear}(x_i)$, with Gumbel-Softmax perturbation for differentiability.
• State Aggregation: Aggregate per-GPU local quantities into global slice states $O(10^4)$0, with AllReduce for cross-GPU consistency.
• Physics-Attention: Multi-head attention mixes the $O(10^4)$1 global states $O(10^4)$2.
• Deslice: Broadcast updated states $O(10^4)$3 back to points: $O(10^4)$4.
• Feed-Forward, Norm, and Residual: Final transformation and skip.

Parallelization is achieved by strictly limiting inter-GPU communication to $O(10^4)$5 per block, independent of mesh size, in contrast to naïve data/model parallelism (Luo et al., 4 Feb 2025). See Table 1 for a schematic of architectural blocks.

Stage	Key Operator	Scaling (per block)
Rep-Slice	$O(10^4)$6	$O(10^4)$7
State Aggregation	$O(10^4)$8	$O(10^4)$9
Slice Self-Attention	$O(N^2)$0	$O(N^2)$1
Deslice	$O(N^2)$2	$O(N^2)$3

Table 1: Transolver++ core block operators and complexity summaries (Luo et al., 4 Feb 2025).

Transolver-3

Transolver-3 extends the above with two core innovations to scale beyond $O(N^2)$4 mesh points:

• Faster Slice and Deslice: By algebraic refactoring, linear projections (e.g., $O(N^2)$5, $O(N^2)$6) are repositioned into the $O(N^2)$7 “slice” domain, removing $O(N^2)$8 terms present in previous designs for both slicing and deslicing operations. Memory for backpropagation is halved, and wall-clock time per block is reduced (Zhou et al., 4 Feb 2026).
• Geometry Slice Tiling: The mesh is partitioned into $O(N^2)$9 non-overlapping tiles; for each tile, only the local slice assignment matrix is materialized, aggregating partial slice states and denominators. This lowers peak memory usage from $N$0 to $N$1, where $N$2 is the tile size.
• Amortized Training: Training is performed over randomly sampled mesh subsets ($N$3), enabling the model to approximate underlying continuous operators without seeing all $N$4 in a single forward pass.
• Two-Stage Inference with State Caching: Full-mesh prediction is realized by caching global slice states (“state cache”) and then efficiently decoding to each mesh point via local attention.

Innovation	Complexity Reduction	Functionality Added
Faster Slice/Deslice	Remove $N$5 ops → $N$6, $N$7	Memory and speed improvement
Geometry Tiling	Peak mem $N$8 vs $N$9	Arbitrary mesh scaling
State Caching	Cache $X \in \mathbb{R}^{N \times C}$0 states only	Efficient full-mesh predictions

Table 2: Major architectural innovations in Transolver-3 and their computational implications (Zhou et al., 4 Feb 2026).

3. Training and Inference Protocols

Transolver++ and Transolver-3 are trained with volume and surface relative $X \in \mathbb{R}^{N \times C}$1 losses, with optional drag/lift coefficient penalties for aerodynamic or fluid dynamics tasks. Output predictions are generated for per-point physical quantities (e.g., pressure, velocity) (Luo et al., 4 Feb 2025).

Boundary conditions (Dirichlet/Neumann) and geometry features are encoded as input channels and respected via the training regime. Multi-GPU training uses synchronized slice state aggregation and DDP for weight gradients. For Transolver-3, random mesh subdomains are drawn per iteration; in inference, two-stage decoding allows field updates on meshes as large as $X \in \mathbb{R}^{N \times C}$2 points by physical state caching (Zhou et al., 4 Feb 2026).

4. Quantitative Benchmarks and Empirical Comparison

Standard PDE and Industrial Benchmarks

Transolver++ achieves up to 13% relative $X \in \mathbb{R}^{N \times C}$3 error reductions against prior neural operator and GNN baselines on six PDE tasks. For industrial-scale tasks (DrivAerNet++ $X \in \mathbb{R}^{N \times C}$42.5M points; aircraft $X \in \mathbb{R}^{N \times C}$5k points), Transolver++ delivers 10–20% improvement in $X \in \mathbb{R}^{N \times C}$6 field errors and coefficient $X \in \mathbb{R}^{N \times C}$7 up to $X \in \mathbb{R}^{N \times C}$8 (Luo et al., 4 Feb 2025).

Transolver-3 further extends capacity and efficiency. On DrivAerML (160M cells) it realizes fieldwise $X \in \mathbb{R}^{N \times C}$9 errors of 5.72% ($L$0), outperforming Transolver++ and AB-UPT (6.70% and 6.08% respectively), with 60% faster inference latency and 20% fewer FLOPs (Zhou et al., 4 Feb 2026).

CarBench Results

In the CarBench benchmark on DrivAerNet++, Transolver++ achieves:

• Rel L2: $L$1 (test set, N=1,154 geometries)
• MAE: $L$2 m²/s², RMSE: $L$3 m²/s²
• Median relative error: 0.09, with 1.81M parameters and mean per-geometry latency $L$4ms on A100 (Elrefaie et al., 25 Nov 2025).

Transolver++ maintains competitive accuracy and high throughput, with transformer-based solvers in general forming the Pareto frontier of accuracy vs. efficiency. Full-mesh inference ($L$5) increases MAE by $L$6–$L$7 (Elrefaie et al., 25 Nov 2025).

5. Comparative Analysis with Related Methods

The Transolver family, particularly with Transolver++ and Transolver-3, demonstrates:

• Substantial capacity gains over classic neural operators (FNO, Geo-FNO, MeshGraphNet), which either do not generalize to unstructured meshes at large $L$8 or face numerical/geometric instability.
• Explicit grouping of mesh points via learned physical states avoids $L$9 computational complexity characteristic of naïve attention or GNN message-passing (Luo et al., 4 Feb 2025).
• Transolver-3’s fast-slice and tiling improvements permit full industrial-scale deployment, setting empirical state-of-the-art on large CFD and multiphysics benchmarks (Zhou et al., 4 Feb 2026).
• In comparative CarBench evaluations, no “Transolver‐3” variant is reported; Transolver++ demonstrates reliability and efficiency, but larger models (e.g., TransolverLarge) and AB-UPT achieve marginally lower interpolation errors (Elrefaie et al., 25 Nov 2025).

6. Scalability, Efficiency, and Limitations

Single-GPU raw mesh capacity scales from $x_i$0M (Transolver) to $x_i$1M (Transolver++) to $x_i$2M (Transolver-3) by stacking architectural and algorithmic improvements (Luo et al., 4 Feb 2025, Zhou et al., 4 Feb 2026). Weak scaling across GPUs is linear for Transolver++ due to fixed per-block communication ($x_i$3).

Transolver-3’s tiling introduces extra compute associated with multiple passes per layer but reduces per-board memory requirements for practical deployment. Amortized training empowers learning from massive domains without full-batch backpropagation but may induce discretization bias if sampling granularity is insufficient. The two-stage inference assumes static global physical context; extension to time-dependent or highly non-stationary PDE regimes remains open (Zhou et al., 4 Feb 2026).

7. Outlook and Future Directions

The modular block design and parallelism of Transolver++ and Transolver-3 recommend them as the backbone for future “foundation models” in simulation-driven science and engineering. Prospective work includes extension to unsteady or multiphysics regimes, dynamic state caching for temporal flows, adaptive subdomain sampling for bias mitigation, and integration into broader digital twin or downstream design optimization pipelines (Luo et al., 4 Feb 2025, Zhou et al., 4 Feb 2026).

These works collectively mark a step change in the scalability and physical consistency of neural PDE surrogates for computational engineering and industrial design.

---

References:

• [Transolver++: An Accurate Neural Solver for PDEs on Million-Scale Geometries, (Luo et al., 4 Feb 2025)]
• [CarBench: A Comprehensive Benchmark for Neural Surrogates on High-Fidelity 3D Car Aerodynamics, (Elrefaie et al., 25 Nov 2025)]
• [Transolver-3: Scaling Up Transformer Solvers to Industrial-Scale Geometries, (Zhou et al., 4 Feb 2026)]