SURF: Benchmarking Graph-Based Fluid Simulations

• SURF is a benchmark for learned graph-based fluid simulators that measures generalization across multiple transfer axes, including topology, mesh resolution, parameter range, and dynamic boundary conditions.
• It comprises seven 2D datasets of 300-step transient simulations on irregular meshes, facilitating evaluations of in-distribution accuracy versus transfer performance.
• Empirical studies with MeshGraphNets and EAGLE reveal that data augmentation and physics-informed biases can significantly improve model transfer in complex fluid dynamics scenarios.

SURF, expanded as “Simulating and Understanding the Role of Generalization in Fluids,” is a benchmark for learned graph-based fluid simulators that is designed to measure whether such models generalize rather than merely interpolate. It targets mesh-based GNN solvers for fluid dynamics and evaluates transfer across four axes that are central to a general-purpose fluid simulator: topology, mesh resolution, thermodynamic or flow-parameter range, and transient boundary conditions. The benchmark comprises seven complementary datasets together with normalized performance and generalization metrics, and it is instantiated in the paper through a comparative study of MeshGraphNets and EAGLE (Künzli et al., 2023).

1. Scope and motivation

Simulating fluid dynamics is crucial for design and development processes ranging from simple valves to complex turbomachinery, but accurately solving the underlying physical equations is computationally expensive. Learning-based solvers that model interactions on meshes have therefore gained interest because they promise orders-of-magnitude speed-ups. SURF is motivated by the observation that most published work evaluates these models primarily through in-distribution accuracy, leaving unresolved whether they capture underlying physical principles or only interpolate among familiar training configurations.

The benchmark is organized around four forms of transfer that a general-purpose fluid simulator should support: new geometries and topologies, finer or coarser mesh resolutions, wider thermodynamic and flow-parameter ranges, and transient boundary conditions. In industrial contexts such as automotive, aerospace, and turbomachinery, retraining a neural solver from scratch for every new part or operating condition is impractical. Within this framing, generalization is treated as evidence that the network has discovered an underlying physical representation rather than memorized dataset-specific patterns. SURF is therefore constructed to quantify how much performance degrades when a model crosses each of these four axes.

2. Dataset suite and simulation content

SURF comprises seven large 2D incompressible-flow datasets. Each datapoint is a 300-step transient simulation on an irregular triangular mesh, and the node-wise state is $(u,v,p,T)$, i.e. velocity, pressure, and temperature. All datasets share the same domain length of $1600\,\mathrm{mm}$ and height of $400\,\mathrm{mm}$, and they differ only in geometry parameters and boundary-condition ranges. After down-sampling the fine CFD mesh, the meshes contain on average between approximately $1.3\,\mathrm{k}$ and $5.4\,\mathrm{k}$ nodes (Künzli et al., 2023).

Dataset	Core statistics	Distinguishing variation
SURF-Base	1200 DP; 300 steps; 1287 nodes; 10.9 GB	Single cylinder, fixed orientation, two inlets, air-like material properties
SURF-Rotated	1200 DP; 300 steps; 1288 nodes; 10.8 GB	Same as Base, but domain and boundary-condition orientations sampled uniformly in $[0^\circ,360^\circ]$
SURF-Range	1200 DP; 300 steps; 1293 nodes; 10.9 GB	Base with twice as wide parameter ranges; Reynolds number from approximately $10^3$ to $1.26\times10^5$
SURF-Topology	1315 DP; 300 steps; 1751 nodes; 16.0 GB	Up to two obstacles, cylinder or one of five airfoil shapes, plus a $0^\circ$–$90^\circ$ elbow bend
SURF-Dynamic	1200 DP; 300 steps; 1287 nodes; 11.3 GB	Base with two small inlets fluctuating sinusoidally in time
SURF-Full	1241 DP; 300 steps; 1787 nodes; 16.7 GB	Superset of Base, Range, Topology, and Dynamic, but not high resolution
SURF-Mesh	1241 DP; 300 steps; 5433 nodes; 68.3 GB	Same sampling as Full, but down-sampled from a finer mesh

The individual datasets are deliberately complementary. SURF-Base fixes the canonical setting: a single cylinder, fixed orientation, constant inlet velocity and temperature, and material properties for air-like conditions. SURF-Rotated keeps the same physical content but rotates both domain and boundary conditions uniformly over the full circle, forcing the model to confront rotational variation. SURF-Range expands inlet speed, cylinder size, wall temperature, conductivity, and heat capacity over ranges that are twice as wide as in Base.

SURF-Topology changes geometry more aggressively. It allows up to two obstacles, each of which may be either a cylinder or one of five airfoil shapes with $1600\,\mathrm{mm}$0 attack, and also introduces a $1600\,\mathrm{mm}$1–$1600\,\mathrm{mm}$2 elbow bend. SURF-Dynamic retains the base geometry but replaces static small-inlet conditions with sinusoidally varying ones whose amplitude and frequency are sampled. SURF-Full combines the variations of Base, Range, Topology, and Dynamic on the coarse mesh, whereas SURF-Mesh holds the same sampling regime as Full but uses a finer mesh, isolating resolution transfer.

3. Performance and generalization metrics

SURF separates within-distribution performance from generalization performance. For a test set $1600\,\mathrm{mm}$3 and rollout horizon $1600\,\mathrm{mm}$4, the per-mode root-mean-square error for velocity is defined as (Künzli et al., 2023)

$1600\,\mathrm{mm}$5

with analogous scalar definitions for $1600\,\mathrm{mm}$6 and $1600\,\mathrm{mm}$7.

To make cross-dataset comparisons possible, SURF defines a normalized performance score

$1600\,\mathrm{mm}$8

where $1600\,\mathrm{mm}$9, $400\,\mathrm{mm}$0, and $400\,\mathrm{mm}$1 are the empirical standard deviations of velocity, pressure, and temperature on the target dataset.

Generalization is formulated as a ratio between transferred performance and an in-domain reference performance. For velocity,

$400\,\mathrm{mm}$2

where $400\,\mathrm{mm}$3 denotes a model trained on $400\,\mathrm{mm}$4 and evaluated on $400\,\mathrm{mm}$5. Analogous quantities define $400\,\mathrm{mm}$6 and $400\,\mathrm{mm}$7, and the aggregate score is

$400\,\mathrm{mm}$8

To isolate the four benchmark axes, SURF defines

$400\,\mathrm{mm}$9

The overall SURF generalization score is then

$1.3\,\mathrm{k}$0

The intended interpretation is explicit: $1.3\,\mathrm{k}$1 indicates performance loss under transfer, and larger values indicate a more severe extrapolation break. This metric design makes generalization degradation itself the primary object of comparison rather than a secondary by-product of ordinary test error.

4. Evaluation protocol

Each dataset is split into $1.3\,\mathrm{k}$2 train, $1.3\,\mathrm{k}$3 validation, and $1.3\,\mathrm{k}$4 test by random shuffling of whole simulations, with no overlap of geometries. Held-out geometry and held-out physics are evaluated by training on SURF-Base and testing on SURF-Topology, SURF-Range, or SURF-Dynamic. Resolution generalization is evaluated by training on SURF-Full, which uses the coarse mesh, and testing on SURF-Mesh, which uses the fine mesh (Künzli et al., 2023).

SURF-Rotated functions as a controlled augmentation dataset: it is effectively a $1.3\,\mathrm{k}$5 rotation augmentation of Base, introducing no extra boundary-condition types but requiring the GNN to learn rotational invariance. Normalization is applied through dataset-wide centering and scaling of node-wise features, while the performance score uses standard-deviation normalization so that scores remain comparable across datasets.

The protocol is therefore not limited to a single train-test split. It specifies a family of train-on-one, test-on-another transfers that operationalize different notions of generalization. This is important because the same model may behave very differently when confronted with unseen mesh density, unseen topology, or unseen temporal forcing.

5. Empirical study with MeshGraphNets and EAGLE

The benchmark paper evaluates two leading mesh-based GNN simulators: MeshGraphNets (MGN), described as a pure message-passing GNN, and EAGLE, described as an encode-process-decode architecture extended with global attention and clustering. Both models were slightly adapted to predict temperature and trained with 7-step supervision, and all reported scores are means over 3 runs (Künzli et al., 2023).

Dataset	MGN	EAGLE
Base	$1.3\,\mathrm{k}$6; $1.3\,\mathrm{k}$7; $1.3\,\mathrm{k}$8; $1.3\,\mathrm{k}$9	$5.4\,\mathrm{k}$0; $5.4\,\mathrm{k}$1; $5.4\,\mathrm{k}$2; $5.4\,\mathrm{k}$3
Rotated	$5.4\,\mathrm{k}$4; $5.4\,\mathrm{k}$5; $5.4\,\mathrm{k}$6; $5.4\,\mathrm{k}$7	$5.4\,\mathrm{k}$8; $5.4\,\mathrm{k}$9; $[0^\circ,360^\circ]$0; $[0^\circ,360^\circ]$1
Full	$[0^\circ,360^\circ]$2; $[0^\circ,360^\circ]$3; $[0^\circ,360^\circ]$4; $[0^\circ,360^\circ]$5	$[0^\circ,360^\circ]$6; $[0^\circ,360^\circ]$7; $[0^\circ,360^\circ]$8; $[0^\circ,360^\circ]$9

After 250 steps, both models achieve similar performance on SURF-Base. EAGLE is consistently better on pressure, whereas MGN is slightly better on temperature. SURF-Full, the most complex dataset, produces the largest overall performance score for both models.

Model	Generalization scores	SURF_GS
MGN	GS-Mesh $10^3$0; GS-Topology $10^3$1; GS-Range $10^3$2; GS-Dynamic $10^3$3	$10^3$4
EAGLE	GS-Mesh $10^3$5; GS-Topology $10^3$6; GS-Range $10^3$7; GS-Dynamic $10^3$8	$10^3$9

These results expose a marked divergence between in-distribution accuracy and transfer behavior. MGN generalizes better to new topologies, with GS-Topology $1.26\times10^5$0 versus $1.26\times10^5$1 for EAGLE, but performs very poorly on dynamic boundary conditions, with GS-Dynamic $1.26\times10^5$2. EAGLE excels on resolution, parameter-range, and dynamic transfer, producing GS-Mesh approximately $1.26\times10^5$3, GS-Range $1.26\times10^5$4, and GS-Dynamic $1.26\times10^5$5, which yields an overall SURF_GS that is approximately $1.26\times10^5$6 lower.

The choice of training set also changes the picture substantially. Training on SURF-Rotated is reported to dramatically improve transfer to all other domains for both MGN and EAGLE; for example, GS to Full drops to approximately $1.26\times10^5$7 for MGN and $1.26\times10^5$8 for EAGLE. By contrast, training on Dynamic data helps only for dynamic tests and not for the other axes.

6. Interpretation, limitations, and research directions

The benchmark establishes that current GNN-based solvers can learn accurate in-distribution flows, but that this accuracy does not imply robust transfer. Their success is mixed across variables, especially between pressure and temperature, and their generalization degrades sharply when asked to predict under new topology or dynamic boundary-condition regimes. The paper interprets this behavior as indicating a reliance on memorized patterns. It also reports that attention-based EAGLE layers and clustering markedly help with long-range interactions, improving extrapolation across resolution and parameter range (Künzli et al., 2023).

Several recommendations follow directly from the reported findings. Rotation-based data augmentation is presented as a simple yet powerful tool for improving generalization across most axes. Physics-informed inductive biases, specifically symmetry and conservation laws, are proposed as mechanisms that may reduce over-fitting to training-set specifics. Scalability to 3D and high-resolution problems is identified as an open requirement, with memory-efficient GNN architectures and multi-scale pooling named as candidate directions. Continual or meta-learning is also suggested as a way to adapt to new geometries or regimes on the fly.

A plausible implication is that SURF reframes the evaluation of learned fluid simulators around transfer failure modes rather than raw rollout fidelity alone. In that sense, it functions not just as a dataset collection but as a measurement framework for general-purpose neural simulation. The benchmark is released as an open-source testbed with datasets, metrics, and scripts; the repository is at https://github.com/s-kuenzli/surf-fluidsimulation, and the dataset is hosted at https://huggingface.co/datasets/SURF-FluidSimulation/FluidSimulation.