CarBench: Benchmark for Car Aerodynamics

• CarBench is a standardized benchmark that trains and compares neural surrogates on high-fidelity 3D automotive aerodynamics using the DrivAerNet++ dataset.
• It standardizes data preprocessing, train/validation/test splits, and evaluation protocols, focusing on surface kinematic pressure prediction for realistic vehicle geometries.
• It evaluates models on both in-distribution interpolation and cross-category generalization using metrics like Relative L2 error to assess accuracy, robustness, and physical interpretability.

Searching arXiv for CarBench and related automotive aerodynamics benchmark context. {"query":"(Elrefaie et al., 25 Nov 2025) CarBench automotive aerodynamics DrivAerNet++ benchmark", "max_results": 5} CarBench is a standardized, open benchmark for training and comparing neural surrogates on high-fidelity, three-dimensional car aerodynamics. It was introduced as the first comprehensive benchmark dedicated to large-scale 3D automotive aerodynamics, built around DrivAerNet++, the largest public dataset for automotive aerodynamics, and designed to unify data, fixed train/validation/test splits, model-agnostic training and evaluation, and uncertainty quantification for steady external-flow prediction. The first release targets per-point surface kinematic pressure prediction, $p/\rho$, on realistic vehicle geometries under a common CFD setup, with particular emphasis on reproducibility, cross-category generalization, and physically interpretable evaluation in denormalized units (Elrefaie et al., 25 Nov 2025).

1. Scope and benchmark rationale

CarBench was created in response to fragmentation in machine learning for CFD: small proprietary datasets, inconsistent preprocessing, incomparable metrics, and few tests of generalization across vehicle categories. Its central claim is not merely to provide another automotive dataset, but to define a benchmark protocol in which input sampling, normalization, train/validation/test partitions, logging, accuracy metrics, efficiency measurements, and statistical uncertainty are standardized across model families (Elrefaie et al., 25 Nov 2025).

The benchmark focuses on steady external aerodynamics for passenger-car geometries. In its initial release, the task is surface-field regression rather than full-flow reconstruction: models predict the surface kinematic pressure field $p/\rho$ on the car body and wheels. This design makes CarBench narrower than a general CFD benchmark, but more controlled for comparing neural surrogates on a large, realistic automotive corpus. A common misconception is that it already evaluates volumetric velocity, turbulence quantities, wall shear stress, or integral coefficients such as $C_D$ and $C_L$; the current release explicitly does not. Those are identified as planned extensions rather than present benchmark targets.

CarBench also distinguishes itself by evaluating both standard interpolation within official dataset splits and cross-category experiments in which models trained on one or more car archetypes are tested on unseen archetypes. This makes geometric transfer, rather than only in-distribution regression, a first-class evaluation axis.

2. Dataset, geometry, and CFD formulation

The benchmark is built on DrivAerNet++, which contains 8,150 high-fidelity steady RANS simulations of realistic car designs. The dataset spans three primary car archetypes—Fastback, Notchback, and Estateback—and eight parametric base models reflecting combinations such as smooth versus detailed underbodies and open versus closed wheels. The simulations use OpenFOAM’s simpleFoam for steady incompressible RANS with the $k$–$\omega$ SST turbulence model, and they incorporate automotive-specific boundary conditions including moving ground and rotating wheels (Elrefaie et al., 25 Nov 2025).

The geometry representation is dual-resolution. Full CFD surface meshes contain typically about $450$k nodes per car, while learning is standardized on uniformly sampled 10,000 surface points per car. For most models, these inputs include coordinates and surface normals; alternative encodings are used where appropriate, such as triplane implicit fields for TripNet and $32^3$ voxel grids for the FNO-style NeuralOperator. Predictions are evaluated both on the 10k-point representation and after interpolation back to the full CFD surface mesh, which exposes whether a method preserves spatial fidelity beyond the sampled training resolution.

The operating variable is kinematic pressure, stored as $p/\rho$ in units of $\mathrm{m}^2\,\mathrm{s}^{-2}$, consistent with OpenFOAM’s incompressible formulation. Targets are normalized during training with training-set-only statistics $p/\rho$0 and $p/\rho$1, but all benchmark metrics are reported after denormalization in physical units. This preserves comparability across methods while keeping reported errors physically interpretable.

The underlying governing equations and derived quantities are part of the benchmark’s physical framing:

$p/\rho$2

$p/\rho$3

Surface forces are related to predicted fields through

$p/\rho$4

and the usual aerodynamic coefficients are

$p/\rho$5

The pressure coefficient is

$p/\rho$6

Because the benchmark target is kinematic pressure $p/\rho$7, pressure-based integrals require conversion back to static pressure via $p/\rho$8. If $p/\rho$9 is unavailable, only pressure-drag estimates can be formed.

3. Tasks, splits, and evaluation protocol

CarBench defines two principal benchmark tasks: standard interpolation and cross-category generalization. The standard setting uses fixed official splits of approximately 5,819 training cars, 1,177 validation cars, and 1,154 test cars. These splits avoid geometry overlap and prevent leakage of normalization statistics or near-duplicate shapes across partitions. No data augmentation is used; the benchmark is intentionally configured for like-for-like comparison under identical point sampling and normalization (Elrefaie et al., 25 Nov 2025).

Cross-category generalization is evaluated zero-shot, without fine-tuning. Reported examples include training on Fastbacks only and testing on Estateback plus Notchback, which yields strong transfer with Relative L2 of approximately $C_D$0, and more difficult directions such as training on Estateback plus Notchback and testing on Fastbacks, which yields Relative L2 of approximately $C_D$1–$C_D$2. Training on two large categories and testing on the third gives the best zero-shot performance, with Relative L2 of approximately $C_D$3–$C_D$4. These results operationalize a central question in neural CFD for design: whether a surrogate learns aerodynamic structure or merely interpolates within a narrow geometry family.

The benchmark’s evaluation criteria span four axes. First, predictive accuracy is measured using MAE, MSE, RMSE, Relative L2, and $C_D$5, all in denormalized physical units, alongside percentile diagnostics such as P50, P90, P95, and P99 absolute errors and median relative error. Second, computational efficiency is measured on identical hardware—an NVIDIA A100-80GB, batch size 1—using peak memory, mean latency, throughput, and parameter count. Third, physical consistency is assessed qualitatively through coherence of stagnation regions, pressure recovery on roof and rear surfaces, and wheel/ground interactions. Fourth, statistical uncertainty is estimated through stratified paired bootstrap resampling with $C_D$6 replicates at the car-sample level, preserving spatial correlation per surface and archetype proportions.

The benchmark therefore evaluates not just regression loss, but an integrated profile of accuracy, robustness, efficiency, and uncertainty. This suggests a deliberate shift from purely ML-centric leaderboard culture toward a surrogate-model evaluation regime aligned with engineering use.

4. Model families and benchmarked architectures

CarBench evaluates eleven architectures spanning neural operators, geometric deep learning, transformer-based neural solvers, and implicit field networks. The benchmark’s purpose is not to optimize one architecture in isolation, but to place heterogeneous surrogate classes under a single protocol (Elrefaie et al., 25 Nov 2025).

Family	Architectures	Core input/output framing
Neural operator	NeuralOperator (FNO-style)	Voxelized input, per-point $C_D$7 output
Geometric deep learning	PointNet, PointNetLarge, RegDGCNN, PointTransformer, PointMAE-style encoder–decoder	Point or graph inputs, per-point $C_D$8
Transformer-based solvers	Transolver, Transolver++, TransolverLarge, AB-UPT	Point-cloud geometry or 6D point features, per-point $C_D$9
Implicit field networks	TripNet	Triplane features with continuous surface sampling

The FNO-style NeuralOperator lifts a $C_L$0 voxelized occupancy/grid-coordinate input into spectral space using eight Fourier modes per dimension and two spectral layers, then decodes to a grid and refines pointwise predictions. Point-based baselines include PointNet and PointNetLarge, while RegDGCNN uses dynamic $C_L$1-NN graphs with EdgeConv and PointTransformer uses local $C_L$2-NN self-attention with relative position encodings. The PointMAE-style encoder–decoder uses efficient pointwise convolutions with a global token and is trained with SmoothL1 loss.

The transformer-based solvers form the strongest benchmark family. Transolver uses physics-aware slice attention on irregular point sets with five layers and eight heads, taking 6D point features $C_L$3. Transolver++ retains slice attention but introduces unified position encoding and compact MLPs for parameter efficiency. TransolverLarge scales depth, width, number of heads, and number of slices for higher accuracy. AB-UPT uses anchor tokens, a low-dimensional latent simulation, and an anchored neural field decoder for high-resolution outputs, and is notable for combining high accuracy with strong memory efficiency.

TripNet represents the implicit-field line of attack. Its triplane embedding over the $C_L$4 planes provides global volumetric context together with dense local sampling, enabling continuous field-value evaluation at surface coordinates.

A methodological point implicit in CarBench is that these families are not evaluated on bespoke preprocessing or task-specific metric choices. The benchmark fixes the surrounding protocol so that differences in predictive behavior can be attributed more cleanly to architectural design.

5. Empirical findings and performance landscape

The first CarBench release reports a clear performance hierarchy. AB-UPT achieves the best overall accuracy with Relative L2 of approximately $C_L$5 and $C_L$6 of approximately $C_L$7, while also maintaining approximately $C_L$8 ms latency and only approximately $C_L$9 GB memory. The Transolver family forms a strong Pareto front: Transolver reaches Relative L2 of approximately $k$0, TransolverLarge approximately $k$1, and Transolver++ approximately $k$2, with compact parameter counts and approximately $k$3 ms inference. TripNet is also competitive, with Relative L2 of approximately $k$4 and runtime of approximately $k$5 ms, albeit at a higher parameter count of approximately $k$6M. Geometric baselines improve over classic point methods but remain weaker: PointTransformer reaches Relative L2 of approximately $k$7, RegDGCNN approximately $k$8, and classical point-network variants trail in either accuracy or efficiency (Elrefaie et al., 25 Nov 2025).

Cross-category results show that data scale is decisive. Training on the large Fastback subset transfers reasonably well to the other archetypes, while training on small categories does not generalize well. The most difficult zero-shot direction is predicting Fastbacks without any Fastback training data, where Relative L2 rises to approximately $k$9–$\omega$0. When Fastbacks are included in the training categories, transfer improves markedly. This indicates that archetype diversity and sample count are major determinants of generalization, not merely model class.

Dual-resolution evaluation reveals additional structure. All models are trained on 10k-point clouds, but when predictions are interpolated to the full CFD mesh of approximately $\omega$1 points, error increases by approximately $\omega$2–$\omega$3 across models. Transformer-based and implicit-field models preserve the best fidelity under this upsampling step, whereas point-based baselines degrade more strongly. This suggests that part of the benchmark is testing continuity and geometric regularity, not only pointwise fitting on the sampled training mesh.

Error localization is also physically meaningful. The largest errors occur near high-curvature regions such as the A-pillars and C-pillars, near stagnation zones, along sharp edges, and in wheel/ground interaction regions. Wheels are especially challenging because of rotating-wall and moving-ground boundary conditions, yet top transformer models and TripNet still maintain coherent circumferential pressure gradients. Bootstrap confidence intervals for Relative L2 are reported as tight, approximately $\omega$4–$\omega$5, and model rankings are described as statistically stable; the gap between AB-UPT and the next-best models exceeds these confidence-interval widths by several multiples.

6. Reproducibility, usage, and limitations

CarBench is designed as a reproducible benchmark framework rather than only a dataset paper. The released resources include training and evaluation scripts, bootstrap routines, pretrained weights, official splits, and preprocessed point-cloud bundles. The benchmark framework is hosted at https://github.com/Mohamedelrefaie/CarBench, while DrivAerNet++ is distributed via the Harvard Dataverse under CC BY-NC 4.0. The software stack includes a UnifiedTrainer with mixed precision when stable, gradient clipping, early stopping, standardized schedulers and optimizers, and fixed seeds, as well as a UnifiedEvaluator that reports core regression metrics, tail percentiles, latency, throughput, memory, dual-resolution results, and bootstrap confidence intervals (Elrefaie et al., 25 Nov 2025).

For practical use, the benchmark recommends AB-UPT or TransolverLarge when accuracy is the primary objective, Transolver or Transolver++ for a compact accuracy-efficiency balance, TripNet for continuous implicit-field evaluation, and NeuralOperator or PointNet baselines for rapid sanity checks and ablations. Best practices include computing $\omega$6 and $\omega$7 on the training set only, preserving the official 10k-point sampling and official splits, reporting metrics in denormalized physical units, evaluating both on the 10k-point representation and on the interpolated full mesh, and using stratified bootstrap with $\omega$8 before interpreting small metric differences.

The benchmark also has explicit limitations. It is restricted to steady RANS surface pressure at a fixed freestream condition; it does not yet include volumetric velocity or pressure fields, wall shear stress, turbulence variables, or direct benchmark targets for force and torque coefficients. Only steady RANS with the $\omega$9–$450$0 SST model is included, so broader physics coverage—such as URANS, DES/LES, multiple turbulence models, or multiple Reynolds numbers and inflow conditions—remains future work. Multi-fidelity couplings and active design loops are not yet part of the benchmark. In addition, each model was trained once to keep computation tractable, so optimizer and initialization variance are not fully characterized beyond bootstrap uncertainty on evaluation metrics.

Taken together, CarBench establishes a benchmark regime for neural CFD in which automotive realism, controlled evaluation, and reproducibility are treated as coequal requirements. Its main contribution is not only the leaderboard it induces, but the standardized experimental substrate it provides for studying how neural surrogates behave under realistic geometry variation, strict physical-unit reporting, and statistically quantified comparison.