CFDLLMBench: CFD LLM Benchmark

• The paper introduces CFDLLMBench as the first benchmark suite to holistically evaluate LLMs on CFD tasks across conceptual understanding, numerical reasoning, and workflow implementation.
• It employs a three-tier architecture—CFDQuery, CFDCodeBench, and FoamBench—to assess graduate-level questions, code-based solver generation, and realistic OpenFOAM workflows.
• The benchmark emphasizes executable, physically accurate outputs, revealing significant challenges in LLM-driven CFD automation and numerical stability.

CFDLLMBench is a benchmark suite for evaluating LLMs in computational fluid dynamics (CFD), introduced to measure not only CFD fact recall but the full progression from conceptual knowledge to numerical reasoning to implementation of realistic simulation workflows (Somasekharan et al., 19 Sep 2025). It is presented as the first benchmark suite designed to evaluate LLMs holistically in CFD, using three components—CFDQuery, CFDCodeBench, and FoamBench—to target three competencies the benchmark identifies as necessary for LLM-driven CFD automation: graduate-level CFD knowledge, numerical and physical reasoning in CFD, and context-dependent implementation of CFD workflows (Somasekharan et al., 19 Sep 2025).

1. Domain rationale and benchmark philosophy

CFD is described as a core workhorse of computational science, used in aerospace, climate, urban physics, robotics, and many engineering applications (Somasekharan et al., 19 Sep 2025). The benchmark’s premise is that CFD is an unusually demanding test domain for LLMs because actual CFD work combines graduate-level fluid mechanics, numerical analysis, physical modeling, mesh generation, solver setup, long-context file editing, and interaction with domain-specific software. Its workflows are labor-intensive and error-prone, involving geometry definition, meshing, boundary and initial conditions, discretization choices, turbulence modeling, solver configuration, and post-processing. Small implementation mistakes can still yield executable code but physically wrong answers (Somasekharan et al., 19 Sep 2025).

This framing is central to CFDLLMBench. The benchmark does not treat scientific capability as a synonym for code executability or textual plausibility. Instead, it is built around the claim that a useful CFD assistant must pass three thresholds simultaneously: it must know CFD, reason numerically and physically about CFD, and implement context-sensitive workflows in real solver ecosystems. The benchmark therefore prioritizes executable, physically accurate, and numerically convergent outputs over text similarity, and it emphasizes reproducibility through open-source software and Dockerized infrastructure (Somasekharan et al., 19 Sep 2025).

2. Three-tier architecture and task taxonomy

CFDLLMBench is structured as three progressively harder components. CFDQuery targets conceptual understanding, CFDCodeBench targets turning physics problem statements into executable numerical solvers, and FoamBench targets practical OpenFOAM workflow generation from natural-language engineering requests. Together they define the benchmark’s task taxonomy (Somasekharan et al., 19 Sep 2025).

Component	Primary competency	Output form
CFDQuery	Graduate-level CFD knowledge	Multiple-choice option number
CFDCodeBench	Numerical and physical reasoning in CFD	Runnable Python solver writing final-time .npy arrays
FoamBench	Context-dependent implementation of CFD workflows	Full OpenFOAM case directory and successful execution

The taxonomy is explicitly progressive. CFDQuery tests whether a model can answer graduate-level questions. CFDCodeBench moves from recognition to construction by requiring complete runnable solvers for PDE problems. FoamBench adds the full operational burden of a real solver ecosystem: project structure, multiple configuration files, meshing, solver dictionaries, command-line workflow steps, and physically meaningful case execution in OpenFOAM v10 (Somasekharan et al., 19 Sep 2025).

The benchmark creation process is expert-driven throughout. Six CFD experts with advanced training participated, including doctoral students, a master’s student, an undergraduate, a postdoc, and a professor, with the postdoc and professor reviewing the others’ work. The benchmark does not use inter-annotator agreement in the usual NLP sense because answers are objective scientific solutions; instead it uses iterative independent review and revision for quality control. All source material is open or newly authored, and the full benchmark is released under BSD 3-Clause with code and data on GitHub (Somasekharan et al., 19 Sep 2025).

3. CFDQuery: graduate-level knowledge evaluation

CFDQuery is the knowledge tier. In the main text it is described as 108 multiple-choice questions, though the introduction also mentions 90 questions; the dataset overview section states 108, which is the fuller figure given in the paper (Somasekharan et al., 19 Sep 2025). The questions were curated by CFD experts from graduate lecture notes and vetted online sources, then reviewed and solved by experts before inclusion.

The task format is minimal: the input is a multiple-choice CFD question with four options, and the output is the option number. The content, however, is explicitly graduate-level. Appendix details described in the paper include PDEs, numerical methods, and error analysis spanning 1D advection, diffusion, Burgers equation, modified-equation analysis of central differences with RK3, Lax–Friedrichs dissipation, upwind bias, high-order stencils such as fourth-order central and compact schemes, WENO, nonuniform and curvilinear grids, geometric conservation law, multidimensional flows including Poisson, Navier–Stokes cavity and channel flow, Rayleigh–Bénard convection, KdV–Burgers, and discontinuous Galerkin topics (Somasekharan et al., 19 Sep 2025).

This component primarily measures graduate-level CFD knowledge. Its evaluation metric is plain accuracy, termed Success Rate, defined as the fraction of questions answered correctly. In the reported experiments, closed models perform strongly on this tier: Success Rate ranges from 60% for Gemma-2-9B-IT to 92% for o3-mini, with o3-mini best overall (Somasekharan et al., 19 Sep 2025).

4. CFDCodeBench: executable numerical solver generation

CFDCodeBench is the middle tier and is designed to probe numerical and physical reasoning in CFD. It contains 24 CFD programming problems in Python. Each input is a structured natural-language problem statement specifying the governing PDE, boundary and initial conditions, domain, variables to save, and optionally a suggested numerical method; each output is complete runnable Python code that numerically solves the problem and writes final-time solution arrays to .npy files (Somasekharan et al., 19 Sep 2025).

The task set spans both 1D and 2D problems, linear and nonlinear PDEs, and multiple areas relevant to CFD: fluid mechanics, heat transfer, and turbulence. Representative tasks listed in the benchmark description include 1D Burgers, 1D diffusion, 1D linear and nonlinear convection, Euler shock tube flow, 2D Burgers, 2D convection, 2D diffusion, 2D inviscid Burgers, 2D Laplace and Poisson equations, lid-driven cavity Navier–Stokes, channel flow with body force, steady and unsteady heat equations, fully developed turbulent channel flow with a Cess model, KdV–Burgers, horizontally periodic Rayleigh–Bénard convection, periodic incompressible shear flow with passive tracer, flow past a circular cylinder via streamfunction-vorticity in polar coordinates, Lane–Emden, and a pipe-flow linear stability eigenvalue problem (Somasekharan et al., 19 Sep 2025).

Benchmark construction is standardized. Problems were curated from public educational CFD repositories and solver projects, especially CFD Python’s “12 Steps to Navier-Stokes,” ENGR 491 materials, and seven harder problems from Dedalus. The easier 17 problems came with tutorials or reference code; for the seven harder Dedalus-based problems, experts inspected source code and authored accompanying descriptions. Average code length is about 70 lines. Prompt generation follows a reproducible JSON-to-natural-language pipeline, with fields such as equation, boundary conditions, initial conditions, domain, save values, and numerical method rendered into a fixed prompt template. The generated code is extracted and executed in a sandboxed environment with a 60-second timeout (Somasekharan et al., 19 Sep 2025).

Evaluation is execution-based and deliberately stricter than ordinary code benchmarks. The reported metrics are executability, solution accuracy, numerical convergence, and an aggregated Success Rate. Executability is binary, with $M_{\mathrm{exec}} = 1$ if the Python code runs successfully, else 0. Accuracy is based on normalized mean squared error against the reference final-time solution:

$$\mathrm{NMSE}\% = \frac{\sum_{i=1}^N (y_i - \hat{y}_i)^2}{\sum_{i=1}^N y_i^2} \times 100, \qquad M_{\mathrm{NMSE}} = \begin{cases} 1, & \mathrm{NMSE}\le 10\% \ 0.5, & 10\% < \mathrm{NMSE}\le 30\% \ 0, & \mathrm{NMSE}>30\% \end{cases}$$

If output and reference arrays differ in shape, interpolation is used before comparison. Numerical convergence is checked by refining the spatial and temporal discretizations; if the relative error decreases under refinement, the solution is considered convergent and receives $M_{\mathrm{conv}} = 1$, otherwise 0. The strict per-problem success criterion is:

$$M_{\mathrm{success}}^{(i)} = \begin{cases} 1, & M_{\mathrm{exec}}^{(i)} = 1 \;\wedge\; M_{\mathrm{NMSE}}^{(i)} = 1 \;\wedge\; M_{\mathrm{conv}}^{(i)} = 1 \ 0, & \text{otherwise} \end{cases}, \qquad \mathrm{Success\ Rate} = \frac{1}{K}\sum_{i=1}^{K} M_{\mathrm{success}}^{(i)}$$

where $K$ is the total number of problems (Somasekharan et al., 19 Sep 2025).

This evaluation design is a defining feature of CFDLLMBench. The benchmark explicitly rejects text similarity as a proxy for scientific correctness because CFD code can be implemented in many correct ways. In the reported results, the best Success Rate on CFDCodeBench is only about 14%. Most closed models execute code in over 60% of cases, and o3-mini executes in 75% of cases, while Sonnet 3.5 has only 42% executability. The paper’s central observation on this tier is that executability is much higher than physically accurate, convergent success (Somasekharan et al., 19 Sep 2025).

5. FoamBench: OpenFOAM workflow generation

FoamBench is the most practically oriented component. It targets context-dependent implementation of CFD workflows using OpenFOAM as the underlying environment. The benchmark uses OpenFOAM v10 in experiments. A typical case requires generation of a valid case directory, proper folder structure, multiple configuration files, correct meshing and solver setup, successful execution, and physically accurate results (Somasekharan et al., 19 Sep 2025).

FoamBench contains 126 OpenFOAM cases across more than 15 geometry or physics scenarios, split into 110 Basic and 16 Advanced. A typical OpenFOAM case has 6–7 configuration files totaling roughly 300–600 lines. Inputs are concise but sufficient natural-language prompts; outputs are the full set of OpenFOAM input files in proper folder structure, plus successful case execution. Prompts specify the physical scenario, geometry and domain, solver, boundary conditions, relevant properties such as viscosity or Prandtl number, turbulence models if needed, timestep, and write interval (Somasekharan et al., 19 Sep 2025).

The Basic split is derived from 11 official OpenFOAM tutorials with 10 variants each, created by changing boundary conditions or parameter values. The 11 tutorial roots are BernardCells, cavity, counterFlowFlame2D, cylinder, damBreakWithObstacle, forwardStep, obliqueShock, pitzDaily, shallowWaterWithSquareBump, squareBend, and wedge. These cover laminar and turbulent incompressible flow, compressible flow, free-surface flow, buoyancy-driven convection, combustion, shallow water, and shock problems (Somasekharan et al., 19 Sep 2025).

The Advanced split is designed specifically to prevent simple tutorial retrieval and force extrapolation. Its 16 hand-authored expert cases introduce three kinds of challenge: turbulence model changes, geometric modifications, and unseen geometry. Models may need to select a different turbulence model, create extra files required by that model, alter domain size and mesh definitions, or generate entirely new obstacle geometries using OpenFOAM meshing tools such as blockMesh. The paper highlights cases such as incompressible turbulent flow over a 2D diamond obstacle with the $k$-$\epsilon$ RANS model and pimpleFoam, and a “doubleSquare” case where the model must infer obstacle placement from text and generate an appropriate mesh (Somasekharan et al., 19 Sep 2025).

FoamBench evaluation uses four component metrics plus a strict Success Rate. Executability $M_{\mathrm{exec}}$ is binary and tests whether the generated OpenFOAM case runs successfully. Folder and file structure $M_{\mathrm{struct}}$ uses ROUGE similarity between generated and reference case structures. File similarity $M_{\mathrm{file}}$ uses ROUGE on file contents. Physical accuracy $$\mathrm{NMSE}\% = \frac{\sum_{i=1}^N (y_i - \hat{y}_i)^2}{\sum_{i=1}^N y_i^2} \times 100, \qquad M_{\mathrm{NMSE}} = \begin{cases} 1, & \mathrm{NMSE}\le 10\% \ 0.5, & 10\% < \mathrm{NMSE}\le 30\% \ 0, & \mathrm{NMSE}>30\% \end{cases}$$0 uses the same NMSE-based scoring scheme as CFDCodeBench, comparing generated and reference final-time solutions. However, the reported holistic Success Rate requires only successful execution and full physical accuracy, meaning cases where both $$\mathrm{NMSE}\% = \frac{\sum_{i=1}^N (y_i - \hat{y}_i)^2}{\sum_{i=1}^N y_i^2} \times 100, \qquad M_{\mathrm{NMSE}} = \begin{cases} 1, & \mathrm{NMSE}\le 10\% \ 0.5, & 10\% < \mathrm{NMSE}\le 30\% \ 0, & \mathrm{NMSE}>30\% \end{cases}$$1 and $$\mathrm{NMSE}\% = \frac{\sum_{i=1}^N (y_i - \hat{y}_i)^2}{\sum_{i=1}^N y_i^2} \times 100, \qquad M_{\mathrm{NMSE}} = \begin{cases} 1, & \mathrm{NMSE}\le 10\% \ 0.5, & 10\% < \mathrm{NMSE}\le 30\% \ 0, & \mathrm{NMSE}>30\% \end{cases}$$2 equal 1. Structure and file similarity are diagnostic rather than pass criteria (Somasekharan et al., 19 Sep 2025).

This design exposes a characteristic failure mode of LLM-generated scientific workflows: a model may produce roughly correct directory trees and partially similar files yet still fail physically. The paper reports concrete failure modes including inconsistent patch or patch field definitions, missing required files such as blockMeshDict or controlDict, undefined OpenFOAM keywords such as missing flux schemes, numerical instability from poor timestep or discretization choices, and geometry or mesh errors such as invalid cells or malformed obstacle definitions. Deficient spatial reasoning is singled out as a major bottleneck (Somasekharan et al., 19 Sep 2025).

6. Experimental results, failure modes, and limitations

The experimental evaluation covers five proprietary models—Claude Sonnet 3.5, o3-mini, Gemini 2.5 Flash, Claude Haiku 3.5, GPT-4o—and one open model, Gemma-2-9B-IT. Temperature is set to 0 where possible. CFDQuery and CFDCodeBench are evaluated zero-shot. FoamBench is evaluated both zero-shot and with agent frameworks, specifically MetaOpenFOAM and Foam-Agent. These frameworks include retrieval-augmented generation (RAG), generation or execution loops, and a Reviewer that inspects failures and supports retry. Three framework variants are studied: RAG + Reviewer, RAG + no Reviewer, and no RAG + Reviewer; absence of both corresponds to zero-shot prompting (Somasekharan et al., 19 Sep 2025).

The main empirical trend across the benchmark is a steep decline from knowledge recall to practical CFD execution. On CFDQuery, closed models perform well, with Success Rate ranging from 60% for Gemma-2-9B-IT to 92% for o3-mini. On CFDCodeBench, the best reported Success Rate is only about 14%, despite much higher executability. On FoamBench, the best model-framework combination reaches 34% Success Rate on Basic and 25% on Advanced. This best result is Foam-Agent with Sonnet 3.5, RAG, and Reviewer (Somasekharan et al., 19 Sep 2025).

For FoamBench Basic under that setting, Sonnet 3.5 achieves $$\mathrm{NMSE}\% = \frac{\sum_{i=1}^N (y_i - \hat{y}_i)^2}{\sum_{i=1}^N y_i^2} \times 100, \qquad M_{\mathrm{NMSE}} = \begin{cases} 1, & \mathrm{NMSE}\le 10\% \ 0.5, & 10\% < \mathrm{NMSE}\le 30\% \ 0, & \mathrm{NMSE}>30\% \end{cases}$$3, $$\mathrm{NMSE}\% = \frac{\sum_{i=1}^N (y_i - \hat{y}_i)^2}{\sum_{i=1}^N y_i^2} \times 100, \qquad M_{\mathrm{NMSE}} = \begin{cases} 1, & \mathrm{NMSE}\le 10\% \ 0.5, & 10\% < \mathrm{NMSE}\le 30\% \ 0, & \mathrm{NMSE}>30\% \end{cases}$$4, $$\mathrm{NMSE}\% = \frac{\sum_{i=1}^N (y_i - \hat{y}_i)^2}{\sum_{i=1}^N y_i^2} \times 100, \qquad M_{\mathrm{NMSE}} = \begin{cases} 1, & \mathrm{NMSE}\le 10\% \ 0.5, & 10\% < \mathrm{NMSE}\le 30\% \ 0, & \mathrm{NMSE}>30\% \end{cases}$$5, $$\mathrm{NMSE}\% = \frac{\sum_{i=1}^N (y_i - \hat{y}_i)^2}{\sum_{i=1}^N y_i^2} \times 100, \qquad M_{\mathrm{NMSE}} = \begin{cases} 1, & \mathrm{NMSE}\le 10\% \ 0.5, & 10\% < \mathrm{NMSE}\le 30\% \ 0, & \mathrm{NMSE}>30\% \end{cases}$$6, and Success Rate $$\mathrm{NMSE}\% = \frac{\sum_{i=1}^N (y_i - \hat{y}_i)^2}{\sum_{i=1}^N y_i^2} \times 100, \qquad M_{\mathrm{NMSE}} = \begin{cases} 1, & \mathrm{NMSE}\le 10\% \ 0.5, & 10\% < \mathrm{NMSE}\le 30\% \ 0, & \mathrm{NMSE}>30\% \end{cases}$$7. On Advanced it achieves $$\mathrm{NMSE}\% = \frac{\sum_{i=1}^N (y_i - \hat{y}_i)^2}{\sum_{i=1}^N y_i^2} \times 100, \qquad M_{\mathrm{NMSE}} = \begin{cases} 1, & \mathrm{NMSE}\le 10\% \ 0.5, & 10\% < \mathrm{NMSE}\le 30\% \ 0, & \mathrm{NMSE}>30\% \end{cases}$$8, $$\mathrm{NMSE}\% = \frac{\sum_{i=1}^N (y_i - \hat{y}_i)^2}{\sum_{i=1}^N y_i^2} \times 100, \qquad M_{\mathrm{NMSE}} = \begin{cases} 1, & \mathrm{NMSE}\le 10\% \ 0.5, & 10\% < \mathrm{NMSE}\le 30\% \ 0, & \mathrm{NMSE}>30\% \end{cases}$$9, $M_{\mathrm{conv}} = 1$0, $M_{\mathrm{conv}} = 1$1, and Success Rate $M_{\mathrm{conv}} = 1$2. GPT-4o under the same Foam-Agent RAG+Reviewer setup ties Sonnet’s Advanced Success Rate at 0.250, but is lower on Basic at 0.282. o3-mini reaches 0.264 on Basic and 0.187 on Advanced. Gemma-2-9B-IT scores 0 across FoamBench in all reported framework settings (Somasekharan et al., 19 Sep 2025).

Zero-shot FoamBench performance is near zero even for strong models. For Sonnet 3.5 zero-shot, Basic scores are $M_{\mathrm{conv}} = 1$3, $M_{\mathrm{conv}} = 1$4, $M_{\mathrm{conv}} = 1$5, $M_{\mathrm{conv}} = 1$6, and Success Rate $M_{\mathrm{conv}} = 1$7; Advanced scores are $M_{\mathrm{conv}} = 1$8 and $M_{\mathrm{conv}} = 1$9, respectively. Across all models in pure zero-shot, almost all Basic and Advanced Success Rates are 0, except Sonnet 3.5 at 4.5% and 0.7%, and o3-mini at 0.9% and 0% (Somasekharan et al., 19 Sep 2025).

The agentic ablations show that both RAG and Reviewer matter. For Sonnet 3.5 on FoamBench Basic with Foam-Agent, Success Rate is 0.336 with RAG+Reviewer, 0.200 with RAG+no Reviewer, and 0.245 with no RAG+Reviewer. On Advanced, the same sequence is 0.250, 0.125, and 0.125. Foam-Agent generally outperforms MetaOpenFOAM: for Sonnet 3.5 on Basic with RAG+Reviewer, MetaOpenFOAM reaches 0.136 while Foam-Agent reaches 0.336; on Advanced, 0.125 versus 0.250 (Somasekharan et al., 19 Sep 2025).

The paper also reports cost figures for these agentic workflows. With RAG+Reviewer on FoamBench, Sonnet 3.5 in Foam-Agent costs on average \$M_{\mathrm{success}}^{(i)} = \begin{cases} 1, & M_{\mathrm{exec}}^{(i)} = 1 \;\wedge\; M_{\mathrm{NMSE}}^{(i)} = 1 \;\wedge\; M_{\mathrm{conv}}^{(i)} = 1 \ 0, & \text{otherwise} \end{cases}, \qquad \mathrm{Success\ Rate} = \frac{1}{K}\sum_{i=1}^{K} M_{\mathrm{success}}^{(i)}$00.42 for GPT-4o and \$0.27 for o3-mini. Thus the strongest reported OpenFOAM performance is also relatively expensive (Somasekharan et al., 19 Sep 2025).

The benchmark’s broader conclusion is that current frontier LLMs can recall CFD knowledge far better than they can reason numerically or implement robust scientific workflows. In CFDCodeBench, failures often come from incorrect boundary conditions, unstable schemes, or lack of convergence despite runnable code. In FoamBench, models may generate plausible file trees and partially similar files but still fail physically. The paper explicitly warns that coding metrics alone overestimate capability (Somasekharan et al., 19 Sep 2025).

Two main limitations are identified. First, there are no human baselines yet, largely because defining a fair human baseline is difficult when expertise varies; future work is suggested on timing-based expert baselines. Second, the reported baselines do not include extensive automated prompt tuning, so stronger prompt engineering or automatically designed agent frameworks may improve results. More broadly, the benchmark highlights open challenges in robust physical reasoning, numerical stability awareness, turbulence-model selection, long-context consistency across many files, solver-specific configuration knowledge, and especially geometry and spatial reasoning for mesh generation (Somasekharan et al., 19 Sep 2025).