CFDCodeBench: CFD Code Generation Benchmark

• CFDCodeBench is a benchmark that tests large language models' ability to generate complete, executable CFD Python code from structured natural language descriptions.
• It covers 24 diverse tasks in 1D and 2D settings with canonical PDEs, using a JSON-to-prompt transformation for clarity and reproducibility.
• The benchmark rigorously measures metrics such as script executability, physical accuracy via NMSE, and numerical convergence to ensure true scientific correctness.

CFDCodeBench is the code-generation component of CFDLLMBench, a benchmark suite for evaluating LLMs on computational fluid dynamics tasks that extend beyond factual recall. It is designed to test whether a model can transform a structured physical and numerical problem description into complete runnable Python code for a CFD-style numerical experiment, with required outputs saved in NumPy format and judged not only by executability but also by physical accuracy and numerical convergence. Within the three-part structure of CFDLLMBench, CFDCodeBench occupies the middle tier between CFDQuery, which tests graduate-level conceptual knowledge, and FoamBench, which tests full OpenFOAM workflow construction and execution (Somasekharan et al., 19 Sep 2025).

1. Position and benchmark objective

CFDCodeBench targets what CFDLLMBench defines as numerical and physical reasoning, expressed through executable scientific programming. The benchmark asks whether a model can read a natural-language description of a CFD problem, understand the governing PDE, domain, initial and boundary conditions, and requested outputs, choose or apply an appropriate numerical method, generate complete runnable Python code, save the required variables in .npy format, and produce a solution that is physically accurate and numerically convergent. In this framing, the benchmark is not a generic code-synthesis task; it is a test of context-dependent implementation of CFD workflows at the single-script level (Somasekharan et al., 19 Sep 2025).

A common misconception is that CFDCodeBench measures general software engineering ability in the style of code completion or bug fixing. It does not. The tasks are framed as from-scratch writing of CFD code from structured problem descriptions. The benchmark does not describe code editing, debugging given code, completion of partial templates, or hidden test-driven implementation from function signatures. A second misconception is that executable code is sufficient. CFDCodeBench explicitly rejects that criterion: runnable scripts that violate physics or fail refinement-based convergence do not count as successful solutions (Somasekharan et al., 19 Sep 2025).

2. Task design, scope, and coverage

CFDCodeBench contains 24 CFD programming problems spanning 1D and 2D settings, linear and nonlinear PDEs, and topics in fluid mechanics, thermal transport, and turbulence, with scripts averaging about 70 lines of code per problem. No separate train/validation/test split is described; it is treated as a benchmark set of 24 tasks (Somasekharan et al., 19 Sep 2025).

The task set covers canonical transport equations, elliptic and heat-equation problems, incompressible Navier–Stokes configurations, compressible flow, buoyancy-driven flow, turbulence modeling, and linear stability analysis. The benchmark therefore mixes introductory finite-difference-style problems with more advanced tasks derived from spectral-method workflows and turbulence models.

Category	Representative problems
Canonical transport and model PDEs	1D linear convection, 2D Burgers, 2D inviscid Burgers, 1D Korteweg–de Vries/Burgers
Elliptic and heat problems	2D Laplace equation, 2D Poisson equation, 2D steady heat equation, Lane–Emden equation
CFD configurations	2D Navier–Stokes in a cavity, channel flow with Navier–Stokes, periodic incompressible shear flow with a passive tracer field
Advanced tasks	Euler shock tube, 2D horizontally periodic Rayleigh–Bénard convection, fully-developed turbulent channel flow with a Cess turbulence model, linear stability eigenvalue problem for pipe flow

The coverage is broad in boundary conditions and geometry. The benchmark includes Dirichlet, Neumann, mixed Dirichlet/Neumann, periodic, reflective, constant boundary values, and no-slip conditions, and it is mostly defined on simple 1D intervals and 2D rectangular or square domains, with at least one non-Cartesian case: flow past circular cylinder in streamfunction-vorticity formulation in polar coordinates. At the same time, the scope is explicitly limited to 1D and 2D tasks; no 3D CFDCodeBench tasks are reported (Somasekharan et al., 19 Sep 2025).

3. Data curation, prompt specification, and required outputs

The benchmark was curated from publicly available GitHub repositories and established numerical solver packages. Its three stated sources are CFD Python: the 12 Steps to Navier-Stokes Equations, ENGR 491 - Computational Fluid Dynamics, and the Dedalus Project. The dataset contains 17 foundational problems from CFD Python and ENGR 491 and 7 advanced problems from Dedalus. Because the Dedalus-derived tasks were more difficult and lacked detailed tutorials, CFD experts reviewed the source code and authored accompanying problem descriptions. Across CFDLLMBench, six CFD domain experts participated: two doctoral students, one master’s student, one undergraduate student, one postdoc, and one professor; the postdoc and professor reviewed the work of the other contributors at each step (Somasekharan et al., 19 Sep 2025).

CFDCodeBench uses a structured JSON-to-natural-language prompt generation pipeline. Each problem is represented initially as a JSON object with fields such as equation, boundary conditions, initial conditions, domain, save values, and optionally numerical method. This JSON is transformed into a standardized natural-language prompt. The design goal is to preserve problem structure and reproducibility while reducing prompt ambiguity (Somasekharan et al., 19 Sep 2025).

The required model output is narrowly specified. The generated response must be complete runnable Python code and nothing else. If the problem is unsteady, only the final time step is to be saved. Each requested variable must be written as a separate NumPy .npy file using the same variable names as in save_values. For 1D problems, each saved variable must be a 1D array; for 2D problems, each saved variable must be a 2D array. The benchmark therefore evaluates not just PDE discretization and time integration, but also adherence to a fixed execution-and-output contract (Somasekharan et al., 19 Sep 2025).

4. Evaluation protocol and metric structure

CFDCodeBench uses four benchmark metrics: three component metrics and one aggregate success statistic. The core design principle is that script validity, physical fidelity, and refinement behavior are distinct properties and must all be measured (Somasekharan et al., 19 Sep 2025).

Metric	Definition
Executability $M_{\mathrm{exec}}$	Binary score: 1 if generated Python code executes successfully, 0 otherwise
Physical accuracy $M_{\mathrm{NMSE}}$	Derived from NMSE% at final time with thresholds 1, 0.5, or 0
Numerical convergence $M_{\mathrm{conv}}$	Binary score: 1 if error decreases under spatial and temporal refinement
Success Rate	Fraction of problems with $M_{\mathrm{exec}}=1$, $M_{\mathrm{NMSE}}=1$, and $M_{\mathrm{conv}}=1$ simultaneously

Physical correctness is measured by a normalized mean squared error percentage at the final time: $$\mathrm{NMSE}\% = \frac{\sum_{i=1}^{N}(y_i-\hat{y}_i)^2}{\sum_{i=1}^{N}y_i^2}\times 100,$$ with score mapping

$$M_{\mathrm{NMSE}}= \begin{cases} 1, & \mathrm{NMSE}\le 10\%,\ 0.5, & 10\%<\mathrm{NMSE}\le 30\%,\ 0, & \mathrm{NMSE}>30\%. \end{cases}$$

If predicted and reference arrays have different shapes, the benchmark interpolates the predicted solution to align dimensions before comparison. Numerical convergence is assessed by refining both mesh spacing and time step; if the error decreases with refinement, the solution is awarded a score of 1, and 0 otherwise (Somasekharan et al., 19 Sep 2025).

The aggregate success criterion is intentionally strict: $$M_{\mathrm{success}}^{(i)}= \begin{cases} 1, & M_{\mathrm{exec}}^{(i)}=1 \;\wedge\; M_{\mathrm{NMSE}}^{(i)}=1 \;\wedge\; M_{\mathrm{conv}}^{(i)}=1,\ 0, & \text{otherwise}, \end{cases}$$ and the benchmark-level score is

$$\mathrm{Success\ Rate}=\frac{1}{K}\sum_{i=1}^{K} M_{\mathrm{success}}^{(i)}.$$

This formulation means that executable code with partial accuracy or non-convergent behavior does not count as a success. The benchmark therefore differs fundamentally from pass@1-style coding evaluations, because it treats numerical analysis properties as first-class evaluation targets (Somasekharan et al., 19 Sep 2025).

Execution is performed in a sandboxed environment with a default timeout of 60 seconds. The benchmark does not rely on hidden tests, symbolic equivalence, or unit-test suites in the HumanEval sense. Its orchestration pipeline is execution-based: task JSON is converted to a prompt, the model response is parsed as Python code, the code is executed, .npy outputs are collected, NMSE is computed against a trusted reference, convergence under refinement is tested, and the component metrics and Success Rate are then computed (Somasekharan et al., 19 Sep 2025).

5. Empirical results and observed failure modes

CFDCodeBench proved difficult for all evaluated models. The headline result is that the best-performing model achieved only about 14% Success Rate. At the same time, most closed-weight models produced executable Python code in over 60% of cases, o3-mini executed successfully in about 75% of cases, and Claude Sonnet 3.5 was the weakest closed model in executability at about 42%. The benchmark’s central empirical message is therefore the gap between “code runs” and “code is scientifically correct” (Somasekharan et al., 19 Sep 2025).

The evaluated model set includes Claude Sonnet 3.5, o3-mini, Gemini 2.5 Flash, Claude Haiku 3.5, GPT-4o, and Gemma-2-9B-IT. Prompting was zero-shot with a standardized template, and temperature was set to 0.0 whenever configurable; o3-mini used its provider-controlled default temperature because it could not be changed. No special agent framework, retrieval augmentation, reviewer loop, or few-shot prompting is reported for CFDCodeBench (Somasekharan et al., 19 Sep 2025).

The reported qualitative examples show why the benchmark is hard. For the 1D Burgers equation, o3-mini, Haiku 3.5, and Gemini 2.5 Flash can closely match the ground-truth solution, whereas Sonnet 3.5 matches the solution near the shock but gets the boundary conditions wrong. For the 2D convection equation, Sonnet 3.5 appears to suffer numerical instability, whereas other models produce a “decent prediction” of the $M_{\mathrm{NMSE}}$0-velocity field. The discussion identifies recurrent failure modes: incorrect boundary conditions, numerical instability, inadequate understanding of flux discretization schemes, poor choice of time integration strategies, inconsistent boundary treatments, and failure to correctly apply fundamental CFD concepts (Somasekharan et al., 19 Sep 2025).

A persistent misconception in interpreting these results is to treat executability as a sufficient metric. CFDCodeBench was explicitly designed to show that executability alone substantially overstates practical usefulness. The benchmark’s performance profile suggests that current models often generate syntactically valid scripts, but much less often generate solvers that are both physically accurate and convergent under refinement. A plausible implication is that the main bottleneck is not generic code generation, but domain-specific reasoning about discretization, stability, and consistency conditions (Somasekharan et al., 19 Sep 2025).

6. Reproducibility, limitations, and relation to adjacent benchmarks

CFDCodeBench is publicly released as part of CFDLLMBench at https://github.com/NREL-Theseus/cfdllmbench/ under the BSD 3-Clause License. The release includes a fully containerized Docker setup and uses only free and open-source software. The paper implies that the benchmark distribution includes prompts or descriptions, reference solutions, evaluation code, and prompt-generation logic, although it does not provide a detailed repository tree or a full dependency specification for the evaluator in the manuscript excerpt (Somasekharan et al., 19 Sep 2025).

The benchmark nevertheless has clear scope limits. It provides 24 problems only, there is no human baseline, extensive automated prompt tuning was not performed, and several methodological details are not fully specified in the paper, including exact per-task reference grids, exact refinement ratios for convergence testing, exact interpolation schemes for shape mismatches, and a full per-model CFDCodeBench score table. Its task space is restricted to 1D and 2D Python PDE solvers rather than multi-file industrial CFD workflows, and that broader workflow role is left to FoamBench within the same suite (Somasekharan et al., 19 Sep 2025).

In the wider landscape of CFD-related benchmarks, CFDCodeBench occupies a distinct niche. FD-Bench is a modular benchmark for data-driven fluid simulation that evaluates spatial, temporal, and loss modules across 10 representative flow scenarios under a unified experimental setup; it addresses neural PDE solvers rather than from-scratch generation of executable CFD scripts (Wang et al., 25 May 2025). CarBench standardizes benchmarking for neural surrogates on high-fidelity 3D car aerodynamics, focusing on geometry-to-surface-pressure prediction rather than code synthesis or script-level numerical implementation (Elrefaie et al., 25 Nov 2025). This suggests that CFDCodeBench is best understood not as a generic CFD benchmark, but as a benchmark for scientific code generation and numerical experiment automation in CFD, situated between conceptual testing and full workflow orchestration.

Its enduring significance lies in that intermediate position. By requiring runnable code, acceptable physical accuracy, and convergence under refinement, CFDCodeBench formalizes a stringent notion of scientific correctness for model-generated CFD scripts. It does not test whether a model can merely talk about CFD or emit plausible-looking code; it tests whether the model can produce a working numerical experiment whose outputs withstand basic scrutiny from computational science (Somasekharan et al., 19 Sep 2025).