---
title: 'CFDCodeBench: CFD Code Generation Benchmark'
url: https://www.emergentmind.com/topics/cfdcodebench
type: topic
---

# CFDCodeBench: CFD Code Generation Benchmark

CFDCodeBench is the code-generation component of CFDLLMBench, a benchmark suite for evaluating large language models on computational fluid dynamics tasks that extend beyond factual recall. It is designed to test whether a model can transform a structured physical and numerical problem description into complete runnable Python code for a CFD-style numerical experiment, with required outputs saved in NumPy format and judged not only by executability but also by physical accuracy and numerical convergence. Within the three-part structure of CFDLLMBench, CFDCodeBench occupies the middle tier between CFDQuery, which tests graduate-level conceptual knowledge, and FoamBench, which tests full OpenFOAM workflow construction and execution [2509.20374].

## 1. Position and benchmark objective

CFDCodeBench targets what CFDLLMBench defines as **numerical and physical reasoning**, expressed through executable scientific programming. The benchmark asks whether a model can read a natural-language description of a CFD problem, understand the governing PDE, domain, initial and boundary conditions, and requested outputs, choose or apply an appropriate numerical method, generate complete runnable Python code, save the required variables in `.npy` format, and produce a solution that is physically accurate and numerically convergent. In this framing, the benchmark is not a generic code-synthesis task; it is a test of context-dependent implementation of CFD workflows at the single-script level [2509.20374].

A common misconception is that CFDCodeBench measures general software engineering ability in the style of code completion or bug fixing. It does not. The tasks are framed as **from-scratch writing** of CFD code from structured problem descriptions. The benchmark does **not** describe code editing, debugging given code, completion of partial templates, or hidden test-driven implementation from function signatures. A second misconception is that executable code is sufficient. CFDCodeBench explicitly rejects that criterion: runnable scripts that violate physics or fail refinement-based convergence do not count as successful solutions [2509.20374].

## 2. Task design, scope, and coverage

CFDCodeBench contains **24 CFD programming problems** spanning **1D and 2D** settings, **linear and nonlinear PDEs**, and topics in **fluid mechanics, thermal transport, and turbulence**, with scripts averaging about **70 lines of code per problem**. No separate train/validation/test split is described; it is treated as a benchmark set of 24 tasks [2509.20374].

The task set covers canonical transport equations, elliptic and heat-equation problems, incompressible Navier–Stokes configurations, compressible flow, buoyancy-driven flow, turbulence modeling, and linear stability analysis. The benchmark therefore mixes introductory finite-difference-style problems with more advanced tasks derived from spectral-method workflows and turbulence models.

| Category | Representative problems |
|---|---|
| Canonical transport and model PDEs | 1D linear convection, 2D Burgers, 2D inviscid Burgers, 1D Korteweg–de Vries/Burgers |
| Elliptic and heat problems | 2D Laplace equation, 2D Poisson equation, 2D steady heat equation, Lane–Emden equation |
| CFD configurations | 2D Navier–Stokes in a cavity, channel flow with Navier–Stokes, periodic incompressible shear flow with a passive tracer field |
| Advanced tasks | Euler shock tube, 2D horizontally periodic Rayleigh–Bénard convection, fully-developed turbulent channel flow with a Cess turbulence model, linear stability eigenvalue problem for pipe flow |

The coverage is broad in boundary conditions and geometry. The benchmark includes **Dirichlet**, **Neumann**, **mixed Dirichlet/Neumann**, **periodic**, **reflective**, **constant boundary values**, and **no-slip** conditions, and it is mostly defined on simple 1D intervals and 2D rectangular or square domains, with at least one non-Cartesian case: **flow past circular cylinder in streamfunction-vorticity formulation in polar coordinates**. At the same time, the scope is explicitly limited to **1D and 2D** tasks; no **3D** CFDCodeBench tasks are reported [2509.20374].

## 3. Data curation, prompt specification, and required outputs

The benchmark was curated from **publicly available GitHub repositories and established numerical solver packages**. Its three stated sources are **CFD Python: the 12 Steps to Navier-Stokes Equations**, **ENGR 491 - Computational Fluid Dynamics**, and the **Dedalus Project**. The dataset contains **17 foundational problems** from CFD Python and ENGR 491 and **7 advanced problems** from Dedalus. Because the Dedalus-derived tasks were more difficult and lacked detailed tutorials, CFD experts reviewed the source code and authored accompanying problem descriptions. Across CFDLLMBench, **six CFD domain experts** participated: two doctoral students, one master’s student, one undergraduate student, one postdoc, and one professor; the postdoc and professor reviewed the work of the other contributors at each step [2509.20374].

CFDCodeBench uses a **structured JSON-to-natural-language prompt generation pipeline**. Each problem is represented initially as a JSON object with fields such as `equation`, `boundary conditions`, `initial conditions`, `domain`, `save values`, and optionally `numerical method`. This JSON is transformed into a standardized natural-language prompt. The design goal is to preserve problem structure and reproducibility while reducing prompt ambiguity [2509.20374].

The required model output is narrowly specified. The generated response must be **complete runnable Python code** and nothing else. If the problem is unsteady, only the **final time step** is to be saved. Each requested variable must be written as a separate NumPy `.npy` file using the same variable names as in `save_values`. For **1D problems**, each saved variable must be a **1D array**; for **2D problems**, each saved variable must be a **2D array**. The benchmark therefore evaluates not just PDE discretization and time integration, but also adherence to a fixed execution-and-output contract [2509.20374].

## 4. Evaluation protocol and metric structure

CFDCodeBench uses four benchmark metrics: three component metrics and one aggregate success statistic. The core design principle is that script validity, physical fidelity, and refinement behavior are distinct properties and must all be measured [2509.20374].

| Metric | Definition |
|---|---|
| Executability \(M_{\mathrm{exec}}\) | Binary score: 1 if generated Python code executes successfully, 0 otherwise |
| Physical accuracy \(M_{\mathrm{NMSE}}\) | Derived from NMSE% at final time with thresholds 1, 0.5, or 0 |
| Numerical convergence \(M_{\mathrm{conv}}\) | Binary score: 1 if error decreases under spatial and temporal refinement |
| Success Rate | Fraction of problems with \(M_{\mathrm{exec}}=1\), \(M_{\mathrm{NMSE}}=1\), and \(M_{\mathrm{conv}}=1\) simultaneously |

Physical correctness is measured by a **normalized mean squared error percentage** at the final time:
\[
\mathrm{NMSE}\% = \frac{\sum_{i=1}^{N}(y_i-\hat{y}_i)^2}{\sum_{i=1}^{N}y_i^2}\times 100,
\]
with score mapping
\[
M_{\mathrm{NMSE}}=
\begin{cases}
1, & \mathrm{NMSE}\le 10\%,\\
0.5, & 10\%<\mathrm{NMSE}\le 30\%,\\
0, & \mathrm{NMSE}>30\%.
\end{cases}
\]
If predicted and reference arrays have different shapes, the benchmark interpolates the predicted solution to align dimensions before comparison. Numerical convergence is assessed by refining both mesh spacing and time step; if the error decreases with refinement, the solution is awarded a score of 1, and 0 otherwise [2509.20374].

The aggregate success criterion is intentionally strict:
\[
M_{\mathrm{success}}^{(i)}=
\begin{cases}
1, & M_{\mathrm{exec}}^{(i)}=1 \;\wedge\; M_{\mathrm{NMSE}}^{(i)}=1 \;\wedge\; M_{\mathrm{conv}}^{(i)}=1,\\
0, & \text{otherwise},
\end{cases}
\]
and the benchmark-level score is
\[
\mathrm{Success\ Rate}=\frac{1}{K}\sum_{i=1}^{K} M_{\mathrm{success}}^{(i)}.
\]
This formulation means that executable code with partial accuracy or non-convergent behavior does not count as a success. The benchmark therefore differs fundamentally from pass@1-style coding evaluations, because it treats numerical analysis properties as first-class evaluation targets [2509.20374].

Execution is performed in a **sandboxed environment** with a default timeout of **60 seconds**. The benchmark does **not** rely on hidden tests, symbolic equivalence, or unit-test suites in the HumanEval sense. Its orchestration pipeline is execution-based: task JSON is converted to a prompt, the model response is parsed as Python code, the code is executed, `.npy` outputs are collected, NMSE is computed against a trusted reference, convergence under refinement is tested, and the component metrics and Success Rate are then computed [2509.20374].

## 5. Empirical results and observed failure modes

CFDCodeBench proved difficult for all evaluated models. The headline result is that the **best-performing model achieved only about 14% Success Rate**. At the same time, most **closed-weight models** produced executable Python code in **over 60% of cases**, **o3-mini** executed successfully in about **75%** of cases, and **Claude Sonnet 3.5** was the weakest closed model in executability at about **42%**. The benchmark’s central empirical message is therefore the gap between “code runs” and “code is scientifically correct” [2509.20374].

The evaluated model set includes **Claude Sonnet 3.5**, **o3-mini**, **Gemini 2.5 Flash**, **Claude Haiku 3.5**, **GPT-4o**, and **Gemma-2-9B-IT**. Prompting was **zero-shot** with a standardized template, and temperature was set to **0.0** whenever configurable; **o3-mini** used its provider-controlled default temperature because it could not be changed. No special agent framework, retrieval augmentation, reviewer loop, or few-shot prompting is reported for CFDCodeBench [2509.20374].

The reported qualitative examples show why the benchmark is hard. For the **1D Burgers equation**, **o3-mini**, **Haiku 3.5**, and **Gemini 2.5 Flash** can closely match the ground-truth solution, whereas **Sonnet 3.5** matches the solution near the shock but gets the **boundary conditions wrong**. For the **2D convection equation**, **Sonnet 3.5** appears to suffer **numerical instability**, whereas other models produce a “decent prediction” of the \(x\)-velocity field. The discussion identifies recurrent failure modes: **incorrect boundary conditions**, **numerical instability**, inadequate understanding of **flux discretization schemes**, poor choice of **time integration strategies**, inconsistent **boundary treatments**, and failure to correctly apply **fundamental CFD concepts** [2509.20374].

A persistent misconception in interpreting these results is to treat executability as a sufficient metric. CFDCodeBench was explicitly designed to show that executability alone substantially overstates practical usefulness. The benchmark’s performance profile suggests that current models often generate syntactically valid scripts, but much less often generate solvers that are both physically accurate and convergent under refinement. A plausible implication is that the main bottleneck is not generic code generation, but domain-specific reasoning about discretization, stability, and consistency conditions [2509.20374].

## 6. Reproducibility, limitations, and relation to adjacent benchmarks

CFDCodeBench is publicly released as part of CFDLLMBench at **https://github.com/NREL-Theseus/cfdllmbench/** under the **BSD 3-Clause License**. The release includes a **fully containerized Docker setup** and uses only **free and open-source software**. The paper implies that the benchmark distribution includes prompts or descriptions, reference solutions, evaluation code, and prompt-generation logic, although it does not provide a detailed repository tree or a full dependency specification for the evaluator in the manuscript excerpt [2509.20374].

The benchmark nevertheless has clear scope limits. It provides **24 problems** only, there is **no human baseline**, extensive automated prompt tuning was **not** performed, and several methodological details are not fully specified in the paper, including exact per-task reference grids, exact refinement ratios for convergence testing, exact interpolation schemes for shape mismatches, and a full per-model CFDCodeBench score table. Its task space is restricted to **1D and 2D Python PDE solvers** rather than multi-file industrial CFD workflows, and that broader workflow role is left to **FoamBench** within the same suite [2509.20374].

In the wider landscape of CFD-related benchmarks, CFDCodeBench occupies a distinct niche. **FD-Bench** is a modular benchmark for **data-driven fluid simulation** that evaluates spatial, temporal, and loss modules across **10 representative flow scenarios** under a unified experimental setup; it addresses neural PDE solvers rather than from-scratch generation of executable CFD scripts [2505.20349]. **CarBench** standardizes benchmarking for **neural surrogates** on **high-fidelity 3D car aerodynamics**, focusing on geometry-to-surface-pressure prediction rather than code synthesis or script-level numerical implementation [2512.07847]. This suggests that CFDCodeBench is best understood not as a generic CFD benchmark, but as a benchmark for **scientific code generation and numerical experiment automation** in CFD, situated between conceptual testing and full workflow orchestration.

Its enduring significance lies in that intermediate position. By requiring runnable code, acceptable physical accuracy, and convergence under refinement, CFDCodeBench formalizes a stringent notion of scientific correctness for model-generated CFD scripts. It does not test whether a model can merely talk about CFD or emit plausible-looking code; it tests whether the model can produce a working numerical experiment whose outputs withstand basic scrutiny from computational science [2509.20374].

Source: https://www.emergentmind.com/topics/cfdcodebench