CFDQuery: Assessing CFD Knowledge in LLMs

• CFDQuery is a benchmark comprising roughly 100 multiple-choice questions that evaluate graduate-level CFD theory and numerical analysis.
• It isolates conceptual CFD knowledge from code generation by focusing solely on theory, mathematical formulations, and discretization methods.
• The dataset is meticulously curated by domain experts using university lecture notes and standard references to ensure objective, reproducible evaluation.

Searching arXiv for the benchmark paper and closely related CFD automation work to ground the article. I’m retrieving the relevant arXiv records for CFDQuery and adjacent CFD-agent work. CFDQuery is the knowledge-oriented component of CFDLLMBench, a benchmark suite for evaluating LLMs in computational fluid dynamics. Within the suite’s three-part decomposition of capability—graduate-level CFD knowledge, numerical and physical reasoning in CFD, and context-dependent implementation of CFD workflows—CFDQuery targets the first category through a pure multiple-choice question-answer format designed to isolate conceptual and analytical knowledge from coding and tool-use ability. The benchmark is explicitly framed as a graduate-level test of CFD theory and numerical methods rather than a proxy for end-to-end simulation competence (Somasekharan et al., 19 Sep 2025).

1. Role within CFDLLMBench

CFDLLMBench defines three complementary evaluation tiers: CFDQuery, CFDCodeBench, and FoamBench. CFDQuery is the “knowledge tier,” described in the abstract as “90 multiple-choice questions curated from graduate-level CFD lecture notes that assess LLM's ability in the conceptual understanding of CFD knowledge,” and in the main text as a dataset of “108 multiple-choice questions pertaining to CFD curated by three domain experts” (Somasekharan et al., 19 Sep 2025). The paper therefore presents a minor internal inconsistency in count; the core methodology describes CFDQuery as a roughly 100-question benchmark.

Its role is sharply delimited. CFDQuery is a single-turn, stand-alone QA task in which each item requires selection of one correct answer among four options. It is intended to measure whether a model knows core CFD concepts in fluid mechanics, linear algebra, and numerical methods, independently of code synthesis or workflow assembly. This separation is central to the benchmark design: CFDCodeBench evaluates numerical code generation for PDE solution in Python, and FoamBench evaluates configuration of OpenFOAM workflows from natural language (Somasekharan et al., 19 Sep 2025).

The benchmark’s interpretive claim is equally precise. Strong performance on CFDQuery is treated as a necessary but not sufficient condition for practical CFD automation. High scores indicate that a model can recall and reason about graduate-level CFD material, but they do not establish that the model can write reliable solvers, configure simulation files coherently, or produce physically credible numerical results. The reported drop in performance from CFDQuery to CFDCodeBench and FoamBench is used to make exactly that distinction (Somasekharan et al., 19 Sep 2025).

2. Construction, curation, and quality assurance

CFDQuery was created specifically for CFDLLMBench rather than imported from a pre-existing benchmark. The questions were manually written or curated by human experts, with source material adapted from university-level CFD lecture notes, vetted online sources, and standard CFD and numerical-methods references. The dataset was curated by three domain experts, and the paper states that the problems were reviewed and revised iteratively by independent experts to ensure correctness and quality (Somasekharan et al., 19 Sep 2025).

The paper does not report crowd-sourced annotation or inter-annotator agreement in the usual NLP sense. Instead, it states that the problems have objective scientific answers and therefore were subjected to an iterative expert review workflow rather than traditional human-agreement analysis. For CFDQuery, this means that the question statements and answer keys were independently checked, corrected where necessary, and also solved by experts to confirm correctness (Somasekharan et al., 19 Sep 2025).

This construction protocol is consequential for how the benchmark should be read. CFDQuery is not a noisy educational QA corpus assembled from heterogeneous public sources; it is an expert-authored examination set whose intended target is graduate-level competence in CFD numerics. The graduate-level character is reinforced by the reference base—lecture notes and vetted instructional materials—and by the inclusion of topics such as modified-equation analysis, geometric conservation law, WENO, compact schemes, and discontinuous Galerkin methods (Somasekharan et al., 19 Sep 2025).

3. Task structure and topic taxonomy

The benchmark uses a highly constrained evaluation format. Every item is multiple-choice, each question has four options, and exactly one option is correct. During evaluation, the model is instructed to respond with only the option number, “1, 2, 3, or 4,” and no explanation is graded. As a result, CFDQuery measures discrete answer selection rather than free-form derivation quality or chain-of-thought exposition (Somasekharan et al., 19 Sep 2025).

The topical coverage is intentionally broad but numerics-heavy. The appendix summary states that the dataset spans PDEs, numerical methods, and error-analysis topics, including classical finite difference and finite volume schemes for 1D advection, 1D diffusion, and 1D Burgers equation; high-order stencils such as fourth-order central differences, compact schemes, and WENO; non-uniform and curvilinear grids; geometric conservation law; multidimensional flows including Poisson, Navier–Stokes channel and cavity problems, Rayleigh–Bénard convection, and KdV–Burgers; stability criteria; leading-order error terms; and advanced topics in high-order discontinuous Galerkin (Somasekharan et al., 19 Sep 2025).

Several features of this taxonomy matter. First, CFDQuery is not limited to Navier–Stokes phenomenology; it emphasizes discretization analysis, truncation error, dispersion and dissipation, and stability. Second, it presupposes auxiliary mathematical competence, particularly in linear algebra and numerical analysis for PDEs. Third, it treats grid and geometry issues as first-class content, including unequal spacings, non-orthogonal meshes, curvilinear coordinates, and moving-grid consistency. The benchmark therefore operationalizes “CFD knowledge” as an overview of fluid mechanics and computational numerics rather than as a purely physical or purely software-oriented skill set (Somasekharan et al., 19 Sep 2025).

The benchmark does not assign explicit easy, medium, or hard labels. Difficulty is instead controlled implicitly through topic selection and expert judgment. The paper’s characterization of the content suggests that the discriminative burden falls on precise numerical-analysis knowledge rather than on general textbook recall (Somasekharan et al., 19 Sep 2025).

4. Mathematical content and representative item

The paper gives one full example question, which is also used in the appendix analysis. It asks for the modified equation closest to the discretized 1D advection equation using second-order central difference in space and third-order Runge–Kutta in time:

$$\frac{\partial u}{\partial t} + a \frac{\partial u}{\partial x} = 0.$$

Among four alternatives, the correct answer is Option 4:

$$\frac{\partial u}{\partial t} + a \frac{\partial u}{\partial x} = \frac{a \Delta x^2}{6} \frac{\partial^3 u}{\partial x^3} - \frac{a^3 \Delta t^2}{6} \frac{\partial^3 u}{\partial x^3} + \mathcal{O}(\Delta x^3).$$

This item probes understanding of modified-equation analysis, the dispersive leading-order truncation error of central differencing, and the way RK3 temporal truncation error is projected onto spatial derivatives through the governing PDE (Somasekharan et al., 19 Sep 2025).

The example is representative of the benchmark’s mathematical profile. The emphasis is not on symbolic manipulation for its own sake, but on whether a model can identify the correct leading-order structure of a discretization error model and distinguish dispersive from dissipative behavior. The paper explicitly associates CFDQuery with questions on truncation error derivations, modified-equation analysis for central-difference plus RK3, Lax–Friedrichs-type schemes, upwind-biased schemes, convergence order, CFL-type stability criteria, and spectral or matrix-based reasoning grounded in linear algebra (Somasekharan et al., 19 Sep 2025).

The benchmark description also indicates that formulas relevant to canonical PDEs and CFD settings appear throughout the dataset, including the Poisson equation,

$\nabla^2 \phi = f,$

and the incompressible Navier–Stokes system,

$$\nabla \cdot \mathbf{u} = 0, \quad \frac{\partial \mathbf{u}}{\partial t} + \mathbf{u}\cdot\nabla\mathbf{u} = -\nabla p + \nu \nabla^2 \mathbf{u} + \mathbf{f}.$$

These appear in the article as examples of the mathematical and physical content that the questions presuppose, together with non-uniform-grid coefficient derivations and geometric-conservation constraints on moving or deforming meshes (Somasekharan et al., 19 Sep 2025).

A plausible implication is that CFDQuery evaluates not only recognition of named schemes but also the ability to map governing equations, discretizations, and asymptotic error terms into one another. That interpretation is supported by the modified-equation exemplar and by the benchmark’s explicit focus on leading-order error structure, stability, and convergence.

5. Evaluation protocol and benchmarked performance

The evaluation procedure is deliberately minimal. CFDQuery uses a single metric, Success Rate, defined as the ratio of correctly answered questions to the total number of questions. Because each item has one correct option and the model response is restricted to an integer in $\{1,2,3,4\}$, correctness reduces to exact equality with the answer key. There is no partial credit, rubric-based scoring, or human grading in the loop (Somasekharan et al., 19 Sep 2025).

The benchmarked model set includes closed-weight models Claude Sonnet 3.5, Claude Haiku 3.5, GPT-4o, o3-mini, and Gemini 2.5 Flash, together with the open-source Gemma-2-9B-IT. The paper states that temperature was set to 0.0 for all models except o3-mini, whose temperature is undisclosed (Somasekharan et al., 19 Sep 2025).

The headline result is that all closed-weight models perform strongly on CFDQuery, whereas Gemma-2-9B-IT achieves only about 60% accuracy. o3-mini is reported as the best-performing model at about 92% correct answers. The paper summarizes the overall range as 60% to 92%, with the other closed-weight systems in the 80–90% band (Somasekharan et al., 19 Sep 2025).

The appendix also reports model responses on the modified-equation example. The correct answer is Option 4. Claude Sonnet 3.5 and Gemini 2.5 Flash selected Option 4; o3-mini and GPT-4o selected Option 3; Claude Haiku 3.5 and Gemma-2-9B-IT selected Option 1. This example is useful precisely because it complicates the aggregate picture: even models with high overall Success Rate can miss fine-grained numerical-analysis questions requiring coefficient-level precision (Somasekharan et al., 19 Sep 2025).

The paper interprets these results as evidence that graduate-level CFD knowledge is largely within reach of frontier models. At the same time, it juxtaposes those scores with far weaker results on the execution-oriented components of the suite: the best Success Rate on CFDCodeBench is about 14%, while FoamBench Basic and FoamBench Advanced reach 34% and 25%, respectively. The contrast is one of the central empirical findings of CFDLLMBench (Somasekharan et al., 19 Sep 2025).

6. Significance, limitations, and place in CFD automation

CFDQuery’s principal significance lies in what it isolates. By removing code generation, solver execution, and multi-file workflow construction, it provides a reproducible measure of graduate-level conceptual knowledge in CFD. That makes it valuable as a diagnostic instrument for model development, prompt design, and pre-screening of models intended for more demanding CFD tasks (Somasekharan et al., 19 Sep 2025).

Its limitations follow directly from the same design choices. The multiple-choice format yields objective, automatable scoring, but it does not evaluate open-ended derivation quality, robustness under ambiguous problem statements, or tool-mediated reasoning. Topic coverage is broad within graduate CFD numerics but not exhaustive across the field; the paper notes an emphasis on finite difference, finite volume, and discontinuous Galerkin numerics for canonical problems rather than on turbulence-model taxonomies in depth, industrial complex-geometry workflows, uncertainty quantification, optimization, or data assimilation. The dataset is also English-language and rooted in a particular university-level CFD curriculum (Somasekharan et al., 19 Sep 2025).

A recurring misconception addressed by the benchmark is that success on a theory-heavy exam is equivalent to practical CFD competence. CFDQuery was designed partly to show that this equivalence does not hold. The benchmark suite’s cross-tier results demonstrate that models can answer graduate-level conceptual questions at high rates yet still fail on numerical implementation and OpenFOAM configuration tasks. Related work on agentic CFD systems reinforces that distinction from the opposite direction: ChatCFD, for example, targets end-to-end OpenFOAM automation with domain-specific structured thinking, configuration validation, and error reflection, and reports 30–40% accurate configuration success and 60–80% 10-step operational success on research-level cases, underscoring that workflow competence is a separate and substantially harder problem than conceptual QA (Fan et al., 28 May 2025).

Within that broader landscape, CFDQuery functions best as a first-tier benchmark. It answers whether a model “knows CFD” in the sense of graduate lectures and numerical-analysis exercises. It does not answer whether the same model can autonomously construct dependable simulations. The paper’s broader argument is therefore methodological: evaluation of LLMs for scientific computing should not collapse knowledge recall, numerical reasoning, and executional reliability into a single score. CFDQuery is the component of CFDLLMBench that formalizes the first of those dimensions with high specificity (Somasekharan et al., 19 Sep 2025).