EngiBench: Engineering Benchmark Suite
- EngiBench is a set of research artifacts that evaluate engineering reasoning, design optimization, and workflow execution under practical constraints.
- The LLM problem-solving benchmark employs hierarchical levels to assess contextual reasoning, trade-off analysis, and uncertainty handling.
- The design framework standardizes simulator-backed engineering design using unified APIs and metrics for reproducible optimization comparisons.
EngiBench is a name used for multiple, non-identical research artifacts in recent engineering-AI literature. In the 2025–2026 arXiv record, it denotes a hierarchical benchmark for evaluating LLMs on engineering problem solving, an open-source framework and dataset suite for data-driven engineering design and optimization, and a later benchmark suite for LLM-driven engineering workflows implemented within the EngiAI multi-agent system (Zhou et al., 22 Sep 2025, Felten et al., 2 Jun 2025, Molinari et al., 19 May 2026). The shared theme is engineering evaluation under realism—context, constraints, simulation, uncertainty, and tool orchestration—but the object of evaluation differs materially across text-based reasoning, physics-governed inverse design, and agentic workflow execution.
1. Nomenclature and disambiguation
Recent usage of the term is best understood as polysemous rather than singular. The main senses of “EngiBench” in the literature are summarized below (Zhou et al., 22 Sep 2025, Felten et al., 2 Jun 2025, Molinari et al., 19 May 2026).
| Usage of “EngiBench” | Core focus | Representative source |
|---|---|---|
| LLM engineering problem-solving benchmark | Hierarchical evaluation of engineering reasoning with controlled variants | (Zhou et al., 22 Sep 2025) |
| Data-driven engineering design framework | Unified API, simulators, datasets, and baselines for optimization and inverse design | (Felten et al., 2 Jun 2025) |
| LLM workflow benchmark in EngiAI | Workflow, RAG, and HPC evaluation for multi-agent engineering systems | (Molinari et al., 19 May 2026) |
This naming overlap has practical consequences. Citations to “EngiBench” without an arXiv identifier are potentially ambiguous, especially because the 2025 papers attach the same name to substantially different technical objects. A further source of confusion is engineering-diagram benchmarking: the separate benchmark “Enginuity” explicitly states that EngiBench is not mentioned in that paper, and its focus is vision-language understanding of engineering diagrams rather than either engineering reasoning or design optimization (Kumar et al., 2 Jun 2026).
2. EngiBench as a benchmark for LLM engineering problem solving
In “EngiBench: A Benchmark for Evaluating LLMs on Engineering Problem Solving,” EngiBench is a hierarchical benchmark designed to evaluate LLMs on engineering tasks that go beyond clean symbolic mathematics (Zhou et al., 22 Sep 2025). Its motivation is explicitly contrasted with GSM8K, MATH, and Omni-MATH: those datasets emphasize symbolic computation under fully specified conditions and often rely on public questions susceptible to data contamination, whereas practical engineering requires interpreting context, handling uncertainty, and balancing trade-offs in open-ended scenarios.
The benchmark is organized into three levels of increasing difficulty. Level 1, “foundational knowledge retrieval,” consists of well-structured, single-step tasks solvable by direct recall and formula application. Level 2, “multi-step contextual reasoning,” contains well-specified problems with unique answers that require interpreting structured descriptions, respecting units and constraints, and linking multiple steps of domain knowledge. Level 3, “open-ended modeling,” is designed to mirror real-world engineering challenges: problems are under-specified, embed implicit constraints and ambiguous inputs, and often involve conflicting objectives. Level 3 evaluates four capability dimensions: information extraction, domain-specific reasoning, multi-objective decision-making, and uncertainty handling.
The benchmark spans three engineering subfields: Systems & Control with 916 problems, Physical & Structural with 334 problems, and Chemical & Biological with 467 problems. Levels 1–2 draw from public benchmarks and university materials, whereas Level 3 introduces 43 open-ended problems curated from mathematical modeling competitions. These Level 3 tasks are accompanied by official scoring criteria and high-quality reference solutions. Annotation for Level 3 was performed by 20 PhD students and engineering professionals, with GPT-4.1 and Gemini 2.5 Flash used as auxiliary tools, and the paper reports high inter-annotator agreement without giving a specific coefficient.
A distinguishing feature is the controlled rewriting of problems. For Levels 1–2, each item is provided in original, perturbed, knowledge-enhanced, and math abstraction forms. The perturbed variant changes numbers and/or wording while preserving logic, the knowledge-enhanced variant appends relevant domain facts before the unchanged question, and the math abstraction variant strips context and reformulates the task as a compact symbolic problem. For Level 3, only original and perturbed versions are used, because knowledge-enhanced and abstraction variants are described as impractical for under-specified modeling scenarios.
The evaluation protocol separates low-level correctness from high-level rubric quality. Levels 1–2 use binary correctness and accuracy, with an automated comparison prompt that tolerates up to numerical deviation for complex calculations and also checks unit requirements for unit-specific tasks. Level 3 uses rubric-based scoring on a $0$–$10$ scale for each of the four capability dimensions, with criteria derived from official contest rubrics and reviewed by domain experts. The paper does not provide a closed-form aggregation formula; instead, scores are averaged across tasks and dimensions. Representative Level 3 rubric examples include explicit engineering formulations such as
together with distributions such as , , and .
Baseline experiments cover 16 zero-shot LLMs, including GPT-4.1, Claude 3.7 Sonnet, Gemini 2.5 Flash, GLM-4, Qwen2.5, Llama 4, DeepSeek-v3, DeepSeek-R1-Distill-Qwen-1.5B, and Mixtral-8x7B. The reported results show a clear stratification by difficulty: models cluster high on Level 1, drop at Level 2, and struggle on Level 3. Across all models, Level 1 average accuracy is 82.9% on original problems, 81.5% on perturbed, 85.5% on knowledge-enhanced, and 89.4% on math abstraction; Level 2 averages are 66.6%, 61.6%, 68.6%, and 79.2%, respectively. The paper reports a human expert Level 3 average of 8.58 overall, and more precisely 8.552 on original tasks and 8.736 on rewritten tasks. Top models are reported as “above 6” on Level 3, while weaker open-source models often average below 4.
The error profile is diagnostic rather than merely comparative. Perturbation sensitivity exposes robustness failures; knowledge-enhancement gains indicate that models do not consistently retrieve or apply domain-specific knowledge; and strong improvements under math abstraction indicate weak upstream contextual reasoning, especially in information extraction and multi-step organization. On Level 3, the recurring failure modes are omission of trade-off analysis, weak uncertainty handling, and mishandling of implicit constraints. The paper’s Llama 4 case study assigns a score of 0 in multi-objective decision-making because trade-off analysis is missing, whereas GPT-4.1 receives 7.5 for structured evaluation.
3. EngiBench as a framework for data-driven engineering design research
In “EngiBench: A Framework for Data-Driven Engineering Design Research,” EngiBench is not a language-model benchmark but an open-source framework, benchmark suite, and curated dataset collection for engineering design optimization (Felten et al., 2 Jun 2025). Its central aim is to standardize how physics-governed design problems are described, simulated, and evaluated, thereby enabling fair and reproducible comparison of optimization and machine learning algorithms. A companion library, EngiOpt, supplies baseline implementations of generative inverse-design and surrogate-assisted optimization methods that plug directly into the EngiBench interface.
The framework addresses several barriers typical of engineering design research. Physics-based simulators such as CFD for airfoils, FEA for structures, FDFD for photonics, and circuit simulators for power electronics are difficult to install and parameterize, computationally expensive, version-sensitive, and domain-specific. Datasets are often private or narrowly problem-specific, and generating new data may require significant domain expertise and thousands of CPU-hours. EngiBench responds by abstracting simulators behind a unified Python API, providing modular problem definitions, built-in constraint checking, dataset generation utilities, and standardized performance analysis.
Its formalism standardizes engineering design as constrained optimization. With design variables , attributes , conditions , objectives $0$0, simulator assumptions $0$1, and inequality/equality constraints $0$2, the framework optimizes an approximate objective $0$3 subject to $0$4 and $0$5. This abstraction is instantiated across heterogeneous physical domains. Airfoil uses RANS-based aerodynamics and optimizes drag $0$6 under prescribed lift $0$7; HeatConduction2D/3D uses the steady-state heat equation with mixed Dirichlet–Neumann boundary conditions and minimizes thermal compliance; Beams2D implements SIMP-based topology optimization minimizing structural compliance; ThermoElasticBeams2D couples thermal conduction and elasticity; Photonics2D uses FDFD for a wavelength demultiplexer with SIMP and Heaviside projection; and PowerElectronics uses NgSpice for a fixed-topology DC-DC converter with two objectives, gain error and voltage ripple.
The benchmarked problems differ substantially in representation and compute profile. Airfoil uses 192 airfoil coordinates plus angle of attack and has a representative simulation time of about 9.85 s. HeatConduction2D and HeatConduction3D use density images or tensors and take about 10.22 s and 31.57 s, respectively. ThermoElasticBeams2D is about 0.22 s, Beams2D about 0.19 s, Photonics2D about 0.3 s, and PowerElectronics about 0.66 s. Dataset organization is similarly domain-specific: Airfoil has 1400 LHS parameter combinations; HeatConduction2D/3D samples a $0$8 grid of condition pairs; ThermoElasticBeams2D includes 1000 optimized designs; PowerElectronics contains 13,824 samples split 7/2/1 across train/validation/test.
A major contribution is standardized evaluation across domains. Similarity is quantified with Maximum Mean Discrepancy (MMD), diversity with Determinantal Point Process (DPP) diversity, optimality with Cumulative Optimality Gap (COG), and constraint adherence with Ratio of Violated Constraints (RVC) and Ratio of Failed simulations (RF). The framework also supports adjoint optimization for polishing candidate designs and for computing optimality-related metrics. Problems are versioned, datasets are hosted on Hugging Face, and the API exposes design spaces, objectives, conditions, simulation, optimization, rendering, and constraint checking in a uniform way. The repositories are associated with IDEALLab, with documentation at engibench.ethz.ch, code under GPL-3.0, and datasets under CC-BY-NC-SA.
The reported experiments illustrate why the benchmark is nontrivial. In a cross-domain inverse-design study on Beams2D, HeatConduction2D, and Photonics2D, unconditional GANs often achieved competitive COG and MMD despite blurrier designs and higher DPP on Beams2D and HeatConduction2D; CGAN generally performed best on constraint satisfaction; and diffusion models excelled in diversity on Photonics2D. In Airfoil, simulation failure ratios were 0.304 ± 0.167 for GAN, 0.034 ± 0.038 for Diffusion, and 0.014 ± 0.027 for BézierGAN, with the paper attributing the benefit of BézierGAN to improved geometric smoothness and fewer meshing failures. In PowerElectronics, surrogate-based Pareto fronts diverged from simulator fronts, with two-sample $0$9 ranging from $10$0 to $10$1 across ten runs and all permutation-test $10$2-values at 0.001, highlighting stiff dynamics, discontinuities, heavy-tailed ripple, numerical artifacts, and near-failure regimes.
4. EngiBench in LLM-driven engineering workflows
In “EngiAI: A Multi-Agent Framework and Benchmark Suite for LLM-Driven Engineering Design,” EngiBench becomes a structured benchmark suite for evaluating multi-agent engineering workflows rather than static design outputs or answer strings (Molinari et al., 19 May 2026). This benchmark is built on LangGraph and operationalizes three evaluation dimensions: workflow execution under distinct cognitive demands, retrieval-augmented parameter selection with gated scoring, and end-to-end HPC orchestration on SLURM clusters. Its execution layer uses EngiBench/EngiOpt APIs, so this usage is technically downstream of the design framework rather than independent of it.
The workflow benchmark uses two engineering problems, Beams2D and Photonics2D, and seven prompt styles: Full, Natural, W-Rand, W-Derived, W-Distract, W-Cond, and W-Multi. These styles target direct tool use, ambiguity detection, instruction adherence, arithmetic derivation, semantic disambiguation, conditional reasoning, and working memory. The underlying tool chain includes optimization, simulation, rendering, and STL export, with parameter validation for STL styles using a $10$3 absolute tolerance for floats and exact matching for booleans.
Scoring is composite. The workflow score combines design quality, tool-use efficiency, and task completion with weights 65%, 20%, and 15%, respectively. Design quality itself includes IoU, pixel accuracy, objective score, constraint score, 2D connectivity, and 3D watertightness. The paper notes that watertightness is uniformly 0.00 across styles because threshold-based STL extraction produces non-manifold meshes; this is treated as a pipeline limitation rather than an agent-capability limit.
The RAG benchmark is designed to isolate retrieval contributions. It uses prompts P0–P3 based on the EngiBench paper, an EngiBench API example, the SOPTX paper, and a mixed prompt. Credit for parameter accuracy is granted only if the designated retrieval tool, search_documents, is invoked; RAG-off scores are therefore exactly 0 by construction, and Empty RAG tests tool availability without indexed content. The HPC benchmark, by contrast, evaluates a four-step SLURM pipeline: generate the training command, submit the job, monitor it to completion, and evaluate the trained model against dataset baselines using EngiOpt metrics such as IOG, COG, FOG, MMD, DPP, and violation rate.
The quantitative results show strong task dependence. On Beams2D, proprietary models achieve average task completion of 0.96–0.97, whereas open-source 4B models reach 0.55 and 0.78. The hardest workflow style is W-Cond: task completion is 0.93 for GPT-5-mini, 0.87 for Gemini-3-Flash, 0.40 for Qwen3-4B, and 0.60 for Qwen3.5-4B. On Photonics2D, W-Rand and W-Distract are solved perfectly by all models, but W-Cond again drops sharply to 0.53, 0.47, 0.40, and 0.20. RAG-on scores approach 1.0, RAG-off is exactly 0 because of gating, and Empty RAG degrades substantially. In HPC orchestration, Gemini-3-Flash completes all pipeline steps in 100% of runs for both explicit and natural prompts, whereas GPT-5-mini drops to 70% and 50% at the final evaluation step, which the paper attributes mainly to long-horizon instruction-following decay rather than tool errors.
5. Downstream adoption in later research
Subsequent papers use different senses of EngiBench as external evaluation targets, which is informative because it shows the benchmark name functioning as infrastructure rather than merely as a single publication artifact (Sun et al., 29 Mar 2026, Zimmel et al., 17 Feb 2026).
Sci-Mind uses the Level 3 engineering-modeling benchmark introduced by the LLM-oriented EngiBench paper and evaluates autonomous agents on 43 open-ended engineering mathematical-modeling tasks spanning Systems & Control, Physical & Structural, and Chemical & Biological (Sun et al., 29 Mar 2026, Zhou et al., 22 Sep 2025). The paper reports two metrics: Rubric Score (RS), expressed as a percentage under the official EngiBench rubric, and Code Executability (CE), the percentage of generated solutions that run to completion in a sandbox without runtime errors. Sci-Mind’s average RS is 62.8 with a GPT-4o backbone and 67.9 with a DeepSeek-R1 backbone, compared with 50.7 and 56.1 for MM-Agent; the paper describes an average CE improvement over MM-Agent of 17.5 points across configurations. It also reports that Sci-Mind with GPT-4o surpasses MM-Agent with DeepSeek-R1 on both RS and CE across all subfields. The benchmark is therefore used not only to assess text quality but also to assess executability.
The SATTS paper uses the design-oriented EngiBench framework as a testbed for test-time adaptation of high-dimensional simulation surrogates, specifically on Beams2D and HeatConduction2D (Zimmel et al., 17 Feb 2026, Felten et al., 2 Jun 2025). Here the benchmark is instantiated as conditional generative design under distribution shift. The paper creates out-of-distribution splits by parameter range: for Beams2D, source $10$4 and target $10$5 with 3,087 and 353 samples; for HeatConduction2D, source volume $10$6 and target volume $10$7 with 231 and 39 samples. The reported metrics are mean absolute error, MMD, and Compliance, computed with FEM and reported only on feasible designs. SATTS improves Beams2D Compliance from 123.7 ± 17.854 for the unadapted source model to 118.8 ± 12.409, and HeatConduction2D Compliance from 0.577 ± 0.561 to 0.537 ± 0.491, which the paper interprets as roughly 4% and 7% relative gains. This use of EngiBench is conceptually distinct from both language-model reasoning and multi-agent workflow evaluation: it treats EngiBench as a controlled source of simulation-grounded inverse-design tasks.
These downstream uses suggest that “EngiBench” functions as a family name for benchmarked engineering AI problems rather than as a single evaluation ontology. The implication is methodological as well as terminological: identical naming does not imply identical task semantics, score meanings, or failure modes.
6. Limitations, misconceptions, and future directions
The limitations of EngiBench depend on which artifact is meant. The LLM engineering problem-solving benchmark is explicitly text-only, excludes very long-context tasks, and requires substantial human curation; planned future work includes broader domain coverage, multimodal evaluation, and long-context reasoning (Zhou et al., 22 Sep 2025). The data-driven design framework currently focuses on static simulations rather than dynamic or morphing systems and assembly-level optimization; unstructured meshes, grammar-based representations, and more flexible configuration spaces are identified as active areas for extension, and the paper notes that simulator biases and fidelity gaps can propagate into learned models (Felten et al., 2 Jun 2025). The EngiAI workflow benchmark is limited to Beams2D and Photonics2D, only four LLM backends, and descriptive analysis without statistical tests; it also lacks a single-agent baseline against the supervisor-based multi-agent system (Molinari et al., 19 May 2026).
A recurring misconception is that all references to EngiBench concern the same resource. They do not. The 2025 LLM benchmark measures hierarchical engineering reasoning with controlled textual variants; the 2025 design framework standardizes simulator-backed engineering design optimization; and the 2026 EngiAI benchmark evaluates tool-intensive multi-agent workflows on top of EngiBench/EngiOpt problems. A second misconception is conflation with engineering-diagram benchmarks. Enginuity is a separate benchmark for vision-language understanding of engineering diagrams and explicitly notes that EngiBench is not part of that work (Kumar et al., 2 Jun 2026).
Across these lines of work, the name’s persistence reflects a common research priority: evaluation under engineering-specific constraints rather than under purely symbolic or generic ML settings. What changes across usages is the locus of difficulty—contextual reasoning and uncertainty in (Zhou et al., 22 Sep 2025), simulator-coupled inverse design and feasibility in (Felten et al., 2 Jun 2025), or long-horizon tool use, retrieval isolation, and SLURM orchestration in (Molinari et al., 19 May 2026). This suggests that precise disambiguation by arXiv identifier is not a bibliographic nicety but a technical necessity.