MD-EvalBench: MD Q&A & Code Benchmark
- MD-EvalBench is an integrated benchmark suite that assesses LLMs on molecular dynamics by evaluating both domain knowledge and executable code generation.
- It comprises three expert-curated sub-benchmarks—MD-KnowledgeEval, LAMMPS-SyntaxEval, and LAMMPS-CodeGenEval—that cover theory, command syntax, and script synthesis.
- The benchmark uses precise accuracy and execution metrics to distinguish between scientific understanding, syntax correctness, and the creation of runnable LAMMPS input scripts.
MD-EvalBench is the first integrated benchmark suite for evaluating LLMs on molecular-dynamics–specific question answering and LAMMPS script generation tasks, introduced in “MDAgent2: LLM for Code Generation and Knowledge Q&A in Molecular Dynamics” (Shi et al., 5 Jan 2026). It provides a standardized, end-to-end evaluation framework for both domain understanding and executable text-to-code generation in molecular dynamics by combining theoretical MD knowledge assessment, LAMMPS command and syntax assessment, and runnable LAMMPS input-script synthesis.
1. Scope and benchmark composition
MD-EvalBench consists of three complementary sub-benchmarks that target distinct but related capabilities required for MD-oriented LLMs. The benchmark is explicitly organized around both question answering and code generation, rather than treating script synthesis as an isolated task. This design allows evaluation of whether a model understands MD concepts, knows LAMMPS command structure, and can translate simulation goals into executable scripts (Shi et al., 5 Jan 2026).
| Sub-benchmark | Purpose | Size |
|---|---|---|
| MD-KnowledgeEval | Theoretical knowledge Q&A in MD | 336 expert-curated questions |
| LAMMPS-SyntaxEval | Command-level and syntax Q&A for LAMMPS scripting | 333 questions |
| LAMMPS-CodeGenEval | Text-to-code generation of runnable LAMMPS input scripts | 566 distinct pairs |
MD-KnowledgeEval targets theoretical knowledge Q&A in MD, including “interatomic potentials, integrators, ensembles, etc.” Its question formats are single-choice, multiple-choice, fill-in-the-blank, and short-answer, with 336 expert-curated questions. LAMMPS-SyntaxEval targets command-level and syntax Q&A for LAMMPS scripting, including “command usage, parameter structures, modules,” and uses the same four question types, with 333 questions. LAMMPS-CodeGenEval targets text-to-code generation: each sample is a natural-language simulation objective, and the model must produce a runnable LAMMPS input script. It contains 566 distinct (description, reference script) pairs, and generated scripts are executed in LAMMPS and scored for structural and functional correctness.
Taken together, these components make MD-EvalBench an integrated benchmark rather than a single-task leaderboard. A plausible implication is that the benchmark was designed to separate failures of scientific understanding from failures of syntax knowledge and failures of executable script construction.
2. Dataset construction and curation
All three sub-benchmarks were designed and curated in close collaboration with domain experts in molecular dynamics (Shi et al., 5 Jan 2026). The source materials included textbooks, reference manuals, and peer-reviewed papers on MD and LAMMPS, as well as official LAMMPS documentation and examples. This choice of sources situates the benchmark in the established MD and LAMMPS literature rather than in synthetic prompt collections alone.
For MD-KnowledgeEval and LAMMPS-SyntaxEval, the question generation pipeline followed three explicit steps. First, domain experts drafted an initial question pool covering all major MD concepts and LAMMPS commands. Second, a uniform question template enforced four answer types: single-choice, multiple-choice, fill-in, and short answer. Third, iterative peer review was applied, and each question–answer pair was validated by at least two experts. The benchmark therefore embeds expert curation not only at the drafting stage but also at the validation stage.
For LAMMPS-CodeGenEval, the construction procedure centered on natural-language task templates combining material system, objective, and conditions. Reference LAMMPS scripts were written or validated by simulation specialists. The final set contains 566 (prompt, script) pairs, each covering a distinct task, with examples including “compute thermal conductivity, melt a nanoparticle, tensile test.” The reference scripts supply the benchmark’s ground-truth implementation targets and also define the simulation parameters used during evaluation.
The final reported statistics are 336 QA pairs for MD-KnowledgeEval, 333 QA pairs for LAMMPS-SyntaxEval, and 566 code-generation prompts with ground-truth scripts for LAMMPS-CodeGenEval. The benchmark’s curation pipeline suggests a deliberate emphasis on expert validity and reproducibility over fully automatic benchmark construction.
3. Metrics and scoring logic
MD-EvalBench uses two classes of metrics: one for QA and one for code generation (Shi et al., 5 Jan 2026). For MD-KnowledgeEval and LAMMPS-SyntaxEval, each question is scored 0 or 1, and overall accuracy is defined as
This is a standard exact-match scoring regime for mixed-format QA, with no partial credit described in the benchmark overview.
For LAMMPS-CodeGenEval, the first metric is Exec-Success@k, defined as “The proportion of tasks for which at least one of the generated candidates can be successfully executed in LAMMPS.” Its formula is
where is the total tasks, is the -th generated candidate for task , and is the indicator function.
The second code-generation metric is Code Human Score, defined as “A subjective, expert–assigned score evaluating readability, robustness, and physical correctness of the generated script.” The final reported value is the average across all examples and three independent runs.
A recurring misconception in code-generation evaluation is that parse success alone is sufficient. MD-EvalBench explicitly distinguishes syntax correctness, runtime stability, and physical sanity. In the benchmark protocol, syntax correctness means that the script launches without parse errors; runtime stability means there are no “lost atoms” or NaN energies; physical sanity means thermodynamic observables follow the requested behavior, such as temperature ramps up as specified. This scoring design indicates that scientific executability is treated as more than syntactic validity.
4. Benchmark protocol and execution environment
The benchmark protocol specifies repeated evaluation, a controlled runtime environment, and explicit correctness criteria (Shi et al., 5 Jan 2026). Each model is run three times with different random seeds, and results are averaged. The authors further recommend repeating each evaluation at least three times with different random seeds and reporting mean ± standard deviation.
The execution environment uses LAMMPS version 2022 (or later) installed in a Docker-sandboxed container, with CPU-only execution. Standard “metal” units are used, and periodic boundaries are assumed unless otherwise specified. Any script that does not terminate within a predefined wall-clock limit is marked as failed. A Linux host with Docker, LAMMPS dependencies, and at least 4 CPU cores per parallel execution is required; GPU is not required.
Simulation parameters are not freely chosen at evaluation time. Instead, each prompt’s reference script defines the precise parameters, including ensemble, timestep, run length, and thermostat/damping. This ties evaluation to reference-task semantics rather than to a loosely specified prompt interpretation. The benchmark thereby constrains variation that would otherwise arise from evaluator-dependent simulation settings.
The correctness criteria are threefold. Syntax correctness requires that a script launches without parse errors. Runtime stability requires that no “lost atoms” or NaN energies occur. Physical sanity requires that thermodynamic observables follow the requested behavior. This layered protocol is especially important for LAMMPS input scripts, where a script may be syntactically valid but scientifically unusable.
5. Baseline results and reported performance gaps
The benchmark includes baseline results for both QA and code generation, showing measurable gains from domain-specific post-training and from runtime feedback loops (Shi et al., 5 Jan 2026). In QA, the reported overall accuracies are: Qwen3-8B at 70.50%, MD-Instruct-8B at 74.67%, Qwen3-14B at 72.91%, Qwen3-32B at 77.34%, and Qwen3-max at 82.49%. The reported deltas versus Qwen3-8B are 0.00, +4.17, +2.41, +6.84, and +11.99, respectively.
The accompanying commentary states that MD-Instruct-8B, though smaller, outperforms the 14B and closed-source flash variants in QA, confirming the value of domain-specific post-training. The largest closed-source model, Qwen3-max, still leads, but at a cost-prohibitive for local deployment.
For code generation, the reported approximate values from Figure 1 are: Qwen3-8B (Direct Prompt) with Exec-Succ@3 of 5.1% and Code Human Score of 7.85; MD-Code-8B (Direct Prompt) with 14.23% and 9.29; MD-Code-8B + MDAgent2-RT with 37.95% and 9.32; and MDAgent (previous version) with approximately 26.0% and 8.8 on the backbone. The paper describes MDAgent2-RUNTIME as a deployable multi-agent system that integrates code generation, execution, evaluation, and self-correction (Shi et al., 5 Jan 2026).
The reported commentary emphasizes two gaps. First, domain-adapted MD-Code-8B roughly triples the execution success of the un-tuned 8B model. Second, the full MDAgent2-RUNTIME loop delivers a ≈2.7× boost in Exec-Succ@3 versus Direct Prompting, underlining the power of automated execution feedback and self-correction. A plausible implication is that MD-EvalBench was intended not only to rank models but also to expose the contribution of deployment-time execution loops.
6. Recommended usage and relation to similarly named benchmarks
The authors recommend that MD-EvalBench be used as a combined evaluation suite rather than as a single headline metric (Shi et al., 5 Jan 2026). They advise always reporting Exec-Success@k alongside a human-expert script-quality score to capture both executability and scientific fidelity. They also recommend using all three sub-benchmarks in combination: QA benchmarks to validate domain understanding, and CodeGenEval to stress-test generation and execution.
For comparability and reproducibility, the recommended practice is to run all scripts in an isolated, reproducible environment such as Docker with the same LAMMPS version and unit conventions, and to release code and data—prompt–script pairs and question files—in the Alpaca or JSON schema to facilitate direct comparisons and fine-tuning by the community. For more granular analysis, the authors recommend including per-question-type breakdown for QA and per-task-type Exec-Succ@k for code generation.
A source of confusion is the benchmark’s name. “MdEval” is a different benchmark for massively multilingual code debugging, covering 18 programming languages, 3,513 unique problems, and three debugging tasks: Automated Program Repair, Code Review, and Bug Identification (Liu et al., 2024). “MDBench” is a different open-source benchmarking framework for data-driven model discovery on dynamical systems, evaluating 12 algorithms on 63 ODE systems and 14 PDE systems under controlled noise (Bideh et al., 24 Sep 2025). In contrast, MD-EvalBench in MDAgent2 denotes the benchmark for molecular-dynamics–specific question answering and LAMMPS script generation (Shi et al., 5 Jan 2026).
This naming overlap suggests that careful citation is necessary when discussing “MD-EvalBench.” Within the broader benchmark landscape, DOMAINEVAL evaluates multi-domain code generation across six domains using a fully automated repository-to-benchmark pipeline (Zhu et al., 2024), whereas MD-EvalBench is specialized to molecular dynamics and executable LAMMPS scripting. That specialization is central to its role as a benchmark for AI-assisted industrial simulation workflows.