Your Simulation Runs but Solves the Wrong Physics: PDE-Grounded Intent Verification for LLM-Generated Multiphysics Simulation Code

Abstract: Execution-based evaluation of LLM-generated code implicitly treats successful execution as a proxy for correctness. In scientific simulation, this proxy is insufficient: a generated input file can run, mesh, and converge while encoding governing equations that differ from the user's intent. We call this mismatch between intended physics and generated code the comprehension-generation gap. We instantiate this in MOOSE, where Kernel and BC objects map compositionally to weak-form residual terms, enabling deterministic reconstruction of the encoded PDE and comparison against an intended contract. We formalize this comparison as the Intent Fidelity Score (IFS), a structural metric covering governing terms, BCs, ICs, coefficients, and time scheme. Building on IFS, we develop a PDE-grounded refinement loop that uses deterministic violation reports to correct generated code iteratively. We evaluate on MooseBench, a 220-case multiphysics benchmark with PDE-level ground truth released with this work. On this benchmark, our method consistently improves mean IFS over direct generation, with gains concentrated on hard cases. On the subset where direct generation falls below IFS 0.7, refinement adds +0.22 to +0.41 absolute IFS. In the deployment audit, execution-only repair improves execution success while leaving 39-40% of all 220 cases runnable but still solving the wrong physics across the three main deployment-audit models, exposing executability and intent fidelity as separable failure modes. Static proof-of-concept experiments on four PDE-oriented DSLs (UFL/FEniCS, FreeFEM, FiPy, and Devito) suggest that the reconstruction-and-comparison pattern extends beyond MOOSE. These findings reinforce that executable simulation code should be verified against the mathematical structure it is intended to encode, not accepted on execution alone.

• The paper introduces a PDE-grounded intent verification approach using deterministic contract extraction and the Intent Fidelity Score (IFS) to compare intended vs. realized physics.
• It reveals that successful code execution does not ensure correct physics, with up to 40% of runnable cases showing misaligned simulation outputs.
• Validated on the MooseBench benchmark, the PDE-Refine loop iteratively improves simulation fidelity, offering a robust method for verifying LLM-generated multiphysics code.

PDE-Grounded Verification for LLM-Generated Simulation: An Expert Analysis

Motivation and Problem Statement

Execution-based evaluation has emerged as the de facto standard for assessing LLM-generated code, incorporating correctness checks via successful parsing, execution, and, often, convergence. However, such metrics are ill-suited for scientific simulation, where the specification is fundamentally mathematical—correctness must be judged at the level of encoded physics, not mere code execution. The paper "Your Simulation Runs but Solves the Wrong Physics: PDE-Grounded Intent Verification for LLM-Generated Multiphysics Simulation Code" (2605.09360) thoroughly dissects this gap, which the authors term the "comprehension–generation gap": an LLM may syntactically realize an executable input file, yet the resultant simulation can silently solve a PDE or physical scenario that diverges from user intent.

This issue is particularly acute in frameworks like MOOSE, where Kernels and Boundary Condition objects deterministically compose into the final encoded weak-form PDEs. Leveraging this, the authors formalize Intent Fidelity as the core correctness metric for scientific code generation, defining it via a deterministic reconstruction of the PDE realized by code and a comparison against an intended specification, contracted as the Intent Fidelity Score (IFS).

Figure 1: Silent-failure gallery for cavity diffusion simulations. Each panel depicts different, yet plausible, simulation outputs from files that violate different aspects of the intended specification (e.g., wrong boundary, coefficients, missing source term, etc.).

Framework and Methodology

Deterministic PDE Reconstruction and IFS

The central methodological contribution is a formal, deterministic mapping between a MOOSE input file and its encoded PDE—each Kernel, BC, and IC block is mapped to normalized PDE-operator tuples, enabling contract-based comparison. This contract includes the set of governing-equation terms, boundary and initial conditions, coefficients/material facts, and the numerical time scheme. The Intent Fidelity Score (IFS) quantifies agreement between the intended and realized contracts, using discrepancy-weighted checkpoints across terms, BCs, ICs, coefficients, and time scheme. This quantification is structural, not based on output behavior or test-case observables.

PDE-Grounded Refinement Loop

The deterministic contract comparison feeds into a refinement loop: LLM-generated code is evaluated against the intent contract, and violations are surfaced as structured, actionable reports, which are then recursively presented to the LLM for minimal, targeted corrections. This procedure, PDE-Refine, is architecturally distinct from both direct generation and from execution-driven self-debugging, as it does not rely on runtime traces or error logs but solely on structural contract discrepancies.

Figure 2: System architecture: The pipeline extracts the physics contract, synthesizes code, reconstructs the encoded contract, compares it, and iteratively refines via violation reports if necessary.

Experimental Design and Benchmarks

The MooseBench benchmark is introduced: a 220-case, multi-physics slate with ground-truth PDE-level contracts aligned to expert-written MOOSE inputs. This benchmark spans seven physical domains (e.g., transient/steady conduction, poromechanics, elasticity, phase-field, plasticity) and three complexity tiers, designed to test both simple and hard multiphysics settings.

Four LLMs—Claude Sonnet 4.6, GPT-5.4, GPT-4.1-mini, and DeepSeek V4 Flash—are evaluated under three primary paradigms: Direct (prompt-to-code), SpecGen (contract-driven code, but no deterministic feedback), and PDE-Refine (contract + deterministic feedback loop). An execution-audit registry is employed to standardize object realization and check for decoupling between executability and physics intent.

Key Results

Separation of Executability and Physics Fidelity

A major empirical finding is the clear decoupling of object executability and physics fidelity. Across all three main LLMs, there is a persistent regime where simulation files run (i.e., pass MOOSE’s InitExec2 and show plausible convergence) but solve the wrong physics; when only execution-based repair is used, up to 40% of runnable cases are misaligned in terms of encoded physics. When PDE-level feedback is incorporated (PDE-Refine or PDE-Reg), both IFS and “GoodExec” (runs + correct physics) increase, and “FalseExec” (runs but wrong physics) decreases substantially.

Figure 3: Compact PDE-pipeline diagnostics show that improved PDE extraction quality correlates with higher final code fidelity, with refinement gains concentrated on cases where direct generation is weakest.

Figure 4: IFS/MCS validation diagnostics: IFS detects structural and boundary-value errors not observable via execution; coefficient mismatches, while sometimes leaving IFS high, are flagged by the MCS diagnostic.

Quantitative and Subdimensional Analysis

• Mean IFS Improvement: Across the 220-case suite, mean IFS is consistently improved by PDE-Refine (e.g., for DeepSeek V4 Flash: Direct 0.60, SpecGen 0.67, PDE-Refine 0.78), with the strongest gains on “hard” cases (IFS_Direct < 0.7) where improvements of +0.22 to +0.41 are observed.
• Structural Error Coverage: 87.5% of imperfect outputs under direct generation harbor structural (term, BC, IC, time) errors detectable by IFS; the remainder are coefficient/material mismatches, for which a secondary Material Consistency Score (MCS) is proposed.

  Figure 5: Execution/fidelity quadrant analysis reveals a substantial "FalseExec" region under execution-only repair, mitigated when PDE-level feedback is added.

• Empirical Validation: 30-case controlled perturbation experiments confirm that IFS cleanly separates structural PDE mismatches from mere coefficient or parameter errors.
• Cross-DSL Transferability: The reconstruction-based methodology is ported to several PDE DSLs (UFL/FEniCS, FreeFEM, FiPy, Devito), with proof-of-concept demonstrations indicating its generality for frameworks with compositional PDE semantics.

  Figure 6: Residual-error analysis for PDE-grounded methods separates the comprehension bottleneck (PDE extraction) from downstream code fidelity.

Implications

Theoretical Implications

• Verifiability of Intent: The proposed methodology demonstrates that, for frameworks with explicit weak-form composition (e.g., MOOSE), it is possible to verify the structural alignment of LLM-generated simulation code deterministically at the contract level, with coverage guarantees over the represented semantic fragment.
• Structural vs. Behavioral Metrics: Execution, while necessary for mechanical validity, cannot serve as a proxy for physics-fidelity. The study exposes the essential requirement for intent-grounded structural verification in scientific code generation and problem specification.

Practical Implications

• Simulational Safety: For safety-critical applications (e.g., nuclear engineering), treating code execution as proof of correctness is fundamentally unsafe. The presented methodology provides a principled approach for discriminating between “runnable” and “intended” simulations, closing a blind spot in current LLM-based engineering workflows.
• Automated Refinement: The deterministic feedback loop enhances correctness without recourse to brittle or model-specific error detection heuristics, thus providing a practical, model-agnostic foundation for reliable scientific simulation code generation.

Future Directions

• Extension to More Complex DSLs: The portability to other PDE-oriented DSLs invites broader adoption, provided access to a semantically auditable intermediate representation.
• Integration with Verification/Validation Pipelines: The methodology sits upstream of classic V&V; integrating structural intent checks with mesh, solver, and discretization checks will provide holistic pipeline assurances.
• Enhanced Semantic Fragment Coverage: Registration of custom objects, higher-order constructs, and wider families of operators/kernels will expand applicability.
• Learned Diagnostic Layers: While current contract comparison is deterministic and symbolic, future work may leverage learned models to automate the mapping process or check for semantic equivalence in ill-specified cases.

Conclusion

The work provides a rigorous framework for PDE-grounded intent verification in LLM-generated multiphysics simulation code, demonstrating that runtime executability and physics fidelity are separable, and that deterministic contract-based diagnosis and refinement can substantially improve alignment with user-specified physics. Benchmark results establish the inadequacy of execution-only validation; physics-level alignment requires explicit PDE contract extraction, mapping, and violation-driven correction. The methodology is extensible, provides type-soundness guarantees within its semantic fragment, and is practically impactful for safety, correctness, and automation in scientific computing.

Figure 7: Refinement convergence diagnostic for DeepSeek V4 Flash, showing monotonic improvement in ground-truth IFS with successive refinement iterations.