Physics Alignment Benchmark

• Physics Alignment Benchmark is a rigorous evaluation framework that tests AI models’ ability to comply with physical laws across various modalities.
• It utilizes law-consistent metrics, adversarial parameter variations, and formal verification methods to assess genuine physical reasoning.
• The benchmark drives targeted improvements by exposing failures in dynamic, multi-step evaluations and guiding model refinement.

A Physics Alignment Benchmark is a rigorously designed evaluation suite that quantifies the extent to which AI models—especially LLMs, multimodal LLMs (MLLMs), and video world models—demonstrate not only formal correctness but also genuine understanding and compliance with physical laws, concepts, and reasoning chains. This class of benchmarks is specifically engineered to expose and measure failures of robust scientific reasoning and to distinguish between surface-level plausibility and true physics alignment across a range of modalities, domains, and difficulty levels.

1. Conceptual Foundations and Motivation

Physics alignment benchmarks emerged in response to recognized inadequacies in existing AI evaluation infrastructure. While models have achieved strong results in mathematics, programming, and even commonsense or visual reasoning, robust competence in genuine physics reasoning—spanning synthesis of diagrams, symbolic derivations, prediction of real-world dynamics, and formal verification—remains elusive. Earlier VQA and video prediction tasks (e.g., IntPhys, CausalVQA, PhysBench, VBench, PhyGenBench) were found to permit models to exploit visual-language shortcuts or perceptual plausibility metrics, circumventing deep physical understanding (Mak et al., 22 Jan 2026).

Physics alignment benchmarks are defined by:

• Explicit targeting of core physical laws, formal derivations, or law-consistent predictions
• Robustness and generalization stress tests (dynamic parameter variation, vision-necessary inputs, multi-step reasoning, real/simulated data)
• Modalities covering text, images, videos, executable code, and, in advanced settings, formal proof assistants
• Metrics that operationalize physical correctness, law consistency, and, in some cases, creativity or process-level alignment

2. Canonical Benchmarks and Their Scope

A suite of recent benchmarks underpins the current landscape of physics alignment assessment:

Benchmark	Scope	Modalities	Notable Features
PhysicsMind (Mak et al., 22 Jan 2026)	Sim+real mechanics, 3 laws	Image, Video	Law-consistent VQA & video gen; real+sim data
PhysUniBench (Wang et al., 21 Jun 2025)	Undergrad-level, 8 subfields	Text+Diagram	Model-in-loop AMT, detailed difficulty grading
ABench-Physics (Zhang et al., 7 Jul 2025)	Grad/Olympiad, static+dynamic	Text	Dynamic param. variation, strict numeric eval.
UGPhysics (Xu et al., 1 Feb 2025)	13 undergrad subjects, 4 skills	Text	Bilingual, 7 answer types, MARJ pipeline
SeePhys (Xiang et al., 25 May 2025)	Multi-domain, vision essential	Text+Diagram	21 diagram types, vision-essential split
Multi-Physics (Luo et al., 19 Sep 2025)	High-school, 11 CN subjects	Text+Diagram	CoT integrity, modality ablation
LeanPhysBench (Li et al., 30 Oct 2025)	College/competition (Lean4)	Formal proofs	Units integration, PhysLib impact
SymPyBench (Imani et al., 5 Dec 2025)	University, parametric, code	Text+Code	Executable code, dynamic metrics

These benchmarks collectively test a spectrum from conceptual understanding (multiple-choice, open-ended, derivations) through dynamic law application (e.g., video rollouts, code execution) to formal symbolic verification.

3. Task Modalities and Law-Aware Protocols

Task and input/output structures are adapted according to modality:

• Law-Consistent VQA and Video Generation: PhysicsMind (Mak et al., 22 Jan 2026) evaluates whether models can infer and predict the evolution of physical systems in compliance with Center of Mass, Lever Equilibrium, and Newton's First Law. VQA tasks require models to parse mass, distance, and geometric features from multimedia scenes and apply mechanical calculations; video generation tasks test whether rollouts obey physical constraints rather than just look plausible. Custom law-aware metrics—such as segmentation-mask IoU for CoM, trajectory RMSE for inertia, and torque balance for levers—disambiguate between correct mechanism and visual surface similarity.
• Multimodal Reasoning with Diagrams: SeePhys (Xiang et al., 25 May 2025), Multi-Physics (Luo et al., 19 Sep 2025), and PhysUniBench (Wang et al., 21 Jun 2025) stress integration of text and heterogeneous diagrams. Typical question types include open-ended derivations based on free-body diagrams, circuit schematics, or geometric figures that cannot be bypassed by text-only shortcuts. Evaluation is conducted both “with images” and “without images” to measure reliance on genuine visual understanding.
• Dynamic/Adversarial and Parameterized Evaluation: ABench-Physics (Zhang et al., 7 Jul 2025) and SymPyBench (Imani et al., 5 Dec 2025) introduce dynamic variation by perturbing numerical constants and requiring models to solve all variants of a template under tight tolerance constraints. This approach distinguishes pattern recall from re-derivation and exposes brittleness to distributional shift.
• Formal and Code-driven Verification: LeanPhysBench (Li et al., 30 Oct 2025) leverages formalized statements in the Lean4 proof system, requiring strictly verified chain-of-thought including correct unit handling. SymPyBench (Imani et al., 5 Dec 2025) associates each problem with executable Python code, enabling direct symbolic/numeric evaluation and consistency checking across dynamic variants.

4. Scoring Metrics, Evaluation, and Failure Modes

Physics alignment benchmarks introduce a range of quantitative measures, each aligned to the underlying physics construct:

• Law-Consistency Metrics: IoU and centroid error (CoM), trajectory RMSE and directional consistency (inertia), final-state accuracy (lever equilibrium) (Mak et al., 22 Jan 2026)
• Dynamic Robustness Drop: Accuracy decay Δ between static and dynamic variants (ABench-Physics (Zhang et al., 7 Jul 2025)), Consistency/Failure/Confusion rates (SymPyBench (Imani et al., 5 Dec 2025))
• Open-ended and Symbolic Equivalence: Use of algebraic checkers (SymPy), stepwise explicit equivalence, and expert/LMM-based scoring for algebraic/numeric responses (Wang et al., 21 Jun 2025, Xu et al., 1 Feb 2025, Imani et al., 5 Dec 2025)
• Chain-of-Thought Integrity: Stepwise evaluation of reasoning traces (ASA in Multi-Physics (Luo et al., 19 Sep 2025)), process-level correctness in open-ended derivations (Wang et al., 21 Jun 2025)
• Formal Pass Rates: Pass@k for formal proof statements under the Lean4/PhysLib environment (Li et al., 30 Oct 2025)
• Creativity and Understanding: Aggregated scoring over correctness, expert-rated difficulty, and surprise metrics (Barman et al., 29 Jul 2025)
• Safety and Harmlessness: In physics safety contexts (SciSafeEval (Li et al., 2024)), metrics such as refusal rate, harmlessness score, and attack success rate quantify alignment with safety/ethical standards in knowledge retrieval.

Failure patterns consistently observed include:

• Misinterpretation of physical diagrams or images (e.g., confusion of force vectors, geometry)
• Reliance on visual or textual heuristics instead of mechanistic reasoning (e.g., “heavier-object-goes-down” in levers)
• Unit, arithmetic, or significant-figure errors in numeric computation
• Incomplete or derailed symbolic reasoning chains
• Marked drop in accuracy under parameter/setting shifts, revealing overfitting or memorization
• In formal settings, inability to manipulate units/dimensions or invoke specialized lemmas

5. Dataset Construction, Curation, and Annotation

Physics alignment benchmarks are characterized by multi-staged, expert-driven curation pipelines to prevent data leakage and ensure high challenge:

• Diverse Sourcing: Problems are drawn from university textbooks, banked exam and Olympiad problems, custom parameterized templates, and real/simulation video recordings (Mak et al., 22 Jan 2026, Wang et al., 21 Jun 2025, Xiang et al., 25 May 2025, Imani et al., 5 Dec 2025).
• Difficulty Calibration: Multi-round model-in-the-loop rollouts and difficulty grading eliminate trivial instances (Wang et al., 21 Jun 2025, Zhang et al., 7 Jul 2025).
• Annotation and Verification: Expert review, LLM draft generation, automated extraction of referents (mass, forces, charges), ambiguity pruning, and bilingual translation/validation pipelines (Xu et al., 1 Feb 2025, Wang et al., 21 Jun 2025).
• Rule-based and Model-Assisted Judgment: Two-stage evaluation for answer equivalence (rule-based for well-defined types, model-based for ambiguous or non-canonical cases) ensures high inter-annotator agreement and adaptability as new model outputs emerge (Xu et al., 1 Feb 2025).

6. Alignment Applications and Future Directions

Physics alignment benchmarks serve both as diagnostic and training-enhancement instruments:

• Targeted Model Improvement: Sub-discipline and skill breakdowns direct curriculum augmentation where weaknesses persist (e.g., symbolic derivation, diagram interpretation, handling of relativistic/quantum phenomena).
• Curriculum and Reward Modeling: Integration of accuracy/process-level correctness as auxiliary losses or reward signals during RLHF or curriculum learning stages (Wang et al., 21 Jun 2025, Zhang et al., 7 Jul 2025).
• Safety and Robustness Testing: Adversarial prompt “jailbreaks” and benign/harmful query distinctions, as in SciSafeEval (Li et al., 2024), are essential for safe deployment.
• Extensible Formal Libraries: The growth and modularization of foundational libraries (e.g., PhysLib in Lean4, dynamic code in SymPyBench) are essential for transferring alignment methodologies to broader STEM domains (Li et al., 30 Oct 2025, Imani et al., 5 Dec 2025).
• Multi-Modal and Dynamic Expansion: Ongoing work emphasizes richer physical phenomena (friction, collisions, energy conservation), more advanced topics (field theory, multi-body dynamics), and incorporation of diagrams, videos, and executable code as evaluation artifacts (Mak et al., 22 Jan 2026, Xiang et al., 25 May 2025, Imani et al., 5 Dec 2025).

7. Significance and Outlook

Physics alignment benchmarks now form the backbone of rigorous diagnostics for high-level physical reasoning in generative AI and formal systems. Early experiments reveal that, despite impressive progress in mathematical and statistical domains, modern LLMs and MLLMs still fall short—sometimes drastically—when confronted by dynamic, multi-step, multimodal tasks that require genuine compliance with physical laws. Continual development of these benchmarks is recommended for model introspection, targeted refinement, and robust alignment, informing both the next generation of AI for Science and the safe, interpretable deployment of LLMs in research and education (Mak et al., 22 Jan 2026, Wang et al., 21 Jun 2025, Zhang et al., 7 Jul 2025, Xu et al., 1 Feb 2025, Li et al., 2024, Li et al., 30 Oct 2025).