ABench-Physics: LLM Physics Reasoning Benchmark

• ABench-Physics is a benchmark that tests LLMs' physical reasoning through high-difficulty, numerical-answer physics problems.
• It consists of a static set of 400 advanced problems and a dynamic set of 100 parameterized problems to assess robustness against variations.
• The benchmark emphasizes precise computations and adaptability, exposing shortcomings in current LLM performance on complex physical models.

Searching arXiv for ABench-Physics and closely related physics-reasoning benchmarks. ABench-Physics is a benchmark for evaluating physical reasoning and generalization in LLMs through high-difficulty physics problems with strict numerical-answer requirements and a dynamic robustness component (Zhang et al., 7 Jul 2025). It was introduced to address limitations attributed to existing physics benchmarks, including limited difficulty, multiple-choice formats, and static evaluation settings that fail to capture physical modeling ability. The benchmark has two components: a static set of 400 graduate- or Olympiad-level problems and a dynamic subset of 100 parameterized problems equipped with an automatic variation engine. Its stated purpose is to expose and quantify the gap between surface-level pattern matching and physical reasoning and modeling, especially when semantically equivalent changes in problem parameters are introduced (Zhang et al., 7 Jul 2025).

1. Motivation and problem setting

ABench-Physics is motivated by the claim that physics reasoning demands both exact numerical computation and deep conceptual understanding. In the benchmark description, these demands include choosing appropriate reference frames, identifying conserved quantities, constructing differential equations, translating textual descriptions into mathematical formulations such as Newton’s second law $F=ma$ or a Lagrangian, and solving the resulting equations (Zhang et al., 7 Jul 2025).

The benchmark is explicitly framed against three perceived deficiencies of prior evaluations. First, many public benchmarks do not reach graduate or Olympiad difficulty. Second, multiple-choice or expression-only formats do not force precise numerical answers and therefore under-test multi-step algebra and units tracking. Third, static test sets permit memorization of prompts and answer keys rather than genuine generalization, because they do not probe how a model reacts to small but semantically equivalent perturbations of the problem instance (Zhang et al., 7 Jul 2025).

This emphasis makes ABench-Physics a benchmark of both correctness and robustness. A plausible implication is that it is designed not merely to test whether an LLM can solve a familiar physics template once, but whether it can preserve the underlying reasoning when parameters, coordinates, or boundary conditions are altered.

2. Benchmark composition

ABench-Physics consists of two complementary subsets, Phy_A and Phy_B (Zhang et al., 7 Jul 2025).

Component	Size	Description
Phy_A_fixed_400	400	Static set drawn from graduate-level coursework and international physics Olympiad sources
Phy_B_dynamic_100	100	Dynamic set of parameterized template problems with automatic variation

Phy_A_fixed_400 is the static portion. It contains 400 problems drawn from graduate-level coursework and international physics Olympiad sources. Its subject coverage includes classical mechanics, electromagnetism, optics, quantum physics, thermodynamics and statistical mechanics, fluid dynamics, semiconductor physics, atomic and molecular physics, and select modern topics. The answer format is numerical output only; each problem specifies required units and the number of significant figures. A response is accepted under a relative-error tolerance of $1\%$, namely

$\bigl|\hat y - y^*\bigr| \le 0.01\,|y^*|\,.$

An example notation used in the benchmark is

$x(t)=A\sin(\omega t+\phi)\,,$

for a simple harmonic motion problem asking for the maximum speed (Zhang et al., 7 Jul 2025).

Phy_B_dynamic_100 is the dynamic portion. It comprises 100 template problems of moderate to high difficulty, parameterized in LaTeX. The automatic variation engine parses the LaTeX source, locates numeric constants and units, and applies controlled perturbations. The described perturbations include changing initial velocity $v_0$ by $\pm 10$–$20\%$, swapping mass $m$, inverting coordinate axes, and altering boundary conditions. The current release ships with 3 variations per template, with future versions intended to expand this number (Zhang et al., 7 Jul 2025).

The distinction between the two subsets is central. Phy_A measures performance on a fixed corpus of difficult problems; Phy_B measures whether success survives systematic variation.

3. Dynamic variation and evaluation protocol

The dynamic subset operationalizes robustness through a variation engine rather than through adversarial paraphrase. The variation types listed for Phy_B include changing initial conditions, varying physical constants, rotating coordinate systems, and altering functional forms, such as replacing $\mu_k$ with $\mu_s$ in friction problems (Zhang et al., 7 Jul 2025). These changes are meant to preserve the physical structure while preventing benchmark success through rote recognition.

Evaluation requires strict answer formatting. Models must return a scalar, or a small tuple if specified, accompanied by the correct unit. A lightweight LaTeX-based evaluator extracts $1\%$0 from model output and compares it with the ground truth $1\%$1. The evaluator handles decimal notation, scientific notation such as $1\%$2, and mixed fractions (Zhang et al., 7 Jul 2025).

The scoring rules differ between the two components. For Phy_A, each problem is judged independently under the $1\%$3 relative-error criterion, and

$1\%$4

For Phy_B, the benchmark uses an all-variants criterion: $1\%$5 A template receives credit only if the model answers correctly on every automatically generated variant under the same $1\%$6 tolerance rule (Zhang et al., 7 Jul 2025).

This all-or-nothing rule is diagnostically important. A plausible implication is that Phy_B penalizes brittle numerical behavior more sharply than conventional averaged-instance evaluation.

4. Reported empirical results

ABench-Physics was evaluated on several state-of-the-art LLMs, including GPT-4.1, GPT-4o, OpenAI o1, o3, o3-mini, Anthropic Claude 3.7 Sonnet (“Thinking”), Google Gemini 2.5 Pro and Pro (“Reasoning”), DeepSeek-R1 and V3, Mistral, Qwen2.5, Qwen3, and QwQ-32B (Zhang et al., 7 Jul 2025).

The reported findings emphasize persistent limitations. Even top systems solve fewer than half of the static graduate/Olympiad problems, and performance consistently degrades under dynamic numeric perturbations. The benchmark summary reports an average drop across all models of 22.5 percentage points, and interprets this as evidence of reliance on memorized solutions rather than stable physical reasoning (Zhang et al., 7 Jul 2025).

The results section also reports that models fine-tuned with reinforcement learning, specified as RLHF or programmatic feedback, show smaller $1\%$7 than purely supervised-fine-tuned counterparts. The stated interpretation is that RL objectives encourage better numerical robustness, although not enough to eliminate the gap (Zhang et al., 7 Jul 2025).

The benchmark therefore presents two quantitative conclusions at once: absolute performance remains limited on hard physics, and relative robustness worsens when a familiar template is perturbed.

5. Failure modes and diagnostic interpretation

The analysis section attributes benchmark failures to several recurring deficiencies. One is superficial pattern matching: models often recognize a familiar form, such as projectile motion, but fail when constants or coordinate dependencies change. Another is units and dimensional analysis errors, where models ignore prescribed units or carry through inconsistent unit conversions. A third is conceptual missteps, including forgetting potential-energy zero-points and making sign errors in vector decomposition (Zhang et al., 7 Jul 2025).

The robustness analysis sharpens this diagnosis. Dynamic variations expose brittleness: a model that answers one numeric instantiation correctly often fails on another instantiation where the same algebraic steps apply. The benchmark authors further report that many LLMs revert to heuristics, exemplified by rote use of $1\%$8, but cannot re-derive from first principles under changed parameters (Zhang et al., 7 Jul 2025).

These observations align with the benchmark’s stated goal of distinguishing pattern completion from physical modeling. A plausible implication is that ABench-Physics is less concerned with breadth of undergraduate curriculum coverage than with invariance of reasoning under controlled perturbation.

6. Relation to adjacent physics benchmarks and terminological disambiguation

ABench-Physics belongs to a broader 2025 wave of physics-oriented evaluation. PHYSICS contains 1,297 expert-annotated university-level problems across six core areas, includes 298 multimodal problems with figures or diagrams, and reports that o3-mini achieves 59.9% accuracy (Feng et al., 26 Mar 2025). UGPhysics provides 5,520 undergraduate-level physics problems in both English and Chinese, covering 13 subjects with seven answer types and four physics reasoning skills, with the highest overall accuracy reported as 49.8% for OpenAI-o1-mini (Xu et al., 1 Feb 2025). PhysUniBench consists of 3,304 undergraduate-level multimodal questions spanning 8 major sub-disciplines, each accompanied by one visual diagram, and reports that GPT-4o mini achieves about 34.2% accuracy (Wang et al., 21 Jun 2025).

Other benchmarks extend beyond static question answering. Gravity-Bench-v1 is an environment-based benchmark in which agents must plan observations in a two-body gravitational simulator under an experimental budget, reaching up to $1\%$9 under full observation but dropping below $\bigl|\hat y - y^*\bigr| \le 0.01\,|y^*|\,.$0 under budget constraints (Koblischke et al., 30 Jan 2025). FEABench evaluates whether LLMs and LLM agents can reason over natural-language engineering specifications and operate COMSOL Multiphysics through its API; its best performing strategy generates executable API calls 88% of the time (Mudur et al., 8 Apr 2025). This suggests that ABench-Physics is part of a larger shift from static answer selection toward benchmarks that test robustness, interaction, and end-to-end scientific task execution.

The term “ABench-Physics” also appears in an unrelated earlier context. In a 2010 Geant4 low-energy electromagnetic data-management R&D study, “ABench-Physics” denotes a benchmarking suite for physics data handling, including file I/O, in-memory organization, interpolation, and lookup, rather than an LLM physics-reasoning benchmark (Han et al., 2010). In current LLM evaluation usage, however, ABench-Physics refers to the 2025 two-part benchmark built around Phy_A and Phy_B (Zhang et al., 7 Jul 2025).