How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks

Abstract: Low reported scores on leading physics benchmarks, including those featured in the Artificial Analysis Intelligence Index (2026), suggest that frontier LLMs still struggle with advanced physics, a demanding test of their scientific reasoning and quantitative problem-solving abilities. Yet this impression does not always align with domain experts' experiences using these models in their work. We revisit these reported findings by evaluating frontier models on six widely used physics benchmarks and auditing them with experts, focusing on text-only problems with verifiable final answers. For each subfield of physics, faculty and graduate researchers with relevant expertise carefully review problem statements, reference solutions, and model responses to distinguish genuine model errors from grader errors, incorrect reference solutions, and ambiguous or underspecified questions. Most audited cases initially evaluated as incorrect reflect these benchmarking issues rather than errors in the models' physics reasoning. We then ask experts to address these benchmarking issues by correcting erroneous reference solutions and repairing or excluding flawed questions. We find that GPT-5.6-Sol's measured mean@4 rises from 47.3% to 78.7% on HLE-Physics and from 61.0% to 87.2% on CMT-Benchmark, while its corrected pass@4 reaches 94.4% on the 54 retained CritPt challenges. Corrected scores are computed on the retained evaluation subsets following expert review. Scores on the audited subsets of UGPhysics, PRISM-Physics, and PHYBench also rise substantially after correction. These findings suggest that current benchmarks substantially understate frontier models' ability to solve well-posed physics problems. Near-saturation on these closed-ended tasks highlights the need for more demanding, expert-validated evaluations.

• The audit revealed that most low scores in physics benchmarks are due to benchmark errors, grader errors, and vague/incomplete questions, not the models' physics understanding.
• After correcting these issues, GPT-5.6-Sol achieved pass@4 of 94.44% on CritPt and above 90.0% on most benchmarks, indicating strong performance on well-posed, closed-ended physics problems.
• The paper concludes that benchmark errors dominate measured errors once model performance exceeds benchmark quality, emphasizing the need to validate and improve benchmark quality to assess further model progress.

Research question and central thesis

The paper examines whether low scores on contemporary physics benchmarks measure genuine deficiencies in frontier LLMs or instead reflect defects in benchmark construction and evaluation. Its central claim is contradictory to the prevailing interpretation of these scores: after expert review, most apparent model failures are attributable to incorrect reference solutions, underspecified or ambiguous questions, and evaluator errors rather than failures of physics reasoning. The authors consequently argue that frontier models are close to saturation on a substantial class of closed-ended, text-only physics problems.

The study focuses on six benchmarks: UGPhysics, PHYBench, PRISM-Physics, HLE-Physics, CMT-Benchmark, and CritPt. These span undergraduate problems, Olympiad-level questions, advanced physics, condensed matter theory, and expert-authored frontier challenges. The distinction between publicly sourced and expert-authored datasets is analytically important. Public-source questions may be contaminated by training data, whereas expert-authored questions offer lower presumed contamination risk but are not thereby guaranteed to be correct or sufficiently specified.

The paper evaluates GPT-5.6-Sol, Claude Fable 5, and Gemini 3.1 Pro. GPT-5.6-Sol and Fable 5 use tools on the expert-authored benchmarks, while Gemini 3.1 Pro is evaluated without tools. The primary reported metric is mean@4, except for the externally reported pre-audit CritPt results, which use mean@5. The corrected evaluations use retained or repaired subsets and therefore are not always directly comparable with pre-audit scores computed on the original question sets.

The paper’s main empirical conclusion is that benchmark error becomes the dominant source of measured error once model capability exceeds benchmark quality. On the four pooled audit sets comprising 250 rejected cases, expert review attributes 143 cases to benchmark defects, 95 to grader errors, and only 12 to model errors. Thus, 95.2% of the audited rejections are not model errors. The stronger 97.37% figure reported for the public-source audit subsets reflects a different denominator and subset: 148 of 152 audited cases were attributed to benchmark or grader errors.

Audit methodology and error attribution

The evaluation pipeline has three stages. First, the models are evaluated using the original benchmark materials and available evaluators. Second, experts inspect problem statements, reference solutions, model responses, and—where available—evaluator decisions. Third, the authors either correct the evaluation procedure, repair the question or reference solution, or exclude the item when no defensible repair is possible.

The audit distinguishes three mutually exclusive categories:

• Model error: the problem is well posed, the reference answer is correct, and the model answer is incorrect.
• Grader error: the problem and reference answer are correct, the model answer is correct, but the evaluator rejects it.
• Benchmark error: the problem statement or reference solution is defective, including ambiguity, missing assumptions, inconsistent conditions, or an incorrect answer.

This decomposition is methodologically appropriate because raw benchmark rejection conflates these distinct failure modes. In particular, a benchmark with a defective reference answer imposes an irreducible error floor: even a model that solves the underlying problem correctly cannot receive credit for an answer that disagrees with the stored ground truth.

The audit design is asymmetric across benchmarks. For HLE-Physics, PHYBench, PRISM-Physics, and UGPhysics, the authors audit questions for which all GPT-5.6-Sol attempts were rejected. This efficiently targets apparent failures but does not estimate the prevalence of defects among accepted items. For CMT-Benchmark and CritPt, all selected questions are audited, regardless of the model’s original outcome. This difference matters when interpreting the error rates: the pooled attribution is a decomposition of rejected cases in four benchmarks, not a fully representative defect estimate for every item in the six datasets.

The audit also reveals substantial inter-reviewer disagreement. Of 196 doubly reviewed questions, 140 received matching labels and 56 required adjudication by a third review. The resulting 71.43% initial agreement indicates that distinguishing an underspecified problem from a model mistake is itself nontrivial, especially when the intended assumptions are implicit in disciplinary conventions. The paper reports the adjudication procedure but does not provide a formal inter-rater reliability statistic such as Cohen’s $\kappa$.

Pre-audit scores and corrected performance

The central quantitative result is the large increase in measured accuracy after validation and repair. The gains are particularly pronounced for benchmarks using brittle rule-based evaluators or defective reference materials.

Benchmark	GPT-5.6-Sol pre-audit	GPT-5.6-Sol corrected	Corrected pass@4
PHYBench	26.50%	90.23%	95.40%
PRISM-Physics	13.00%	94.59%	95.95%
UGPhysics	83.00%	92.07%	93.90%
HLE-Physics	47.28%	78.66%	91.38%
CMT-Benchmark	61.00%	87.24%	97.96%
CritPt	32.29%*	87.50%	94.44%

*Pre-audit CritPt is Artificial Analysis’s mean@5 on 70 challenges; the corrected result is mean@4 on 54 retained or repaired challenges.

The largest relative changes occur on PRISM-Physics and PHYBench. GPT-5.6-Sol’s mean@4 rises from 13.00% to 94.59% on PRISM-Physics and from 26.50% to 90.23% on PHYBench. These changes do not imply that the original benchmark questions were uniformly defective. Rather, the audited rejected subsets contain a high concentration of evaluator and question defects, and the corrected scores are computed after excluding flawed questions and replacing the original grading process with a shared HLE-adapted evaluator.

On the expert-authored datasets, the effect remains substantial but is less extreme. GPT-5.6-Sol’s HLE-Physics score rises from 47.28% to 78.66%, while its pass@4 rises from 55.94% to 91.38%. On CMT-Benchmark, mean@4 increases from 61.00% to 87.24%, and pass@4 reaches 97.96%. These results support the paper’s claim that expert authorship alone does not ensure benchmark validity. The CMT audit identifies defects in 30 of 50 questions; 29 are repaired and one is excluded.

Figure 1: Pre-audit and validated/repaired performance on six physics benchmarks for GPT-5.6-Sol, Fable 5, and Gemini 3.1 Pro.

CritPt provides the most pronounced example. GPT-5.6-Sol’s externally reported pre-audit mean@5 is 32.29% on 70 challenges. After experts audit 56 challenges, repair 19, exclude two, and independently derive reference solutions, the corrected mean@4 on 54 challenges is 87.50%, with pass@4 of 94.44%. The comparison is not a controlled before-and-after measurement on an identical item set: the attempt budget, metric, retained questions, reference solutions, and evaluation process differ. The result nevertheless demonstrates that the original aggregate score is highly sensitive to benchmark validity.

The same pattern appears across models. On the corrected public-source subsets, GPT-5.6-Sol reaches 90.23% on PHYBench, 94.59% on PRISM-Physics, and 92.07% on UGPhysics. Fable 5 reaches 87.64%, 84.80%, and 87.80%, respectively, while Gemini 3.1 Pro reaches 89.94%, 87.84%, and 90.85%. On expert-authored benchmarks, the corrected scores remain lower for Gemini, but its HLE-Physics score still rises from 40.97% to 64.87%, and its CritPt score rises from 17.71% to 54.63%.

Sources of benchmark and grader failure

The pooled audit provides the clearest evidence for the paper’s diagnosis. Of 250 rejected cases across HLE-Physics, PHYBench, PRISM-Physics, and UGPhysics, 143 are benchmark errors, 95 are grader errors, and 12 are model errors.

Figure 2: Error attribution across audited benchmarks, showing the prevalence of benchmark defects and the concentration of grader errors in rule-based evaluation pipelines.

The distribution varies by benchmark. PHYBench contains 13 benchmark errors, 40 grader errors, and 3 model errors among 56 audited rejections. PRISM-Physics contains 26 benchmark errors and 48 grader errors among 74 rejected cases, with no audited model errors. UGPhysics contains 18 benchmark errors, 3 grader errors, and 1 model error among 22 audited rejections. HLE-Physics contains 86 benchmark errors, 4 grader errors, and 8 model errors among 98 rejected cases.

These figures support two distinct conclusions. First, benchmark construction defects are widespread, including among expert-authored datasets. Second, rule-based graders are especially vulnerable to mathematically equivalent answers expressed in different syntactic forms. The two failure types should not be conflated: correcting a reference solution changes the target, whereas improving the evaluator changes recognition of an answer that already satisfies the target.

A representative PHYBench failure involves a rope-tension answer written as

$\frac{P}{3\sqrt{6}}$

when the reference answer is

$\frac{\sqrt{6}}{18}P.$

The expressions are algebraically identical, but the Expression Edit Distance evaluator assigns a score of zero. Similar failures arise from reordered scalar factors, equivalent notation, alternative sign conventions, and derivations whose limiting form matches the requested answer. These are not marginal formatting issues in a quantitative benchmark: when binary accuracy is derived from exact or near-exact symbolic comparison, syntactic brittleness directly distorts model rankings.

The PRISM-Physics audit exposes a more fundamental problem. One item asks for an electron’s time of flight while providing neither the required distance nor timing information. The reference answer selects a numerical option despite the quantity being underdetermined. In this case, no improvement to answer normalization can repair the item; the problem lacks enough information to define a unique target.

The UGPhysics audit identifies a conceptual ambiguity involving damped motion. The reference answer treats critical damping as the fastest return to equilibrium, but that statement is true only under an additional condition such as no overshoot or a specified optimization criterion. With different initial conditions and without the no-overshoot restriction, an underdamped trajectory can reach equilibrium sooner. The defect lies in the unexpressed criterion, not necessarily in the model’s physical reasoning.

Repairs to expert-authored problems

The expert-authored benchmarks reveal that advanced content and benchmark validity are separate properties. CMT-Benchmark contains technically sophisticated condensed-matter questions, but many require assumptions about dimensionality, normalization, boundary conditions, operator conventions, or correlation functions that are not stated.

One repaired CMT item concerns a quantum Ising model. The original problem does not specify the spatial dimension or bond-counting convention, yet asks whether the excitation gap vanishes at $h=1$. It also uses an ordinary two-point correlator where exponential decay applies to the connected correlator in the presence of a longitudinal field. The repair specifies a one-dimensional chain, counts each bond once, restricts the symmetry-breaking statement to $g=0$, and replaces the ordinary correlator with

$$\langle \sigma_i^z \sigma_j^z \rangle - \langle \sigma_i^z \rangle \langle \sigma_j^z \rangle.$$

The corrected answer changes from the original reference’s $(a;c)$ to $(a;b;c)$. The model’s original response already gives $(a;b;c)$, so the model is correct under the repaired formulation.

CritPt exhibits analogous problems in more advanced settings. One Kitaev honeycomb model question asks for ground-state degeneracies and energies while specifying only $J_x=J_y=J_z=1$. It does not state whether the Hamiltonian uses Pauli matrices $\frac{P}{3\sqrt{6}}$0 or spin operators $\frac{P}{3\sqrt{6}}$1. The two conventions produce energies differing by a factor of four. The authors repair the question by writing the Hamiltonian explicitly, thereby converting an apparently numerical question into a well-posed one.

This type of repair is not cosmetic. Numerical answers in many-body physics depend on normalization, sign, lattice geometry, boundary conditions, gauge sector, and operator conventions. If these are omitted, a model can produce a physically coherent derivation while receiving an objectively unverifiable score. The audit therefore treats missing assumptions as benchmark errors when they are necessary to determine the intended answer and cannot be inferred unambiguously.

Interpreting near-saturation

After correction, the leading models achieve approximately 90–98% pass@4 on most retained benchmarks. GPT-5.6-Sol reaches 95.40% on PHYBench, 95.95% on PRISM-Physics, 93.90% on UGPhysics, 91.38% on HLE-Physics, 97.96% on CMT-Benchmark, and 94.44% on CritPt. These results are consistent with the paper’s claim of near-saturation on the studied task class: closed-ended, text-only problems with verifiable final answers and, for several expert-authored benchmarks, tool access.

The distinction between mean@4 and pass@4 is informative. Mean@4 measures average success across four attempts, whereas pass@4 measures whether at least one attempt succeeds. The gap between the two indicates residual stochasticity in model inference. For GPT-5.6-Sol on HLE-Physics, mean@4 is 78.66% while pass@4 is 91.38%; on CMT-Benchmark, the corresponding values are 87.24% and 97.96%. Thus, repeated sampling materially improves coverage even after benchmark correction.

The results do not establish that frontier models can perform end-to-end physics research. The benchmark tasks are closed-ended and generally terminate in a definite answer. They do not measure sustained problem formulation, experimental design, theory selection, model criticism, identification of unknown unknowns, or validation against empirical data. The paper explicitly reports that agentic attempts on open theoretical-physics problems made considerably less progress than comparable efforts in mathematics and did not fully solve any of the selected problems. This distinction is central: high performance on validated problem-set-style questions is evidence about quantitative problem solving under specified conditions, not about autonomous scientific research.

The paper’s interpretation is independently supported by an external CritPt correction reported in the Claude Fable 5.1 and Claude Mythos 5.1 system card. Anthropic reports a mean@16 of 88.4% on an internally corrected CritPt variant after revising 31 of 71 problem statements. The evaluations are not directly comparable because they use different models, question sets, attempt budgets, and judges, but the direction of the effect is consistent.

Limitations and open questions

The principal limitation is that corrected scores are not always measured on the same questions, with the same attempt budget, or under the same evaluator as the pre-audit scores. The comparison therefore quantifies the effect of changing the evaluation regime, not a pure causal effect of correcting labels while holding every other variable fixed. This is especially important for CritPt, where pre-audit mean@5 on 70 challenges is compared with corrected mean@4 on 54 retained or repaired challenges and independently derived reference solutions.

The audit sample is also selected in ways that constrain generalization. For four benchmarks, only questions rejected in all GPT-5.6-Sol audit attempts are reviewed. Accepted questions are not audited at comparable rates, so the reported attribution cannot be interpreted as the overall defect rate of each benchmark. The public-source evaluation is limited further by incomplete access to reference solutions: only 100 PHYBench questions with available solutions are analyzed, and the PRISM-Physics and UGPhysics results use sampled subsets.

Expert judgment is treated as ground truth, but expert review is not error-free. The 28.57% disagreement rate among doubly reviewed cases demonstrates that attribution depends on interpretation and disciplinary conventions. The protocol uses third-party adjudication, but the paper does not establish the reliability of the final labels with an independent blinded replication or a formal multi-rater reliability analysis.

Tool access creates another comparability issue. GPT-5.6-Sol and Fable 5 use coding or agentic tools on expert-authored benchmarks, while Gemini 3.1 Pro does not. Corrected cross-model differences therefore combine model capability, tool access, reasoning configuration, and evaluator behavior. The paper’s strongest claims concern the existence of benchmark defects and the inflation of apparent failure rates, not a clean ranking of the three systems.

Finally, near-saturation is task-class-specific. The retained questions may be easier than the original benchmark distributions after defective and ambiguous items are removed, and repeated attempts increase pass@4. The paper leaves open whether a similarly rigorous audit of genuinely novel, open-ended, experimentally grounded, or adversarially specified physics tasks would produce comparable corrected performance.

Conclusion

The paper presents an evaluation audit rather than a new model architecture or reasoning algorithm. Its contribution is to show that raw physics benchmark scores can substantially underestimate frontier-model performance when benchmark defects and evaluator brittleness are not separately measured. Across 250 audited rejected cases in four pooled datasets, only 12 are attributed to model errors; the remainder arise from flawed questions or grading procedures.

After expert validation and repair, GPT-5.6-Sol reaches 90% or higher pass@4 on most retained benchmarks and 94.44% on the retained CritPt challenges. These results support the narrower but consequential conclusion that frontier models are highly capable on well-posed, closed-ended physics problems. They do not establish comparable competence in open-ended physics research. The paper therefore reframes the empirical problem: once benchmark error exceeds model error, improving the benchmark’s validity becomes a prerequisite for measuring further progress.