TheoPhysEval: AI in Theoretical Physics

Updated 15 October 2026

TheoPhysEval is a benchmark designed to evaluate AI agents' ability to solve complex, technically demanding theoretical-physics problems autonomously, requiring skills in literature search, code execution, numerical computation, and interpretation.
The benchmark includes 50 unpublished problems across various areas of theoretical physics, such as quantum field theory, scattering amplitudes, and cosmology, each solvable by experts in about a day, highlighting challenges including parameter changes, dimensionality shifts, and specialized analytical expressions.
Evaluation occurs through end answers verification without requiring complete derivations, agents are tested on final numerical or symbolic outputs and procedural correctness, indicating the capability of AI in precise technical calculations rather than broader research competencies.

TheoPhysEval is a benchmark for autonomous, research-level theoretical-physics problem solving by AI agents. It evaluates whether an agent can take a technically demanding, well-posed calculation through to a verifiable final answer without human hints, using tools such as literature search, code execution, numerical computation, symbolic algebra, differential-equation solvers, exact diagonalization, and simulation. Its 50 unpublished problems span areas including quantum field theory, scattering amplitudes, holography, gravitation and cosmology, quantum phases of matter, electronic-structure theory, nonequilibrium statistical mechanics, and classical physics. The benchmark measures concrete technical problem solving rather than the broader capacities required for autonomous scientific research, such as selecting important questions, formulating theories, judging significance, or generating breakthroughs (Chandra et al., 9 Oct 2026).

1. Scope and conceptual foundations

TheoPhysEval targets a specific layer of theoretical-physics research: technically difficult calculations whose methods are known or recoverable, but whose parameter choices, dimensionality, geometry, numerical values, or computational requirements are nontrivial. A typical problem is estimated to be solvable by a world expert in approximately a day or less, while requiring substantially longer for experts from adjacent specialties.

The benchmark deliberately excludes major open problems and breakthrough questions. Its problems are instead “turn-the-crank” research calculations that can require:

adapting equations from recent literature;
changing dimensionality, geometry, or parameter values;
deriving specialized analytical expressions;
writing numerical or symbolic code;
solving differential equations;
performing numerical integration or diagonalization;
interpreting computational output;
returning a prescribed numerical, symbolic, or discrete result.

This scope separates calculational competence from broader research competence. The benchmark does not directly test whether an agent can identify important questions, formulate a new theory, select scientifically significant subproblems, recognize an interesting result, or redirect an investigation after failure.

The benchmark’s central research-assistance pipeline can be represented as:

$\text{physical question} \longrightarrow \text{theoretical framework} \longrightarrow \text{tractable subproblem} \longrightarrow \text{calculation} \longrightarrow \text{interpretation}.$

TheoPhysEval primarily evaluates the final calculation-oriented part of this sequence. This distinction is important when interpreting its high agent accuracies: success on a specified research calculation does not establish autonomous theoretical-physics research or breakthrough generation.

The benchmark and its Harbor configurations are distributed through the TheoPhysEval dataset repository at https://huggingface.co/datasets/RameshChandra1/TheoPhysEval (Chandra et al., 9 Oct 2026).

2. Problem-set construction and thematic coverage

The 50 problems were submitted by experts across theoretical physics and refined by moderators. Contributors were asked to provide problems satisfying three criteria:

relevance to current physics research;
solutions that are easy to verify but difficult to guess;
answers not publicly available in the literature or elsewhere at evaluation time.

Construction strategies included adapting calculations from papers while changing parameters or dimensionality, converting unpublished research notes into numerical tasks, adding numerical values to analytical problems, and requiring code, numerical integration, diagonalization, or simulation.

The thematic distribution is:

Area	Problems
Classical physics	5
Stochastic processes and biological physics	5
Nonequilibrium statistical mechanics	3
Lattice theories	3
Quantum phases of matter	5
Electronic structure theory	4
Quantum field theory	7
Particle physics	2
Scattering amplitudes	5
Holography and entanglement	5
Gravitation and cosmology	6
Total	50

The Appendix identifies problem families including boundary-layer perturbations, autonomous ordinary differential equations, inertial-particle attractors, radiation damping, stochastic ecological models, lattice correlators, fermion-parity pumps, Chern numbers, exact diagonalization, pseudopotentials, quantum-field-theory beta functions, soft limits, holographic interface transmission, black-string Page times, Euclidean wormholes, Bondi-mass fluctuations, and microstate scalar spectra.

The benchmark does not provide a systematic numerical difficulty rating for each problem or a complete accuracy breakdown by thematic area. Difficulty ranges qualitatively from short analytical derivations to large numerical and software-construction tasks.

Representative task types

TheoPhysEval includes several distinct classes of research calculations.

Literature-adaptation problems require an agent to locate a relevant paper, understand its notation and assumptions, and modify its equations. The black-string Page-time problem asks for a calculation in an $\mathrm{AdS}_4$ black-string geometry with branes at angle $\pi/10$ , using units with $T=1/(2\pi)$ . The expected workflow involves adapting equations from a related five-dimensional study, solving a singular boundary-value problem numerically, and comparing extremal areas. Agents commonly failed by using five-dimensional formulas without modifying them for the four-dimensional geometry (Chandra et al., 9 Oct 2026).

Conceptual construction problems test whether the agent can reconstruct a theoretical construction rather than merely perform numerical work. The fermion-parity-pump problem requires a two-parameter Hamiltonian, identification of the central charge at a diabolical point, and analysis of instability under free-fermion perturbations. Its intended approach uses suspension constructions, odd-parity zero-dimensional ground states, chains of copies, and interpolation between nearest-neighbor dimerizations.

Simulation problems require code and physical interpretation. The boundary-layer perturbation problem specifies $Re=160000$ and $y/L=0.01$ , then asks for the dominant initial spanwise scale of perturbation structures. The expected solution involves three-dimensional unsteady incompressible Navier–Stokes simulations, extraction of a velocity field on a horizontal plane, and analysis of normalized spanwise structure scales.

Numerical physics problems specify parameters and require numerical outputs. An axion-conversion task uses

$\operatorname{Im}\!\left[-\frac{1}{\varepsilon_L(\omega,q)}\right] = \frac{\pi}{2}\omega_p\delta(\omega-\omega_p)$

with specified values of $g_{a\gamma\gamma}$ , $\omega_p$ , $n$ , and $\mathrm{AdS}_4$ 0, and asks for $\mathrm{AdS}_4$ 1 at two axion masses. Numerical grading prevents a formally equivalent but physically incorrect symbolic expression from receiving credit.

Inverse problems ask the agent to infer a model from data or a plotted trajectory. The autonomous-ode problem specifies a trajectory and expects a fractional-delay differential equation involving Riemann–Liouville derivatives:

$\mathrm{AdS}_4$ 2

No evaluated agent solved this task. The authors acknowledge that many equations could fit the curve and that the problem is partly a deliberate play on the phrase “differential equation.”

3. Answer verification and evaluation protocol

TheoPhysEval primarily evaluates final answers rather than complete derivations. For analytical problems, numerical values are supplied for relevant parameters, including irrelevant parameters where appropriate, and agents are required to return numerical outputs. Correctness is determined by matching the grading key within problem-specific tolerances or satisfying exact discrete conditions.

Accepted answer types include:

numerical roots or eigenvalues;
coordinates of band crossings;
Chern numbers;
exact integers;
yes/no outcomes;
numerical simulation observables;
outputs in prescribed formats such as ZEROES=[z1,z2,z3].

This design makes grading objective and avoids relying extensively on automated symbolic-equivalence judgments. It also narrows what is measured. A correct final number can receive credit despite a poorly explained derivation, while a nearly correct derivation fails if its final answer is incorrect.

The initial reference solutions were checked by experts. In several cases, experts initially believed that an agent had made an error, but later consultation with contributors revealed mistakes in the reference solutions. The grading keys were then corrected. This demonstrates both the value of expert review and the fallibility of reference-answer construction.

Two convention ambiguities were identified after evaluation, involving dispersion-pi-series and iib-euclidean-wormholes. The grading key accepts convention choices observed during evaluation. The autonomous-ode problem is intrinsically difficult to grade numerically because multiple equations may fit the displayed trajectory.

TheoPhysEval therefore differs from process-oriented benchmarks that score derivations explicitly. “PhysReason” evaluates theorem application, intermediate steps, and first-error localization through its Physics Solution Auto Scoring Framework (Zhang et al., 17 Feb 2025). “PRISM-Physics” represents derivations as causal formula DAGs and uses ancestor-closure scoring (Zhao et al., 3 Oct 2025). “Lean4PHYS” evaluates whether a formalized physics theorem can be proved in Lean 4 (Li et al., 30 Oct 2025). TheoPhysEval instead emphasizes end-to-end autonomous completion of concrete research calculations.

This distinction creates several interpretive limitations:

final-answer correctness does not establish derivational validity;
a physically incorrect method may pass if it produces the correct numerical output;
a correct derivation may fail because of code-formatting or implementation errors;
intermediate assumptions, approximations, and dimensional consistency are not generally scored separately;
the benchmark does not formally certify the full reasoning chain.

4. Agents, tools, and experimental conditions

Five model–harness combinations were evaluated:

Agent	Model	Harness
Astra	GPT-6-Astra	Codex
Opus	Claude Opus 5.5	Claude Code CLI
Gemini	Gemini 3.8 Flash	Antigravity
DeepSeek	DeepSeek-V4.1-Flash	Codex
GLM	GLM 5.3	Claude Code CLI

Reasoning effort was set to the maximum available level for all models except Gemini, for which high was the highest available setting. Agents were expected to operate autonomously without human hints, intermediate corrections, method selection, or rescue after failure.

Available tools included:

code writing and execution;
numerical computation;
symbolic algebra;
differential-equation solving;
exact diagonalization;
simulation;
literature search and reading;
access to local files and supplied background papers.

This environment evaluates a composite capability:

$\mathrm{AdS}_4$ 3

Realistic and isolated tracks

The realistic track permits web search, literature access, local tools, coding, and computation. It is intended to model professional use of an AI research assistant.

The isolated track permits local tools but denies general internet access and background-paper retrieval, except for calls to the model provider. It is intended to measure internalized knowledge and the ability to reconstruct calculations without external literature search.

Isolated evaluations ran in Docker sandboxes managed by Harbor. Automated trajectory inspection found no sandbox escapes. The evaluation used Google Cloud e2-standard virtual machines with:

$\mathrm{AdS}_4$ 4

There was no explicit token or total-cost ceiling. A task was terminated after eight hours without output and counted as a failure; linked electronic-structure tasks received a total allocation of 20 hours.

Each agent was run once on the complete suite in each track. The principal metric is binary accuracy:

$\mathrm{AdS}_4$ 5

No repeated-run confidence intervals, calibration scores, or systematic runtime tables were reported.

5. Results and empirical interpretation

Overall accuracy

Realistic-track performance was:

Agent	Solved	Accuracy
Astra	49/50	98%
Opus	46/50	92%
Gemini	34/50	68%
DeepSeek	27/50	54%
GLM	21/50	42%

Isolated-track performance was:

Agent	Solved	Accuracy
Astra	46/50	92%
Opus	46/50	92%
Gemini	31/50	62%
GLM	26/50	52%
DeepSeek	25/50	50%

The strongest agent, GPT-6-Astra in Codex, achieved 98% in the realistic condition at an inferred cost of approximately $2.33 per problem, below the paper’s headline threshold of USD 3 per problem (2610.33319).

Token use and cost

Total token usage was:

Agent	Realistic	Isolated
Astra	37.41 million	24.53 million
Opus	211.18 million	228.43 million
Gemini	606.71 million	594.20 million
DeepSeek	1,025.60 million	833.97 million
GLM	971.96 million	1,115.61 million

Inferred costs were:

Agent	Realistic	Isolated
Astra	$\mathrm{AdS}_4$695.20</td> <td style="text-align: right"></td> </tr> <tr> <td>Opus</td> <td style="text-align: right">$\mathrm{AdS}_4$7275.46</td> <td style="text-align: right"></td> </tr> <tr> <td>Gemini</td> <td style="text-align: right">$\mathrm{AdS}_4$890.48</td> <td style="text-align: right"></td> </tr> <tr> <td>DeepSeek</td> <td style="text-align: right">$\mathrm{AdS}_4$936.23</td> <td style="text-align: right"></td> </tr> <tr> <td>GLM</td> <td style="text-align: right">$\pi/10$0180 total under a coding plan

The paper does not provide comprehensive runtime statistics, median task durations, or per-problem latency comparisons.

Literature access and memorization

The realistic–isolated accuracy differences were:

$\pi/10$1

These correspond to changes of $\pi/10$2, $\pi/10$3, $\pi/10$4, $\pi/10$5, and $\pi/10$6 percentage points, respectively.

The authors interpret the relatively small differences as evidence that frontier agents have internalized substantial theoretical-physics literature. Specialized problems could sometimes be solved without web access even when a human specialist would normally consult a paper to reconstruct notation or formulas.

The result does not establish that agents merely retrieve memorized answers. Many tasks required changing dimensions or parameters, adapting equations, writing new code, debugging numerical calculations, interpreting outputs, and producing prescribed numerical answers. The observed performance is compatible with several mechanisms:

memorization or internalization of literature;
genuine adaptation of known methods;
coding and numerical problem solving;
retrieval of related but inapplicable formulas;
partial reconstruction followed by computational verification.

The benchmark does not include systematic notation perturbation, hidden-paper ablations, controlled source removal, or mechanistic tests of model memory. Therefore, the memorization interpretation remains suggestive rather than conclusive.

Representative failure modes

Astra’s only realistic-track failure was autonomous-ode; it also failed three isolated problems, including two that it incorrectly judged impossible. Opus used initial data from a related paper rather than the values specified in the prompt, mischaracterized a convergence rate, exceeded its output limit on a quantum-field-theory problem, and made a modular differential-equation setup error.

Gemini, GLM, and DeepSeek commonly failed through:

applying five-dimensional black-string equations to a four-dimensional problem;
computing geodesic lengths instead of minimal areas;
treating a problem as equivalent to another when it was not;
selecting an incorrect basis of differential-equation solutions;
propagating an early sign error through linked electronic-structure tasks.

These failures demonstrate that literature retrieval alone is insufficient. Agents must verify dimensionality, conventions, initial conditions, observables, and the precise computational target.

6. Significance, limitations, and relation to autonomous research

TheoPhysEval establishes that current AI agents can perform a broad range of specialized theoretical-physics calculations with high end-to-end accuracy under a realistic tool-access condition. The strongest result is the 98% realistic-track accuracy of GPT-6-Astra in Codex on 50 research-level problems. The result is notable because the tasks span unrelated specialties and often require literature adaptation, numerical computation, code generation, debugging, and interpretation.

The benchmark nevertheless measures a narrower capability than autonomous theoretical-physics research. It does not directly evaluate:

selection of important research questions;
construction of new theoretical frameworks;
scientific taste;
interpretation of unexpected results;
assessment of novelty or significance;
sustained research planning;
generation of experimentally testable hypotheses;
productive interaction with human researchers;
recognition that a technically correct calculation is scientifically unimportant.

Dataset and statistical limitations

The benchmark contains only 50 problems, and each agent was run once per condition. Consequently, reported accuracies have substantial statistical and model-sampling uncertainty. Ignoring task dependence, a binomial approximation gives standard errors of approximately $\pi/10$ 7 for 49/50 and $\pi/10$ 8 for 21/50. The paper reports no formal confidence intervals or statistical tests.

The problem distribution reflects contributor networks and the relative ease of verifying certain calculations. It is not an exhaustive survey of theoretical physics. No systematic difficulty scores or thematic performance aggregates are supplied.

Contamination and provenance

“Unpublished” does not guarantee that a problem is absent from model training data. Potential exposure arises through:

public source papers used to construct related calculations;
contributor use of AI assistance;
internal circulation of problems;
memorization of related formulas;
subsequent publication or release of source material.

One solution, interface-transmission, appeared publicly after the experiments. The benchmark does not provide a complete contamination audit, systematic notation perturbation, or guarantee that all tasks were untouched by AI systems.

Reference-answer uncertainty

Several reference answers were corrected after expert review of agent outputs. Two problems had under-specified conventions, and the inverse autonomous differential-equation task admits multiple approximate fits. These cases show that answer-key construction remains a source of error even in research-level benchmarking.

Harness and model dependence

Results depend on the chosen model–harness pairings, reasoning settings, tool interfaces, sandbox policies, and output limits. Alternative harnesses or physics-specialized orchestration systems could produce different results. The benchmark therefore measures the performance of particular agent configurations rather than an invariant capability of the underlying models.

Process-level evaluation

Because TheoPhysEval primarily grades final answers, it does not distinguish a valid derivation from an accidental or poorly justified answer. Process-sensitive frameworks provide complementary diagnostics. “PhysReason” separates answer-level and step-level correctness and classifies first errors into theorem application, process understanding, calculation, and condition analysis (Zhang et al., 17 Feb 2025). “PRISM-Physics” scores formula dependencies through causal DAGs and ancestor closure (Zhao et al., 3 Oct 2025). “PHYBench” uses expression edit distance to give partial credit for structurally close symbolic answers (Qiu et al., 22 Apr 2025). “Lean4PHYS” provides kernel-verified proofs for formalized college-level physics statements (Li et al., 30 Oct 2025).

A fuller TheoPhysEval protocol could therefore report multiple layers:

final numerical or symbolic correctness;
dimensional and unit consistency;
intermediate-equation validity;
physical-assumption correctness;
literature-adaptation fidelity;
code and numerical reproducibility;
uncertainty and sensitivity analysis;
interpretation of the computed result;
robustness under changed parameters, notation, conventions, and dimensionality.

TheoPhysEval’s principal contribution is consequently methodological as well as empirical. It demonstrates that autonomous agents can already execute many concrete research calculations across theoretical physics, often at low monetary cost. At the same time, the benchmark clarifies the boundary between technical calculation and scientific research: strong performance on prescribed, verifiable problems does not by itself provide evidence of autonomous problem selection, theory creation, scientific judgment, or breakthrough discovery.