TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solving

Abstract: We present a suite of 50 unpublished research-level theoretical physics problems with verifiable answers. Without AI assistance, a typical problem in our collection could be solved by a world expert in about a day or less but would take considerably longer for experts in adjacent specialties. We test the ability of off-the-shelf AI agents to tackle these problems autonomously, without human hints. When given access to tools available in a realistic setting, including the ability to search the literature and write scripts for numerical and symbolic work, the best-performing agent, GPT-6-Astra harnessed in Codex, obtains the final answers with a near-perfect accuracy of 98% at an average cost under USD 3 per problem. On average, agents show only small changes in accuracy when denied web search and access to background research papers, suggesting significant memorization of the literature. These results suggest that state-of-the-art agents can accurately perform concrete calculations required across theoretical physics research. We comment on the broader implications of these findings and discuss possible explanations for the current lack of agent-led breakthroughs in theoretical physics despite their outstanding problem-solving abilities.

• The paper demonstrates that advanced AI language models can solve university-grade theoretical physics problems with accuracy comparable to or exceeding that of domain experts even without web access.
• The benchmark, consisting of 50 unique problems, highlights AI models' ability to recall and adapt physical formulars and problem-solving methods from literature which are not provided in prompts.
• Despite these advancements, the authors note that AI still lacks the 'taste' for problem selection and form construction to autonomously research, but its performance does represent the potential of augmenting theoretical physics research.

Scope and objective

“TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solving” (2610.16733) evaluates whether contemporary tool-using AI agents can autonomously solve concrete, research-level theoretical-physics problems. Its target is not theory construction, problem selection, or open-ended scientific discovery, but the calculational stage at which a researcher has a well-posed problem, known techniques, and a verifiable symbolic or numerical answer.

The benchmark contains 50 unpublished problems contributed by specialists across 11 areas: classical physics; stochastic and biological physics; nonequilibrium statistical mechanics; lattice theories; quantum phases of matter; electronic-structure theory; quantum field theory; particle physics; scattering amplitudes; holography and entanglement; and gravitation and cosmology. The problems are deliberately “turn-the-crank” research problems: they are novel in their precise parameter choices or formulations, and some could be publishable, but none is intended to constitute a major open problem or require a new physical theory.

The central empirical claim is strong but narrowly defined: frontier agents can solve a broad collection of specialized, verifiable theoretical-physics calculations with accuracy comparable to that of domain experts, even when the agents operate without human hints. The paper explicitly distinguishes this ability from autonomous scientific research in the broader sense.

Benchmark construction and problem characteristics

The problem-selection criteria impose three constraints. Each problem must concern current research, possess an answer that is difficult to guess but straightforward to verify, and have no publicly available solution at the time of evaluation. To satisfy these requirements, contributors commonly modified calculations from recent work by changing parameters, dimensions, boundary conditions, or other details. This construction preserves the relevant physics while preventing direct retrieval of the final answer.

The resulting tasks span a substantial range of computational modalities. Some require literature reconstruction followed by analytic adaptation; others require numerical integration, symbolic manipulation, Monte Carlo or exact diagonalization, electronic-structure calculations, differential-equation solvers, or code generation. For example, the black-string Page-time problem requires an agent to locate a related holographic calculation, adapt it from five to four dimensions, formulate a singular boundary-value problem, and solve it numerically. The electronic-structure sequence requires an all-electron atomic solver, norm-conserving pseudopotentials, Kleinman–Bylander separabilization, ghost-state analysis, and a plane-wave SCF calculation.

This design tests more than isolated mathematical competence. A successful trajectory must identify the relevant physical formalism, reconstruct missing notation from the literature, choose a numerically stable method, write and execute code, inspect outputs, and extract a specified numerical answer. The benchmark therefore evaluates an agent–harness system rather than a LLM in isolation.

The authors also acknowledge that the “unpublished” criterion is imperfect. Some human contributors used AI assistance while preparing their problems, and the authors cannot exclude data leakage. One problem was published after the experiments were completed. Consequently, the benchmark establishes that agents solve the supplied tasks under the stated conditions; it does not establish that every task was wholly absent from model pretraining or indirect exposure.

Evaluation protocol

The experiments use five agents paired with coding and terminal harnesses:

Agent system	Realistic track	Isolated track	Realistic cost
GPT-6-Astra with Codex	49/50, 98%	46/50, 92%	USD 116.28
Opus 5.5 with Claude Code CLI	46/50, 92%	46/50, 92%	USD 240.43
Gemini 3.8 Flash with Antigravity	34/50, 68%	31/50, 62%	USD 95.01
DeepSeek-V4.1-Flash with Codex	27/50, 54%	25/50, 50%	USD 37.42
GLM 5.3 with Claude Code CLI	21/50, 42%	26/50, 52%	USD 311.71 nominally

The realistic track permits web search and local computational tools. The isolated track permits local tools but disables web access apart from provider-internal calls. Both tracks use containerized execution under the Harbor framework, with two virtual CPUs, 4 GB of memory, and 10 GB of storage per task. Tasks are terminated after eight hours, except for the linked electronic-structure problems, which receive a total allocation of 20 hours.

The use of numerical answer targets is an important methodological choice. Symbolic equivalence is difficult to judge reliably because physically identical expressions can differ in notation and algebraic form. The benchmark therefore supplies numerical parameter values, including irrelevant quantities where necessary, and grades numerical outputs. This reduces dependence on an LLM judge but introduces sensitivity to conventions, numerical tolerances, and ambiguities in the prompts. The authors identify two problems with incompletely specified conventions and note that the autonomous differential-equation problem is not fully amenable to numerical answer conversion.

The runs were performed once per agent and track. Since agent outputs are stochastic, the reported pass rates lack repeated-trial confidence intervals. The authors explicitly caution that differences of a few percentage points may not be statistically meaningful.

Quantitative results

The realistic-track results show a pronounced separation between the two leading systems and the remaining agents.

Figure 1: Pass rates on the realistic track, where agents have access to web search and local computational tools.

Astra solves 49 of 50 problems, corresponding to 98%, and Opus solves 46, corresponding to 92%. Gemini reaches 68%, while the open-weight systems reach 54% and 42%. The best system therefore misses only one task despite the breadth of the benchmark, and does so at an average cost below USD 3 per problem. The implication is operational rather than merely comparative: for many concrete research calculations, a frontier agent can provide a low-cost first-pass solution that is sufficiently accurate to be useful to a specialist.

The isolated-track results are unexpectedly close to the realistic-track results.

Figure 2: Pass rates on the isolated track without general web access.

Astra and Opus each solve 46 of 50 problems, or 92%. Gemini solves 31, DeepSeek 25, and GLM 26. The leading systems therefore lose little accuracy when web access is removed. For Astra, the decrease is six percentage points; for Opus, it is zero. The paper interprets this as evidence that frontier models have memorized substantial portions of the theoretical-physics literature, including formulas and methodological structures that are not provided in the prompt.

That interpretation is plausible but not uniquely established by the experiment. Isolated performance may reflect memorization, broad latent representations, strong reconstruction ability, or a combination of these. The benchmark does not directly distinguish retrieval of memorized formulas from independent derivation. Nevertheless, the result is consequential: web search appears to contribute less to the leading agents’ accuracy than expected, even on tasks whose human solution would normally begin by locating a specific background paper.

Token usage varies substantially across systems. Astra consumes approximately 37.4 million total tokens on the realistic track and 24.5 million on the isolated track, whereas Opus consumes 211.2 million and 228.4 million, respectively. Gemini, DeepSeek, and GLM consume between approximately 600 million and 1.1 billion tokens per track. Thus, pass rate alone does not characterize efficiency. Astra dominates both accuracy and token economy in this evaluation, while Opus achieves nearly the same accuracy at considerably higher cost.

Error analysis and what the agents actually solve

Astra’s sole realistic-track failure is the autonomous differential-equation task. This is an inverse problem in which the agent must infer an autonomous equation from a plotted trajectory under constraints on derivative order, term count, and coefficient values. The supplied solution uses fractional derivatives and a delay term:

Figure 3: Numerical solution of the differential equation used to fit the supplied trajectory.

The task is difficult to grade by a unique numerical target because multiple equations can produce comparably good fits. More importantly, it contains information asymmetry: the benchmark creator knows the synthetic construction of the trajectory, whereas the agent sees only the resulting curve. Every evaluated model fails this problem. This establishes a specific boundary of the benchmarked capability: agents are highly effective at forward calculations with well-defined physical structure but unreliable at reverse-engineering an underspecified generative mechanism.

Astra’s three isolated-track failures include two cases in which it incorrectly rejects solvable problems as impossible. Conversely, several initial apparent failures were discovered to be errors in the benchmark’s reference solutions. After expert review, Astra’s answers were judged correct. This episode demonstrates both the value of expert adjudication and the vulnerability of automated benchmark construction: even research specialists can introduce subtle reference-answer errors in technically heterogeneous tasks.

Opus exhibits a different failure profile. It sometimes imports formulas from a relevant paper without adapting them to the altered prompt. In the inertial-particle problem, it uses initial data from the cited source rather than the initial data specified by the benchmark. In the black-string Page-time problem, Gemini and GLM similarly use five-dimensional formulas without performing the required four-dimensional modification. In the Schur-index MLDE task, Opus makes a small modular-differential-equation setup error in the isolated track. These are not failures of surface fluency; they are failures of problem-specific control over assumptions and dimensional dependence.

The weaker systems make more conspicuous methodological errors. For the black-string task, some agents compute geodesic lengths instead of minimal areas. For the microstate scalar-spectrum problem, agents conflate the requested problem with a related one, choose an incorrect basis of radial solutions, or fail to produce an answer. In the linked electronic-structure tasks, an early sign error propagates through later calculations, illustrating the fragility of long computational trajectories even when individual subroutines are straightforward.

The qualitative results therefore qualify the headline pass rates. A correct final number can arise from a robust derivation, a successful numerical computation, memorized formulas, or an accidental cancellation. Conversely, an incorrect answer may reflect a single convention error rather than a wholesale inability to perform the physics. Since grading is based on final answers, the benchmark measures task completion rather than the reliability, interpretability, or pedagogical quality of the reasoning trajectory.

Interpretation for theoretical-physics research

The authors argue that the breadth of the best agents exceeds that of any individual physicist, while their calculation-level performance approaches that of world experts in specialized domains. This claim is supported by the cross-disciplinary composition of the benchmark: no single researcher would normally be equipped to attempt all 50 problems, and many experts would be unfamiliar with the terminology and computational methods required outside their subfield.

The paper’s most important conceptual distinction is between calculation and research direction. A theoretical-physics project typically involves at least three stages: identifying an interesting question, formulating it within an appropriate theory, and carrying out the resulting calculation. TheoPhysEval concentrates on the third stage. Its results show that agents can often execute that stage autonomously, but they do not demonstrate that agents can reliably select important questions, recognize when an initial problem is ill-posed, redirect a project toward a simpler but scientifically meaningful alternative, or assess whether a result is conceptually significant.

This limitation directly constrains the paper’s broader claim. High performance on verifiable research calculations does not imply autonomous theoretical-physics research. The authors report that current agents lack reliable “taste” in problem selection and tend to emphasize technically elaborate but scientifically unimportant details when asked to generate problems. These observations are based on experience rather than controlled measurement, and the paper correctly presents them as qualitative judgments rather than benchmarked conclusions.

The results nonetheless have immediate implications for research practice. Problems traditionally assigned to beginning researchers as incremental exercises may no longer reliably serve as independent research projects if the objective is to obtain a novel calculation. They may remain valuable pedagogically, but their function would shift toward training in formulation, validation, interpretation, and critical assessment rather than simply obtaining the final number. The paper also raises an access issue: if advanced research requires increasingly expensive agentic inference, computational resources may become a larger determinant of research capacity even in a field historically characterized by relatively low material costs.

Limitations and open questions

The benchmark has several limitations that bear directly on the interpretation of its numerical results. First, the evaluation suite is not adversarial, and its authors intentionally retain problems that agents can solve. It is therefore a sample of realistic calculational tasks, not a worst-case test of robustness. The universal failure on the inverse differential-equation task shows that alternative task constructions could produce substantially lower scores.

Second, the sample of 50 problems is not statistically representative of theoretical physics as a whole. Its thematic distribution reflects contributor networks and the fact that some fields naturally yield verifiable calculations more readily than others. The benchmark is also dominated by problems with numerical or otherwise objectively checkable outputs, which may underrepresent conceptual, interpretive, and model-building work.

Third, each agent is paired with a particular harness, and the pairings are not systematically crossed. Performance differences therefore conflate model capability with tool orchestration, prompting, runtime behavior, and coding-interface quality. The study also uses a single stochastic run per agent and track, preventing precise uncertainty estimates.

Fourth, the isolated-track result does not by itself prove literature memorization. A controlled contamination study, with newly generated problems structurally related to but absent from the training distribution, would be needed to separate memorization from generalization. Similarly, final-answer grading cannot establish whether an agent’s intermediate derivation is physically valid or whether it would survive expert scrutiny under perturbations of the input.

Finally, the benchmark does not evaluate iterative human–agent collaboration, where a physicist inspects intermediate results, supplies corrections, and requests targeted refinements. The paper suggests that such interaction would improve weaker systems, but this remains an empirical question. Other open questions include whether scientific-specific harnesses improve robustness, whether repeated sampling and self-verification reduce subtle errors, and whether agents can reliably recognize when a formally correct calculation answers an uninteresting or incorrectly posed question.

Conclusion

TheoPhysEval (2610.16733) presents a demanding evaluation of autonomous agentic calculation across 50 unpublished research-level theoretical-physics problems. GPT-6-Astra achieves 98% with web access and 92% without it, while Opus reaches 92% on both tracks. The small realistic–isolated gap supports the paper’s claim that frontier systems possess extensive internalized knowledge of theoretical-physics methods and literature.

The evidence supports a precise conclusion: leading AI agents can already perform a large class of specialized, verifiable calculations at high accuracy and low marginal cost. It does not establish autonomous theory construction, problem selection, scientific judgment, or reliable project-level research. The benchmark therefore identifies a substantial advance in computational execution while preserving a clear distinction between solving a specified physics problem and deciding which physics problem should be solved.