DiscoverPhysics: AI Scientific Discovery

• DiscoverPhysics is a benchmark protocol that evaluates scientific reasoning by requiring active experiment design and iterative hypothesis refinement in noncanonical physics environments.
• It features 22 diverse simulated worlds with unique force laws, such as dimensional anomalies and hidden interactions, to reveal latent physical structures.
• The scoring system integrates predictive accuracy with explanation quality, penalizing mere curve-fitting and emphasizing genuine mechanistic insight.

DiscoverPhysics is both a conceptual approach and a precise technical protocol for benchmarking scientific reasoning of both humans and AI agents in physics. The central motif is the active discovery of physical laws through deliberate interaction with complex, nonstandard environments—whether by experiment, simulation, or algorithmic inquiry. The core innovation is to force the agent to transcend rote recall, requiring the design of informative experiments, measurement of raw data, and iterative refinement of mechanistic hypotheses in worlds that explicitly deviate from canonical physics. This paradigm emphasizes deep engagement with unknown rules, the process of hypothesis formulation and revision, and robust explanation anchored in mathematical structure (Wiemann et al., 25 May 2026).

1. Experimental Protocol and Interactive Discovery

In DiscoverPhysics, the agent is placed in a simulated world governed by a force law unknown to the agent and often structurally distinct from familiar physics. During a session (typically up to 16 rounds), the agent alternates between:

• Proposing new experiments, expressed as configurations of test particles (initial positions $\mathbf r(0)$, velocities $\mathbf v(0)$, and charges/masses where applicable), with a measurement schedule for position and velocity at specified times.
• Receiving back noisy, multi-time trajectory data $$\left\{ \mathbf r_i(t_j),\,\mathbf v_i(t_j) \right\}$$ integrating the full $N$-body equations under the hidden law.
• Optionally submitting a candidate physical law as a Python function and indicating free parameters for system fitting, receiving back parameter fits and error metrics.
• Interpreting diagnostic feedback, updating the hypothesized mechanism, and deciding on subsequent interventions or theory revisions.

This approach differs sharply from classical textbook or question-answering benchmarks where the task is to select a correct formula or parse a proof. Instead, DiscoverPhysics quantitatively requires agents to generate the experimental data they most need, to analyze correlations, nontrivial dependencies, and structural anomalies, and to bridge observation and underlying law through hypothesis revision (Wiemann et al., 25 May 2026).

2. Simulated World Types and Underlying Physics

The 22 curated worlds in DiscoverPhysics exemplify noncanonical dynamical laws. The force structures range across:

• Dimensional anomalies: 2D gravity ($F \propto 1/r$), fractional-power exponents ($F \propto 1/r^\alpha$ with $\alpha \notin \{1,2\}$).
• Screening and range effects: Yukawa forces ($F(r) \propto e^{-r/\lambda}/r$) reflecting Helmholtz operators.
• Multi-species interactions: Particle labels encode species and response charges, producing asymmetric or even repulsive/attractive couplings not reducible to simple $1/r^2$ potentials.
• Hidden structure: Invisible "dark matter" halos, extra-dimensional Kaluza-Klein terms with force laws interpolating between $1/r$ and $\mathbf v(0)$0.
• Time-dependent/coordinate-dependent modifications: Externally imposed "wind" or time-varying coupling constants ($\mathbf v(0)$1), Hubble-type expansion terms ($\mathbf v(0)$2).
• Control variants: Standard Coulomb ($\mathbf v(0)$3) and gravity to validate baseline agent competence.

Each world is instantiated as a full $\mathbf v(0)$4-body dynamical system: $\mathbf v(0)$5 Generalized agent queries must uncover the structure of $\mathbf v(0)$6, the roles of all indices, and the relevant constants, often inferring the need for nontrivial operators (e.g., fractional Laplacians or non-conservative body forces) (Wiemann et al., 25 May 2026).

3. Benchmark Scoring: Predictive Accuracy and Explanation Quality

DiscoverPhysics performance is measured along two rigorously orthogonal axes:

• Trajectory MSE: The mean squared error (MSE) between the predicted and ground-truth trajectories of held-out probe particles across the experiment horizon, normalized by trajectory variance. A threshold of 10% of trajectory variance defines a "pass."
• Explanation Score: Agents provide a natural-language explanation of the law, which is scored on a rubric (0–10) by an LLM judge against human-written gold standards. The rubric credits correct identification of the differential operator, parameter inference (e.g., exponent, screening length), recognition of hidden variables or species, physical clarity, and role separation (source/response charges). A normalized explanation score $\mathbf v(0)$7 is required for a "pass."

An expected "pass@k" metric reports the fraction of worlds solved within $\mathbf v(0)$8 independent attempts (typically $\mathbf v(0)$9), with a "pass" defined as both MSE and explanation criteria satisfied. This dual-scoring protocol is specifically designed to penalize pure curve-fitting devoid of mechanistic insight and to highlight the gap between direct computational accuracy and conceptual scientific understanding (Wiemann et al., 25 May 2026).

4. Empirical Results and Failure Modes

Evaluation across eleven state-of-the-art LLMs (five commercial, six open-source) demonstrates that:

• Top commercial agents (Claude-4.7, GPT-5.5) solve about half of DiscoverPhysics worlds within five attempts (pass@5 ≈ 50% for Claude-4.7, 36% for GPT-5.5).
• Open-source models do not substantially outperform random search (pass@5 ≤ 10%); they fail to use experimental design or to effect iterative hypothesis revision.
• The hardest worlds are those with non-obvious or latent structure: multi-species couplings, hidden matter, extra dimensions, or nontrivial operator structure.
• Explanatory performance and predictive accuracy are largely orthogonal. GPT-5.5 often achieves lowest MSE by flexible function fitting but plateaus in explanation quality; Claude-4.7's MSE is slightly higher, but its explanations improve over rounds, reflecting more consistent hypothesis refinement and structural insight.
• Observed failure modes include "prior knowledge bias" (agents never testing the correct law family), numerical instability in stiff/singular force laws, lack of self-monitoring (failure to update/fix incorrect conceptual models), and ineffective discrimination of signal versus noise (Wiemann et al., 25 May 2026).

A plausible implication is that active discovery—especially in the presence of latent or hidden physical structure—remains a challenge for modern LLM systems, separating genuine scientific reasoning from data-centric optimization.

5. Design Principles: Scientific Discovery as Benchmark

DiscoverPhysics formalizes a set of essential elements for any benchmark intending to measure out-of-the-box scientific thinking:

• Active experimentation: The agent must design informative interventions, not merely observe passively or classify given data.
• Latent structure: The physical law should encode nontrivial, non-observable variables (species, hidden matter, higher dimensions) or require inference of operator structure.
• Iterative hypothesis refinement: Success depends critically on the agent's ability to revise its conceptual model, not just fit data, based on both new evidence and previous discrepancies.
• Decoupled scoring: Explanatory and predictive scores are computed independently to prevent curve-fitting dominance.
• No reliance on prior physics knowledge: The structure of each world and its physics is deliberately orthogonal to standard textbook examples to preclude the use of lookup or memorization.
• Open-ended final output: Agents submit both code-level implementations and textual explanations, enabling multidimensional judgment.

This protocol targets the robust measurement of both algorithmic and scientific "thinking" in the discovery of lawlike structure (Wiemann et al., 25 May 2026).

6. Broader Implications and Comparison with Human Discovery

DiscoverPhysics extends the traditional paradigm of science education and assessment. Rather than emphasizing recall or solution of prespecified problems, it operationalizes the scientific method in controlled but nontrivial simulated contexts. This is conceptually aligned with studies connecting physics learning to inquiry and model-building, e.g., laboratory/field-based approaches (Bouquet et al., 2023), and kinesthetic/experiential learning (0706.2717). However, the DiscoverPhysics framework is tailored for machine reasoning and algorithmic benchmarking, allowing automated grading, scaling to large model cohorts, and codified comparison between model and human-level capacity for real discovery, including measurement of the full process from experiment design to explanation. Critically, the observed divergences between pure numerical fitting and conceptual explanation reflect longstanding epistemological distinctions explored in the physics education literature.

A plausible implication is that such benchmarks will catalyze advances in AI systems targeting the integration of scientific reasoning, experiment design, and explanation, rather than mere data-driven prediction or answer generation. This benchmark potentially opens a quantitative pathway towards evaluating and improving genuine mechanistic understanding in artificial agents.