---
title: 'NewtonBench: Interactive Law Discovery'
url: https://www.emergentmind.com/topics/newtonbench
type: topic
---

# NewtonBench: Interactive Law Discovery

NewtonBench is a benchmark framework for evaluating scientific law discovery under conditions that are intended to be scientifically grounded, scalable, and resistant to memorization. In its principal formulation, NewtonBench comprises 324 tasks across 12 physics domains and reframes discovery as interactive model discovery rather than static function fitting: an agent must design experiments, probe a simulated model system, and recover a hidden law that has been generated by a “metaphysical shift” of a canonical physical relation. Subsequent work uses NewtonBench as an evaluation suite for active symbolic law recovery in known-variable domains, and a distinct benchmark, “NewtonBench-60K,” extends the name to physics-grounded evaluation of Newtonian motion in video generation [2510.07172] [2605.24043] [2512.00425].

## 1. Motivation and conceptual basis

NewtonBench was introduced to address what its authors describe as a methodological trilemma in benchmarking LLM-based scientific law discovery: scientific relevance, scalability, and resistance to memorization. Existing alternatives are presented as unsatisfactory in different ways. Out-of-Distribution Laws preserve relevance and resist memorization but are scarce and not scalable; Transformed Laws scale but remain vulnerable to recall; Synthetic Laws scale and avoid memorization but lose scientific grounding. NewtonBench responds by constructing benchmark tasks from counterfactual but dimensionally coherent mutations of canonical laws, thereby attempting to preserve all three desiderata simultaneously [2510.07172].

A second conceptual departure is the shift from static function fitting to interactive model discovery. In static function fitting, the agent passively receives data and regresses to a target formula. In interactive model discovery, the sought law is embedded inside a multi-equation model system, and the agent must control inputs and run schedules, elicit targeted measurements, and use assisting equations to isolate confounders. The benchmark therefore treats discovery as experiment design, hypothesis testing, and refinement rather than as supervised regression over a fixed table [2510.07172].

The mechanism that enables this construction is the “metaphysical shift.” Each canonical law is represented as an expression tree, and new tasks are generated through grammar-bounded mutations of operators, constants, couplings, and structural depth while preserving dimensional coherence by compensating constant units. The benchmark documentation gives several representative patterns: inverse-square exponents can be shifted, canonical trigonometric relations can be replaced by alternate trigonometric forms, and canonical polynomial dependencies can be replaced by nonlinear or non-integer variants. This design is explicitly intended to force inference from experimental evidence rather than retrieval of memorized textbook formulas [2510.07172].

## 2. Task construction and interactive protocol

NewtonBench comprises 324 tasks derived from 108 shifted equations across 12 domains, each instantiated in three system complexities: Vanilla Equation, Simple System, and Complex System. The 12 domains are gravitation, electrostatics, magnetostatics, thermal conduction, geometrical optics, nuclear physics, oscillations, physical optics, acoustics, elasticity, statistical mechanics, and calorimetry. Equation difficulty tiers are labeled easy, medium, and hard, and are defined by cumulative mutation count and structural complexity, while system difficulty varies independently via the depth and confounding degree of the model [2510.07172].

The three task configurations define progressively richer discovery settings:

- **Vanilla Equation**: the target law is the only equation.
- **Simple System**: the target law feeds into one assisting equation.
- **Complex System**: multi-equation pipelines introduce confounders that must be isolated through assisting equations.

The benchmark environment is interactive. Agents may propose experiments via `<run_experiment>` with a JSON array of parameter assignments to system-level inputs, optionally use `<python>` in the code-assisted setting, and submit a final symbolic law via `<final_law>`. The protocol allows up to 10 rounds, one action per round, and at most 20 parameter sets per experiment round. Model execution is deterministic and noiseless by default, with outputs computed by an ordered sequence of equations rather than by implicit ODE integration. Controlled Gaussian additive observation noise can also be injected, with studied levels of $\sigma = 10^{-4}, 10^{-3}, 10^{-2}, 10^{-1}$ [2510.07172].

The target law is a scalar-valued equation represented as an AST over a fixed operator set and input variables. Agents submit a Python function `discovered_law(...)` encoding the hypothesis. Symbolic equivalence is defined at the level of functional form rather than literal syntax: variable renaming and constant rescaling that preserve the structural law are ignored. This makes the benchmark a test of mechanistic recovery rather than string matching [2510.07172].

## 3. Evaluation methodology and empirical profile

The principal NewtonBench paper evaluates 11 LLM agents in two settings, Vanilla Agent and Agent with Code Assistance. Symbolic accuracy is the primary metric and is defined as
$$
\mathrm{Accuracy}_{\mathrm{sym}} = \mathbb{I}\big(\mathrm{equiv}(\hat{f}, f_{\text{target}})\big).
$$
Equivalence is adjudicated by an LLM-as-a-judge protocol with 98.3% agreement with human experts. Predictive fidelity is measured by RMSLE over 5,000 held-out samples per task:
$$
\mathrm{RMSLE} = \sqrt{\frac{1}{N}\sum_{i=1}^{N}\left(\log(\hat{y}_i+1)-\log(y_i+1)\right)^2}.
$$
RMSLE is computed after outlier filtering via modified Z-scores [2510.07172].

The reported results characterize current capability as “clear but fragile.” In the Vanilla Agent setting, GPT-5 achieves average symbolic accuracy of 72.9% with RMSLE 0.4525, while Gemini-2.5-pro reaches 65.0% with RMSLE 0.4697. Performance degrades sharply with both equation and system complexity. For GPT-5, symbolic accuracy falls from 94.4% on simple/easy tasks to 29.9% on complex/hard tasks; for Gemini-2.5-pro, the corresponding drop is from 90.3% to 13.9%. Non-reasoning models are substantially weaker: GPT-4.1-mini attains approximately 4.6% symbolic accuracy in the vanilla setting, and DeepSeek-V3 approximately 9.7% [2510.07172].

The benchmark also exposes two major fragilities. First, symbolic discovery is highly sensitive to even small observation noise: on GPT-5-mini, Gaussian noise with $\sigma = 10^{-4}$ causes a 13–15% drop in symbolic accuracy, and further increases in noise produce proportional declines, while RMSLE remains relatively stable. Second, tool assistance has a paradoxical effect. Weaker models benefit from code access, but stronger models exhibit slight degradation; for example, GPT-5 drops from 72.9% to 69.6%, and Gemini-2.5-pro from 65.0% to 62.0%. The reported interpretation is that code can induce a premature shift from exploration to exploitation, with stronger models allocating more code use to function fitting and thereby satisficing on locally adequate but structurally incorrect hypotheses [2510.07172].

Cross-domain performance is heterogeneous. Average accuracies are reported as approximately 54% for Acoustic Velocity, 50% for Ampère’s law, 49% for Gravitation, 43% for Snell’s law, and 18% for Bose–Einstein. Complexity amplifies these disparities: Fourier’s Law reaches approximately 58% in simple settings but approximately 2% in complex settings, while Acoustic Velocity remains comparatively robust at approximately 45% in complex settings. A qualitative calorimetry example illustrates the intended mode of use: the agent varies $m$, $c$, and $\Delta T$ over orders of magnitude, infers heat from assisting equations, estimates a non-integer exponent by log–log analysis, and successfully recovers the shifted law $Q = m\,c^{2.5}\,\Delta T$ [2510.07172].

## 4. NewtonBench in active symbolic law recovery

A later line of work uses NewtonBench to evaluate active, closed-form symbolic law discovery when the relevant physical variables are known. In that setting, NewtonBench is described as probing an agent’s ability to recover memorization-resistant counterfactual laws via interactive querying of a noiseless oracle, with emphasis on symbolic identifiability rather than passive regression. The evaluation uses the `vanilla_equation` setting across 12 physics domains: Newtonian gravitation, Coulomb force, magnetic force, Fourier heat conduction, Snell’s law, radioactive decay, underdamped harmonic motion, Malus’s law, speed of sound, Hooke’s law, Bose–Einstein distribution, and heat transfer. The study runs 12 domains $\times$ 3 difficulty levels $\times$ 3 law versions $\times$ 3 seeds, for 324 runs total, with fixed-budget comparisons at $B = 20$ and extended studies up to $B = 100$ [2605.24043].

The formal task is a budget-constrained sequential design problem. The oracle is a noiseless black-box function $u \mapsto f_{\text{target}}(u)$ over an open set of achievable target-input values. Let $\mathcal{M}$ denote the space of candidate mechanisms, each $m \in \mathcal{M}$ defining a predictive mapping $f_m : \mathcal{X} \to \mathcal{Y}$. At round $t$, the learner selects an experiment $x_t \in \mathcal{X}$ and observes
$$
y \sim p(\cdot \mid x, m^\star),
$$
accumulating
$$
D_t = \{(x_i, y_i)\}_{i=1}^t
$$
under fixed budget $B$. The discovery state is
$$
S_t = (D_t, E_t, H_t),
$$
with memory $E_t$ and hypothesis set $H_t = \{m^{(k)}\}_{k=1}^K \subset \mathcal{M}$, and the objective is to minimize
$$
\mathbb{E}[\mathcal{L}(\hat{m}_B, m^\star)].
$$
Symbolic accuracy is judged by an LLM evaluator that asks whether the predicted expression can be made equivalent to the ground-truth expression by choosing constant parameter values and allowing algebraic rewriting and variable renaming; predictive error is measured by RMSLE, and exact numerical recovery is defined by $\mathrm{RMSLE} < 0.01$ [2605.24043].

The method evaluated in this setting, LLM-AutoSciLab, iterates hypothesis generation, hypothesis-conditioned experiment selection, and mechanism refinement. Candidate symbolic mechanisms are sampled with a smaller LLM, clustered by structural skeleton, and synthesized by a larger LLM into primary and alternate hypotheses. Acquisition is driven by predicted disagreement,
$$
\Delta(x) = \mathrm{Std}\big(\log_{10}\hat{f}_1(x), \ldots, \log_{10}\hat{f}_K(x)\big),
$$
and a bootstrap confidence score
$$
\mathrm{conf}_{\mathrm{boot}} = 1 - \frac{1}{N}\sum_i \frac{\mathrm{std}_b(\hat{y}^{(b)}_i)}{|\mathrm{mean}_b(\hat{y}^{(b)}_i)| + \epsilon},
$$
clipped to $[0,1]$, gates transitions between disambiguation and refinement. Refinement uses PySR with 800 iterations per fitting call and direct numerical fitting of candidate skeletons when available; GPT-4o-mini is the primary LLM and Qwen/Qwen2.5-7B-Instruct is used for ensemble generation [2605.24043].

The quantitative results at $B = 20$ are as follows:

| Method | SA | Exact / RMSLE |
|---|---:|---:|
| LLM-AutoSciLab | 67.60% | 81.50% / 0.150 |
| PySR | 24.07% | 74.54% / 0.182 |
| Bayesian Optimization | 24.54% | 68.52% / 0.334 |
| BED | 11.11% | 63.86% / 0.577 |
| LLM-only | 6.48% | 7.41% / 5.039 |
| Code-assisted LLM | 7.41% | 10.19% / 4.912 |

Difficulty-specific performance for LLM-AutoSciLab is 79.20% symbolic accuracy on easy tasks, 72.20% on medium, and 51.40% on hard, with corresponding exact accuracies of 93.10%, 84.70%, and 66.70%, and RMSLE of 0.018, 0.040, and 0.404. The study reports that the strongest active baselines require 2.60–3.10$\times$ more queries to match LLM-AutoSciLab’s fixed-budget performance on NewtonBench. The reported interpretation is that fit-driven baselines can achieve relatively high numerical fidelity while failing to recover the true symbolic mechanism, whereas hypothesis-conditioned acquisition directly targets structural disambiguation [2605.24043].

## 5. NewtonBench-60K and physics-aware video generation

A distinct benchmark, “NewtonBench-60K,” applies the NewtonBench name to video generation rather than symbolic law discovery. It is a controlled, large-scale benchmark built to test whether video generators produce not just visually plausible motion, but motion that respects Newtonian structure. The benchmark focuses on canonical regimes with time-invariant external forces acting on a single rigid body under gravity or ramp contact, and it is paired with a post-training method called NewtonRewards [2512.00425].

NewtonBench-60K contains 60,000 short videos: a training set of 50,000 simulated clips, with 10,000 per Newtonian Motion Primitive, and a held-out benchmark of 10,000 clips, with 2,000 per primitive split evenly into 1,000 in-distribution and 1,000 out-of-distribution samples. Each video is 32 frames at 16 fps and 512×512 resolution under HDRI lighting, with a fixed side-view camera and static background. The five primitives are free fall, horizontal throw, parabolic throw, ramp sliding down, and ramp sliding up. Their expected dynamics are specified by constant-acceleration kinematics, projectile equations under gravity, and ramp accelerations with kinetic friction, including
$$
a_x = 0,\quad a_y = -g
$$
for ballistic motion and
$$
a_{\text{down}} = g\sin\theta - \mu g\cos\theta,\quad
a_{\text{up}} = -g\sin\theta - \mu g\cos\theta
$$
for ramp motion [2512.00425].

The benchmark supplies RGB frames, instance masks, depth maps, and metadata including camera parameters and sampled physical parameters such as initial height, launch speed and angle, ramp angle, and friction coefficients. Simulation uses Kubric for scene orchestration, PyBullet for rigid-body dynamics, and Blender for rendering, with objects sampled from the Google Scanned Objects dataset. For generated videos, where renderer masks are unavailable, object masks are extracted with SAM2 [2512.00425].

Evaluation combines physics metrics and standard visual metrics. Using centroids extracted from masks, image-plane velocity and acceleration are defined by first and second discrete derivatives:
$$
\mathbf{v}_t = \frac{\mathbf{c}_{t+1}-\mathbf{c}_t}{\Delta t},\qquad
\mathbf{a}_t = \frac{\mathbf{c}_{t+2}-2\mathbf{c}_{t+1}+\mathbf{c}_t}{\Delta t^2}.
$$
The resulting errors are $\mathrm{RMSE}_{\mathbf{v}}$ and $\mathrm{RMSE}_{\mathbf{a}}$. Visual metrics are trajectory position error, Chamfer Distance, and IoU. Physics verification also uses measurable proxies from frozen utility models: RAFT optical flow as a velocity proxy and V-JEPA 2 embeddings as a mass-related appearance proxy. These define a discrete constant-acceleration residual,
$$
\boldsymbol{\phi}_{t+1} - 2\boldsymbol{\phi}_t + \boldsymbol{\phi}_{t-1} \approx \mathbf{0},
$$
a mass reward,
$$
\mathcal{R}_{\text{mass}} = \frac{1}{T}\sum_{t=0}^{T-1}\left\|\mathbf{z}^{\text{gen}}_t - \mathbf{z}^{\text{sim}}_t\right\|_2^2,
$$
and a combined objective
$$
\mathcal{L}_{\text{phys}} = \lambda_{\text{kinematic}}\,\mathcal{R}_{\text{kinematic}} + \lambda_{\text{mass}}\,\mathcal{R}_{\text{mass}}.
$$
The benchmark is therefore designed to verify Newtonian structure directly, rather than only appearance similarity [2512.00425].

All methods in the study start from the same supervised fine-tuning of OpenSora v1.2 on the 50K training subset. On the 5K in-distribution benchmark, NewtonRewards improves over SFT across all reported visual and physics metrics: L2 0.0962, CD 0.2930, IoU 0.1266, $\mathrm{RMSE}_{\mathbf{v}}$ 0.2628, and $\mathrm{RMSE}_{\mathbf{a}}$ 3.0432, with average change +9.75%. On the 5K out-of-distribution benchmark, it again improves across all metrics: L2 0.1206, CD 0.3780, IoU 0.1025, $\mathrm{RMSE}_{\mathbf{v}}$ 0.3816, and $\mathrm{RMSE}_{\mathbf{a}}$ 5.1561, with average change +8.60%. Residual maps of $\boldsymbol{\phi}_{t+1} - 2\boldsymbol{\phi}_t + \boldsymbol{\phi}_{t-1}$ are reported to have the smallest magnitudes for NewtonRewards, indicating the closest adherence to constant-acceleration dynamics [2512.00425].

## 6. Limitations, misconceptions, and significance

A recurrent misconception is to treat NewtonBench as a passive symbolic regression dataset. In its original law-discovery form, the benchmark is interactive: agents choose experiments, receive structured outputs, and must isolate hidden principles embedded in model systems. In the LLM-AutoSciLab setting, the benchmark is explicitly used for active, closed-form symbolic law discovery under a query budget, with evaluation focused on symbolic identifiability rather than only predictive accuracy. In NewtonBench-60K, the benchmark no longer concerns symbolic formulas at all, but instead tests whether generated videos satisfy constant-acceleration structure and related Newtonian constraints [2510.07172] [2605.24043] [2512.00425].

The limitations reported across these formulations are correspondingly different. For the original interactive benchmark, the current tasks are deterministic algebraic pipelines and do not yet include multi-physics couplings, PDE-constrained dynamics, or real experimental apparatus; future directions include richer noise models, temporal processes, calibrated simulators, and robotic-lab extensions. For the LLM-AutoSciLab use of NewtonBench, simulator-based oracles omit physical-lab noise, failures, costs, and operational constraints, while performance depends on LLM hypothesis quality and on parser and refinement coverage; symbolic recovery can be fragile under high noise or extreme budget constraints. For NewtonBench-60K, the scope is restricted to single-object, time-invariant-force regimes with a static camera and weak perspective, while collisions, multi-body interactions, fluid and elastic dynamics, nonstationary forces, and camera motion remain out of scope; proxy extraction and segmentation can also fail [2510.07172] [2605.24043] [2512.00425].

Taken together, these works suggest that “NewtonBench” now denotes a family of benchmark constructions rather than a single artifact. What unifies them is not a shared implementation but a shared evaluative emphasis: recovery or verification of Newtonian or law-like structure under conditions where naive memorization, appearance-based scoring, or purely local curve fitting are insufficient. In that sense, NewtonBench has become a technical label for benchmarks that elevate experiment design, mechanistic identifiability, and physically constrained evaluation to first-class objectives in AI-for-science and physics-aware generative modeling [2510.07172]

Source: https://www.emergentmind.com/topics/newtonbench