PACE-Bench: Benchmarking Physics Adaptation via Code Evolution in Dynamic Environments

Abstract: Self-evolving agents improve future behavior from interaction experience, yet existing evaluations typically optimize under fixed execution conditions and do not test recovery after those conditions change. To address this gap, we introduce PACE-Bench (Physics Adaptation via Code Evolution), a simulator-grounded benchmark of 144 source-to-target adaptation pairs across six physics domains. Each pair links a source environment to a mutated target environment with the same goal and interface. A code-driven design that succeeds in the source fails in the target, where agents must iteratively adapt it into a working target design using diagnostic sandbox feedback within a limited attempt budget. We compare ten self-evolving methods from four paradigms. The benchmark remains far from saturated: Reflexion + Qwen3-14B succeeds on only 35.9\% of full-benchmark pairs, while GPT-5.5 solves 66.7\% of the Statics subset under the full budget. Together, these results show that simulator-grounded reflection is more reliable than unverified self-revision, while memory anchors agents to early designs and broad tree search explores without converging. Even revealing exact physical changes does not raise the performance ceiling, pointing to mechanism redesign rather than parameter inference as the central bottleneck. Data and code are available at https://github.com/thunlp/PACE-Bench.

• The paper introduces PACE-Bench, a simulator-grounded benchmark of 144 source-to-target adaptation pairs across 36 physics tasks, four mutation stages, and six domains, evaluated with executable Python designs and diagnostic feedback.
• The paper finds that Reflexion with Qwen3-14B achieves the strongest full-benchmark result at 35.9% Pass@2, while GPT-5.5 reaches 66.7% on Statics, showing that dynamic physics adaptation remains unsolved even for advanced models.
• The paper shows that revealing changed parameters does not overcome the main challenge, while conservative memory methods risk design fixation and radical search methods stagnate, indicating that agents need better mechanism redesign rather than more unverified revisions.

Motivation and problem setting

Self-evolving agents improve future behavior by updating parameters, context, memory, or tools from interaction experience, but existing evaluations optimize under fixed execution conditions: they measure improvement over task streams, knowledge transfer to related tasks, or generalization of prompt/code/harness updates to held-out tasks. None of them tests whether an agent can repair a previously successful design after the execution environment itself changes while the goal and interface remain fixed. PACE-Bench addresses this gap with a simulator-grounded benchmark in which a code-driven physical design that succeeds in a source environment fails after systematic physics mutations, and the agent must iteratively adapt it using diagnostic sandbox feedback within a bounded attempt budget (2608.14441).

Benchmark design

Each of 36 base tasks spans six physics domains (Statics and Equilibrium, Kinematics and Linkages, Dynamics and Energy, Granular/Fluid Interaction, Cybernetics and Control, Exotic Physics) and defines five environments sharing one context template $\mathcal{C}$: a source environment $\mathcal{E}_0$ and four mutated targets $\mathcal{E}_1$–$\mathcal{E}_4$, yielding $36 \times 4 = 144$ source-to-target pairs. A solution is a Python program defining build_agent() (structure assembly) and agent_action() (control logic), executed in a Box2D sandbox that returns constraint satisfaction $v(x)$, task score $s(x)$, and a diagnostic report $d(x)$ reporting what failed and by what margin, without prescribing fixes.

The evaluation protocol enforces two conditions per pair: the target is solvable ($\mathcal{X}_k \neq \emptyset$), and the verified source solution fails on the target ($x_0 \in \mathcal{X}_0 \setminus \mathcal{X}_k$). Agents receive a Uniform Suffix listing variables that might differ across stages without identifying which changed, and have $\mathcal{E}_0$0 attempts; the final score is the best over attempts. The protocol thus measures both physical inference ("know what": diagnosing hidden parameter changes) and mechanism redesign ("know how": revising structure and control).

Construction proceeds in three phases. Reference solutions for all environments are derived via iterative refinement with Claude Code; automated audits enforce cross-module consistency, scrub numeric values of invisible physics variables from prompts, and verify API exclusivity. Difficulty is calibrated so Qwen3-4B fails two independent runs while the target remains solvable, with diagnostic feedback enriched only if it does not enable the model to pass. Two authors with graduate-level physics or engineering training audited all 36 tasks over three passes, reducing moderate-to-critical issues from roughly 86% of tasks to none. On average a prompt contains 1,051 tokens, 7 hard constraints, and 8 primitive APIs, with mutated parameters escalating from 2 at Stage 1 to 10 at Stage 4 (945 stage-level mutations total across 180 environments).

Experimental setup

The full benchmark evaluates Vanilla iterative refinement plus ten self-evolving methods from four paradigms — Context-based (Reflexion, Self-Refine), Memory-augmented (ACE, ExpeL, ReasoningBank), Inference-time Search (Tree-of-Thoughts, CodeEvolve), and Parameter-based (SEAL, RAGEN, TTT-Discover) — on Qwen3-4B/8B/14B, with two independent runs per pair. A full-budget study on the Statics subset adds Qwen3-32B, DeepSeek-V4-Pro, and GPT-5.5; a separate five-attempt study covers six frontier models. Metrics are Pass@2 (at least one run succeeds) and Score@2 (mean of run-best scores). Run-reliability analysis supports the two-run protocol: 86.6% pass/fail agreement between runs, zero median absolute score difference, and Spearman $\mathcal{E}_0$1 up to 0.976 at 14B.

Headline results

The benchmark is far from saturated. The strongest full-benchmark configuration, Reflexion + Qwen3-14B, reaches only 35.9% Pass@2, and even GPT-5.5 solves just 66.7% of Statics pairs under the full 20-attempt budget (Score@2 78.1). Within-family scaling plateaus: Qwen3-32B does not improve Pass@2 over Qwen3-14B, whereas DeepSeek-V4-Pro (45.8%) and GPT-5.5 do. From-scratch construction remains non-trivial (Pass@2 11.3–18.3% across scales), and success drops sharply from the source environment (32.3%) to mutated targets (6.4–13.4%), confirming that the mutations create substantially harder targets.

Paradigm-level patterns are pronounced:

Paradigm	Key finding
Context-based	Reflexion leads overall (+4–11 points over Vanilla); Self-Refine never exceeds 7.1%
Memory-augmented	ACE/ReasoningBank beat Vanilla at 8B but fall below it at 14B (25.0%, 24.2% vs. 32.0%)
Inference-time search	ToT doubles Vanilla at 4B but erodes at scale; CodeEvolve degrades monotonically (10.7% → 5.3%)
Parameter-based	All three trail Vanilla at 14B; RAGEN's deficit widens from −3.8 to −17.5 points

Cost-normalized comparison on Kinematics shows ToT as the efficiency leader at every scale (45.7–57.8 Score/Hr), while Self-Refine's unverified inner loop yields only 2.0–3.6 Score/Hr and parameter-based methods incur high runtime without consistent gains. Additional self-evolution compute therefore does not reliably translate into cost-normalized performance.

Failure analysis

Two complementary analyses trace these patterns to an exploration–exploitation tension. Code similarity analysis shows conservative methods converging to narrow design families: Vanilla exhibits 41.0% Design Fixation with global similarity around 0.78 and positive convergence trends; ACE drives Fixation to 47.1% because score-ranked retrieval repeatedly reinserts early moderate successes into later prompts, creating a positive-feedback anchoring loop — at 14B ACE's global similarity peaks at 0.807, suggesting memory constrains models that reason better from scratch. Conversely, highly radical methods change code without directed progress: ToT has near-zero Fixation (0.3%) but 50.2% Stagnation, and CodeEvolve collapses its population prematurely (66.3% Fixation, similarity 0.835). Self-Refine's unverified revisions produce the highest radicality and 57.1% Stagnation, explaining its worst-in-class results. Among parameter-based methods, SEAL's error profile mirrors Vanilla because fine-tuning on its own successes preserves the base generation distribution, while RAGEN and TTT-Discover suffer from insufficient rollouts for RL-style convergence within 20 attempts.

The nine-type error taxonomy also reveals strong category dependence: Design Fixation reaches 58.2% in Control, and category predicts difficulty more strongly than model scale — the 4B→14B gain ranges from +16.3 points in Exotic Physics to +1.9 in Kinematics, suggesting scaling improves reasoning more than the combinatorial search required by linkage-design tasks.

Interventions: information asymmetry and visual feedback

To separate "know what" from "know how", a Change-Exposed (CE) condition discloses the changed variables and their exact new values. CE helps all 4B configurations (+2.6 to +8.4 points) but leaves most 8B/14B settings within ±5 points, and Vanilla-14B drops 7.2 points. Critically, the best CE result (Reflexion-14B, 14.6%) remains below the best Change-Hidden result (Reflexion-8B, 17.9%). Revealing exactly what changed does not raise the performance ceiling, indicating that mechanism redesign rather than parameter inference is the central bottleneck.

A VLM video-feedback intervention (Gemma4-26B-A4B generating natural-language failure descriptions from execution videos) polarizes by paradigm at larger scales: context-based methods degrade sharply (Vanilla-14B falls from 40.9% to 18.2%), whereas memory-augmented methods improve substantially (ExpeL-14B rises from 13.6% to 36.4%). Error-taxonomy shifts suggest fresh visual evidence counteracts stale retrieved memories for memory-based methods but scatters focused reasoning for context-based ones.

Limitations

The benchmark is confined to controlled 2D Box2D simulation, so findings may not transfer to 3D environments or real robots with noisy sensing and actuation. The fixed 20-attempt budget leaves sustained adaptation over longer horizons untested. Computational cost restricts the frontier-model, parameter-disclosure, and visual-feedback evaluations to selected domains and methods, and the paper notes that extending proprietary-model coverage to the full protocol would cost roughly $264,000 per run. Whether the observed paradigm-level conclusions hold beyond this simulator family and budget regime remains open.

Conclusion

PACE-Bench provides a reproducible testbed of 144 solvable but non-trivial source-to-target adaptation pairs combining hidden physics mutations, executable code-driven design, and systematic comparison of self-evolving paradigms. Its principal empirical claims are that simulator-grounded reflection outperforms unverified self-revision, that memory can anchor stronger models to early designs, that broad tree search explores without converging within budget, and that disclosing exact physical changes does not lift performance — locating the bottleneck in mechanism redesign ("know how") rather than parameter inference ("know what").