PhysicsArena Benchmark Suite

• PhysicsArena is a comprehensive benchmark that assesses multimodal physical reasoning via 3D construction tasks and structured physics problem instances.
• It integrates a suite of 3D engineering tasks with a dataset of over 5,000 physics problems categorized by difficulty and spanning 14 topics.
• The benchmark employs rigorous scoring with per-dimension accuracy and simulation-based validations to ensure real-world outcome verifiability.

PhysicsArena is a comprehensive benchmark suite designed to rigorously evaluate the multimodal and agentic physical reasoning capabilities of LLMs and multimodal LLMs (MLLMs). It provides two orthogonal but complementary approaches: (1) a suite of 3D physically-grounded engineering construction tasks built on symbolic geometric planning, block-wise simulation, and language-to-action workflows, and (2) a multimodal reasoning dataset targeting stepwise variable identification, process modeling, and solution derivation in physics problem solving. Together, these frameworks enable fine-grained analysis of models’ physical understanding, reasoning, and the translation of language and vision into verifiable, real-world outcomes.

1. Multimodal Dataset Composition and Task Dimensions

PhysicsArena’s multimodal reasoning benchmark comprises 5,103 structured physics problem instances, each consisting of textual context $\boldsymbol{T}$ (description), visual context $\boldsymbol{I}$ (static diagrams or images), and a question prompt $\boldsymbol{Q}$ (Dai et al., 21 May 2025). The problems are systematically categorized by difficulty (easy: 2,077, medium: 1,847, hard: 1,179) and span 14 physics topics, including magnetic field, electromagnetic induction, and Newton’s laws.

Each instance requires the model to output a triad $$\boldsymbol{O}_i = (\boldsymbol{O}_{V,i}, \boldsymbol{O}_{P,i}, \boldsymbol{O}_{S,i})$$ that must be matched against expert-labeled ground truths $$(\boldsymbol{G}_{V,i}, \boldsymbol{G}_{P,i}, \boldsymbol{G}_{S,i})$$ across three evaluation axes:

• Variable identification: Entity, geometry, field, structure, connection, external influence.
• Physical process formulation: Entity state, process detail, force/energy descriptors, state change, process relations.
• Stepwise solution derivation: Sequential justification and symbolic/numeric answer.

Metric-wise, per-dimension accuracy is computed via categorical matching, and LaTeX-formatted solutions are compared for final correctness. Hierarchical evaluation aggregates per-category accuracy and correlates the extraction of variables and processes with correct derivations; strong Pearson correlations ($p < 10^{-3}$) are observed between accurate variable/process chains and final solution accuracy, underscoring the holistic evaluation design.

2. Engineering Construction and Physics-Coupled Simulation

PhysicsArena’s agentic construction suite is inherited from BuildArena’s physics-aligned 3D pipeline (Xia et al., 18 Oct 2025). Tasks are interactively specified in natural language, parsed by multi-agent LLM workflows (Planner, Drafter, Reviewer, Builder, Guide, Control), and then instantiated in a 3D blockwise construction environment tightly coupled to a physics engine mirroring Besiege’s rigid-body dynamics.

Core system components:

• Natural-language prompts define the objective, constraints, and evaluation protocol.
• An open-source 3D Spatial Geometric Computation Library enforces module placement, pose, collision constraints, joint feasibility, and transformation invariance using SE(3) operations.
• Physics-based simulation validates resultants for motion, stability, and load-bearing performance strictly under physical laws (gravity $g$, friction, rigid collision, joint torque).

Table: Task Categories in PhysicsArena’s Construction Suite

Task Type	Physics Mechanic	Example Metric
Support	Static equilibrium	Maximum supported load $W_{\max}$
Transport	Horizontal dynamics	Min. trajectory deviation/displacement
Lift	Vertical thrust	Max. height or TWR

Each task-type is offered at three incremental difficulty tiers, scaling span, mass, load, trajectory complexity, or compositional requirements.

3. Evaluation Protocols and Scoring Methodology

PhysicsArena implements tailored scoring regimes for each task family:

• Multimodal Reasoning:
  • Variable accuracy $$\mathrm{Accuracy}_V = \frac{1}{N_V}\sum_{j=1}^6 \mathbb{I}(s_{v,j} = \text{True})$$.
  • Process accuracy $$\mathrm{Accuracy}_P = \frac{1}{N_P}\sum_{k=1}^5 \mathbb{I}(s_{p,k} = \text{True})$$.
  • Solution accuracy $\boldsymbol{I}$0.
  • Automated evaluation (GPT-4o) is used for large-scale scoring, augmented by human sampling for annotation verification.
• Construction Tasks:
  • 64 randomized trials per (task, model): recording success rate, part count, task-specific indicators (e.g., $\boldsymbol{I}$1 or $\boldsymbol{I}$2).
  • Cost metrics: total input/output tokens, number of LLM requests.
  • Rankings per metric and aggregation into radar-style profiles quantifying quantification, robustness, magnitude, compositionality, precision, and ambiguity.

For construction, the simulation-based testbed ensures that all symbolic plans produce physically valid, buildable outcomes with direct, quantitative mapping to the stated engineering constraints. Error and failure feedback is tracked, but no fully closed feedback loop from the simulator to agent is currently present.

4. Baseline Models and Empirical Results

Evaluated MLLMs include InternVL2.5, Qwen2.5-VL, Yi-VL, LLaVA, Qwen-VL-Max, GPT-4o, and Claude 3.5 Sonnet. All demonstrated significant degradation from variable identification (max $\boldsymbol{I}$3 on sub-tasks) through process formulation (max $\boldsymbol{I}$4) to complete symbolic solution accuracy ($\boldsymbol{I}$5 for best models) (Dai et al., 21 May 2025). Typical failure modes for construction tasks include misinterpretation of joint configuration, inadequate support distribution, and poor translation of geometric plans into physically viable assemblies (Xia et al., 18 Oct 2025).

PhysicsArena’s results highlight critical bottlenecks:

• Visual grounding deficiencies (misrecognition of diagram elements);
• Incomplete process chaining, particularly for dynamic, multi-phase systems;
• Scaling plateaus for larger models without targeted, task-specific finetuning.

5. Technical Frameworks, Algorithms, and Implementation Concepts

The construction component’s 3D geometric computation library supports:

• Block-level entity management $\boldsymbol{I}$6, pose mapping $\boldsymbol{I}$7, and action sequencing $\boldsymbol{I}$8.
• Mesh-based collision, connection feasibility, and pose transformation.
• API functions: block attachment, connection, relative twists, machine summary, and low-level control sequence insertion.

For stepwise reasoning, all annotations and model outputs adhere to formal JSON and LaTeX representations, enabling high-fidelity symbolic comparison. The domain-specific language interacting with the construction library is isomorphic to human GUI actions in Besiege, ensuring real-world alignability.

6. Comparative Positioning and Benchmark Significance

PhysicsArena extends beyond earlier text-only benchmarks by integrating parallel, multimodal axes and simulation-aligned action validation. In contrast to PHYRE (Bakhtin et al., 2019), which assesses sample-efficient generalization in a rigid 2D Newtonian context with restricted action space and limited verbs (“touch for ≥3 s”), PhysicsArena’s construction track generalizes to 3D object-oriented, compositional assembly, and rigorous mechanics (including static equilibrium, horizontal and vertical dynamics). The multimodal track robustly evaluates variable, process, and multistep reasoning.

This design supports more realistic assessment of both agent models’ visuospatial understanding and their ability to implement stepwise manipulations that drive toward physically meaningful outcomes. Both tracks place strong emphasis on compositionality, generalization to new problem types, and tightly-coupled evaluation with real or emulated physics engines.

7. Limitations and Prospective Extensions

PhysicsArena’s current limitations include:

• Absence of closed-loop correction from simulation feedback to the LLM agent in construction;
• Limited module diversity restricting the engineering design space;
• Visual reasoning dataset constrained to static images and high-school physics, with insufficient coverage of higher-level or dynamic temporal problems;
• Automated judging tools that may miss subtle annotation misalignments.

Ongoing and future work aims to:

• Expand the construction asset base and introduce more sophisticated modules (e.g., custom meshes, gears);
• Incorporate reinforcement or gradient-driven design refinement utilizing simulation logs;
• Extend evaluation to fluid/thermal dynamics or multi-agent collaborative assembly;
• Increase dataset complexity with advanced-level problem sets and video-based reasoning;
• Adopt more robust, hybrid human–AI evaluation methods.

PhysicsArena thus establishes a unified, extensible infrastructure for research into high-level, multimodal, physically-grounded reasoning—bridging symbolic, agentic, and simulation-centric paradigms in LLM and MLLM assessment (Dai et al., 21 May 2025, Xia et al., 18 Oct 2025).