Physics-IQ Verified: A Benchmark for Physics Mastery

• Physics-IQ Verified is a framework that defines and validates physics mastery through diverse, rigorously designed assessments spanning classical reasoning and AI-driven problem solving.
• It integrates instruments like the Force Concept Inventory, PIQL, and specialized AI benchmarks to reveal subtle differences in conceptual and quantitative physics reasoning.
• The framework supports actionable diagnostics for both educational and AI development via quantifiable scoring, psychometric validation, and detailed performance analytics.

Physics-IQ Verified

Physics-IQ Verified refers to a rigorous set of protocols, benchmarks, and instruments explicitly developed to quantify and validate an entity’s (human or AI) conceptual and quantitative mastery of core physics principles. This verification framework spans multiple modalities—classical reasoning inventories, agentic AI benchmarking against Olympiad-grade problems, large-scale evaluation of foundation models on university-level physics, and direct assessments of a model’s ability to understand physical reality in video generation scenarios. The unified purpose of Physics-IQ Verified is to establish a psychometrically sound, interpretable, and discriminative signal of "physics understanding" suitable for both educational diagnostics and benchmarking AI scientific reasoning.

1. Conceptual Foundation and Domains of Physics-IQ Verification

Physics-IQ Verified is grounded in the identification of central constructs that underpin expert-level physics reasoning:

• abstraction and formulation of physical scenarios,
• application of first-principles laws (e.g., conservation, field equations, Lagrangian/Euler–Lagrange formulation),
• quantitative computation (symbolic, numeric, and unit-aware manipulation),
• meta-reasoning (self-critique and peer review),
• integration of external, domain-specific knowledge.

Verification addresses both content-specific mastery (e.g., Newtonian mechanics, energy/momentum, quantum mechanics) and generic quantitative reasoning (e.g., ratios, covariation, negativity in physical contexts). Physics-IQ assessment spans from tightly focused conceptual inventories (such as the Force Concept Inventory) to full-stack tool-augmented AI agents solving Olympiad and university-level benchmark problems (Qiu et al., 1 Sep 2025, Feng et al., 26 Mar 2025, Chrysostomou et al., 2021).

2. Physics-IQ Assessment Instruments: Structure and Psychometric Properties

Human-Oriented Instruments

Force Concept Inventory (FCI):

• 30 multiple-choice items targeting Newtonian concepts, designed for robust pre/post testing.
• Six "polarising" questions isolate predominant misconceptions (e.g., fictitious forces, impetus theory).
• Psychometric properties: item difficulty range 0.2–0.8, discrimination index $D_i>0.3$, cohort Cronbach's $\alpha \ge 0.7$.
• Normalized gain formula for instructional effect:

$$G = \frac{\langle\%\;S_f\rangle - \langle\%\;S_i\rangle}{100 - \langle\%\;S_i\rangle}$$

• A high cut-off (e.g., 5/6 correct polarising items) is recommended for "Physics-IQ Verified" designation (Chrysostomou et al., 2021).

Energy and Momentum Concept Inventory (EMCI):

• 25-item, multiple-choice, fine-grained energy/momentum understanding.
• Scaled for cognitive complexity (recall, interpretation, application).
• Internal consistency: graduate sample $\alpha > 0.80$, calculus-based undergraduate $\alpha \approx 0.75$.
• Factor analysis supports unidimensional structure (largest component <15% variance indicates lack of unintentional multidimensionality) (Singh et al., 2016).

Physics Inventory of Quantitative Literacy (PIQL):

• Probes three expert-identified PQL facets: ratios/proportions, covariation, signed quantities.
• 20 items; includes six multiple-choice–multiple-response (MCMR) for diagnostic richness.
• Four-level MCMR rubric (from "Completely Correct" to "Completely Incorrect").
• Cronbach’s $\alpha$ reaches 0.80 in v2.2, all items with discrimination $D_i>0.30$.
• Factor analysis: EFA and CFA results indicate a strong unidimensional latent trait (CFI/TLI>0.93; RMSEA<0.04) (Smith et al., 2020).

General Equation-based Reasoning inventory of QuaNtity (GERQN):

• Algebra-based version of PIQL for broader accessibility.
• Three guide facets (sign, covariational, proportional reasoning), but factor analysis supports a single-factor structure at this level.
• Reliability: Cronbach’s $\alpha$ 0.67–0.73, Ferguson’s $\delta$ 0.95–0.96 indicates adequate score spread.
• Scoring: 17 dichotomous items; MCMR stringency (full credit only if all and only correct choices are selected).
• Diagnostic application for pre/post tests and targeted remediation (Zimmerman et al., 20 Apr 2025).

AI/Model-Oriented Benchmarks

IPhO-Grade Agent Benchmarking:

• Example: Physics Supernova system, orchestrating high-level LLM reasoning with tool integration (ImageAnalyzer, AnswerReviewer, WolframAlpha QA, and external database retrieval).
• Evaluated on IPhO 2025 theory problems (30 points max): median gold medalist score $S_{\rm med}=22.8$, with Supernova achieving $\alpha \ge 0.7$0 (ranking 14th / 406; surpasses human elite median).
• Ablations systematically quantify tool contributions ($\alpha \ge 0.7$1 full vs. LLM only).
• Scoring is based on official rubrics, with granular sub-part points and expert raters for human–machine comparability (Qiu et al., 1 Sep 2025).

University-Level Physics Foundation Model Benchmarking:

• PHYSICS benchmark: 1,297 graduate-level exam problems, six subfields.
• Rigorous, automated pipeline (SymPy symbolic equivalence, GPT-based fallback).
• Composite accuracy metric:

$\alpha \ge 0.7$2

• Top model accuracy (o3-mini): 59.9%; exposes systematic limitations (domain integration, vision, multi-step reasoning).
• Failure taxonomy: knowledge gaps, erroneous assumptions, symbol manipulation, visual misreadings, and prompt comprehension errors (Feng et al., 26 Mar 2025).

Physics-IQ for Video Generative Models (VGMs):

• Reference-based evaluation using real-world experiments (396 8s clips, four principal metrics: Spatial-IoU, Spatiotemporal-IoU, Weighted-Spatial-IoU, MSE).
• Original protocol aggregated metrics per-dataset, but systematic audit revealed shortcomings: prompt ambiguity (34.8% prompts), ground-truth artifacts (30% samples), and aggregation bias.
• Physics-IQ Verified introduces: (1) prompt restructuring (six-field, best-practice format), (2) manual artifact removal (frame freezing, regional cleaning for artifacts), (3) per-sample composite scoring (ensuring equal weight per sample and metric).
• Results: 57.6% of samples and 34.8% of prompts refined; moderate ranking change among six VGMs (Kendall’s $\alpha \ge 0.7$3).
• Statistical validation via Wilcoxon signed-rank, Cohen’s $\alpha \ge 0.7$4, and bootstrap rank correlation (Rädsch et al., 17 Jun 2026).

3. Methodologies, Scoring, and Validation Protocols

Physics-IQ Verified protocols enforce strict psychometric and statistical discipline:

• Classical Test Theory (CTT): item difficulty ($\alpha \ge 0.7$5), discrimination (point-biserial $\alpha \ge 0.7$6, index $\alpha \ge 0.7$7), overall reliability (Cronbach’s $\alpha \ge 0.7$8).
• Factor Analysis: EFA (data-driven), CFA (hypothesis-driven), fit indices (CFI, TLI, RMSEA, SRMR).
• Four-level and dichotomous scoring rubrics for multiple-response and single-answer items.
• For AI benchmarks, symbolic equivalence is established via code-assisted parsing (SymPy), with fallback to LLM inference where necessary.
• For VGM benchmarks, per-sample scoring metrics are clipped and equal-weighted:

$\alpha \ge 0.7$9

Final score:

$$G = \frac{\langle\%\;S_f\rangle - \langle\%\;S_i\rangle}{100 - \langle\%\;S_i\rangle}$$0

• Cross-validations and human rater interconsistency are required for high-stakes assessments.

4. Diagnostic and Educational Implications

Physics-IQ Verified tools yield actionable diagnostics at the individual, cohort, and system level:

• Subscale analysis reveals domain-specific weaknesses (e.g., energy vs. momentum, sign reasoning).
• Distractor selection patterns map to entrenched conceptual errors (e.g., path-dependence, "force-as-cause," confusion of momentum vs. kinetic energy).
• Item-level analytics delimit interventions—remediation can be tailored to targeted constructs (e.g., sign reasoning tutorial if sign item performance is low).
• For AI agents, ablation studies expose bottlenecks in vision, symbolic manipulation, or external knowledge retrieval pipelines.
• Physics-IQ Verified flags both instructional effectiveness (via normalized gains, cohort pre/post deltas) and areas resilient to standard pedagogy.

5. Impact, Benchmarking, and Limitations

Physics-IQ Verified functions as a cross-modal, development-driving benchmark:

• In video generation, revised protocol and artifact cleaning realign VGM evaluation with meaningful physical understanding (moderate but meaningful ranking reordering, illustrating that spurious prompt/artifact effects had inflated previous results) (Rädsch et al., 17 Jun 2026).
• In agentic AI, tool integration (vision, peer-review, external constants) is quantitatively shown to be necessary for surpassing the "standalone LLM" plateau (Qiu et al., 1 Sep 2025).
• In human assessment, verified protocols support rapid, discriminative screening (e.g., for departmental honors, scholarship selection, or transfer).
• Thorough psychometric and statistical validation limit spurious findings and ensure cross-institutional reliability.

Identified limitations include reliance on single-continuation reference data for VGM evaluation (future extensions may use multi-modal ground truths or invariants), labor-intensity of manual artifact annotation, and the need for further multi-institutional norming (notably for PIQL and GERQN in non-research university environments).

6. Future Directions and Research Trajectories

• Physics-IQ Verified methodologies are expected to incorporate formal proof assistants (e.g., Lean) for programmatic algebraic derivation verification.
• Expansion of benchmarks to interdisciplinary and experimentally richer problem suites (e.g., beyond mechanics to plasma, nuclear, or instrumented laboratory physics) is recommended (Feng et al., 26 Mar 2025).
• VGM evaluation may incorporate trajectory-based, invariant-based, or physics-informed neural network (PINN) metrics to handle plausible alternative outcomes.
• Automated artifact detection and scalable prompt validation would increase throughput and cross-benchmark consistency.
• Hybrid neuro-symbolic and Retrieval-Augmented Generation (RAG) approaches are projected to further close the AI–expert gap in university-level physics.

Physics-IQ Verified represents a paradigm for interpretable, robust, and cross-modal evaluation of physics understanding in both humans and artificial agents, with reproducibility and developmental impact at its core.