PRISM-Physics: DAG-Based Evaluation Framework

• PRISM-Physics is a process-level framework that represents physics derivations as directed acyclic graphs (DAGs) capturing causal dependencies among formulas.
• It employs a rule-based symbolic matcher and ancestor closure scoring to assess intermediate reasoning steps with fine-grained precision.
• The framework addresses limitations of final-answer-only and linear evaluations, offering interpretable, nuanced evaluation of competition-level physics problems.

PRISM-Physics is a process-level evaluation framework and benchmark for competition-style physics reasoning in which solutions are represented as directed acyclic graphs (DAGs) of formulas, with edges encoding causal dependencies among intermediate derivations. It is designed to address two limitations identified in prior physics benchmarking: final-answer-only scoring, which misses partial reasoning competence, and linear stepwise evaluation, which is either too restrictive or dependent on heuristic judging. By combining a DAG-structured representation of reference solutions, a causality-aware scoring policy based on ancestor closure, and a fully rule-based symbolic equivalence matcher, PRISM-Physics aims to provide fine-grained, interpretable, and theoretically grounded evaluation for complex physics derivations (Zhao et al., 3 Oct 2025).

1. Motivation and problem setting

PRISM-Physics is situated in the broader problem of evaluating scientific reasoning in physics, where intermediate derivations are often as important as the final numerical or symbolic answer. The framework argues that existing physics benchmarks are inadequate in two main ways. First, final-answer-only scoring misses partial reasoning competence: a model may derive many correct intermediate equations and fail only near the end, so answer-only accuracy underestimates reasoning ability and yields sparse training signal. Second, linear stepwise evaluation is too restrictive or too heuristic. Prior process-based methods often assume a strict step order or rely on LLM-as-judge grading, while physics derivations routinely involve equivalent symbolic forms, constant substitutions, and unit conversions that make heuristic grading fragile (Zhao et al., 3 Oct 2025).

A central misconception addressed by the framework is that process supervision in physics can be reduced to a linear list of steps. PRISM-Physics explicitly rejects that assumption. Its premise is that physics reasoning is better modeled by causal structure: a later equation may depend on multiple earlier results, and correct later derivations should credit the intermediate formulas they causally require. This suggests a graph-based rather than sequence-based formalization.

The framework is specifically formula-centric. Purely textual questions, such as descriptive or yes/no conceptual prompts, are excluded from the current benchmark because the scorer is formula-based. This restriction defines the intended scope of PRISM-Physics: formal derivations with machine-checkable mathematical content rather than unrestricted scientific discourse.

2. DAG representation of physics solutions

In PRISM-Physics, a reference solution is represented as a DAG $G=(V,E)$, where each node $v\in V$ is a canonicalized formula and each directed edge $(u,v)\in E$ means that formula $v$ is derived from prerequisite formula $u$. Edges always point forward in logical order, redundant algebraic steps are removed, and every node must lie on a path to at least one final-answer node. The paper characterizes this graph as the “logical skeleton” of a derivation (Zhao et al., 3 Oct 2025).

The DAG construction enforces four properties. Nodes are canonicalized LaTeX expressions representing key steps. Edges encode derivational dependencies. Minimality requires that only essential formulas are retained. Completeness requires that every retained node connect causally to a final answer node. These constraints are intended to make the representation machine-interpretable while preserving global derivational structure.

This representation is presented as superior to both strict line-by-line matching and simple prefix credit. Strict matching is too harsh because a correct but differently organized derivation may be unfairly penalized. Prefix credit is too generous because matching a later formula in a linear sequence may incorrectly credit earlier steps that were not actually required. The DAG formalism avoids both failures by assigning credit along causal prerequisite chains rather than along arbitrary textual order.

The framework further defines a justification-theoretic perspective. It introduces a justification system $(\mathcal F,\Rightarrow)$, where $X\Rightarrow B$ means that once every formula in $X$ is matched, formula $B$ is automatically warranted. Under the assumptions of singleton minimal justifiers and causality, the paper proves a bijection between order-keeping justification systems and DAGs. Concretely,

$$\Phi(\vdash)=\bigl(\mathcal F,\;E_\vdash:=\{(A,B):A\vdash B\}\bigr),$$

$v\in V$0

The maps $v\in V$1 and $v\in V$2 are stated to be injective and mutual inverses, yielding the bijection

$v\in V$3

The stated interpretation is that the DAG is not merely a visualization device but the minimal encoding of the justification structure.

3. Scoring policy and rule-based symbolic matching

The scoring rule in PRISM-Physics is based on ancestor closure. For a set of matched formulas $v\in V$4, the framework defines

$v\in V$5

and the final score as

$v\in V$6

where $v\in V$7 is the set of all formulas in the reference DAG. This policy grants credit to a matched node and to all of its causal ancestors (Zhao et al., 3 Oct 2025).

The paper characterizes this policy axiomatically. An admissible scoring policy $v\in V$8 over matched formulas $v\in V$9 must satisfy three properties: matched inclusion, ancestor closure, and soundness. The theorem stated in the paper gives an exact characterization: $(u,v)\in E$0 Within the paper’s setup, ancestor-closure scoring is therefore not presented as a heuristic choice but as the unique admissible policy.

A second core component is the formula matcher used to determine whether a model-produced formula matches a reference node. This matcher is fully rule-based and does not rely on LLM judgment. The paper identifies three challenges: equations rather than bare expressions, constant substitution, and unit conversion. The matching procedure is two-stage. First, variables, constants, and units are normalized by substituting predefined constant expressions and values into formulas. Second, equivalence is checked by randomized solution-set comparison. For two equations $(u,v)\in E$1 with free variables $(u,v)\in E$2, the algorithm randomly chooses a target variable $(u,v)\in E$3, assigns random values in the positive range $(u,v)\in E$4 to the remaining variables, solves both equations for $(u,v)\in E$5, and compares the resulting solution sets numerically within tolerance $(u,v)\in E$6. This is repeated up to $(u,v)\in E$7 trials, with a success requirement $(u,v)\in E$8.

The intended effect is deterministic symbolic validation across diverse formulations. The algorithm declares equivalence only if there are enough valid trials and no trial rejects equivalence. The pipeline described in the paper has three stages: extract and normalize student formulas, match them to reference DAG nodes using the rule-based equivalence checker, and score the result by ancestor closure.

4. Benchmark construction and annotation

The benchmark underlying PRISM-Physics is described as a large collection of competition-level physics problems with curated DAG-structured solutions. Its main source is Major American Universities Ph.D. Qualifying Questions and Solutions, extracted from PDFs and converted into structured JSON (Zhao et al., 3 Oct 2025).

Each sample is processed through a three-stage rewriting pipeline. Formula normalization standardizes LaTeX formatting, enforces significant-figure rules, and unifies notation. Context clarification defines all variables explicitly and makes answer requirements unambiguous. DAG construction converts each worked solution into a DAG and verifies dependency structure with rule-based and LLM-based checks. Each stage is checked by an LLM-based verifier, and failures trigger regeneration with corrective feedback.

Problems are labeled both by domain and by difficulty. The framework uses seven domain labels: Mechanics; Electromagnetism; Optics; Atomic, Nuclear, and Particle Physics; Thermodynamics and Statistical Physics; Quantum Mechanics; and Solid State Physics and Miscellaneous Topics. Difficulty is composite, combining conceptual depth $(u,v)\in E$9, computational burden $v$0, and entropy-based DAG complexity $v$1. The entropy term is defined as

$v$2

and the overall score

$v$3

is then mapped to Easy, Medium, or Hard.

This benchmark design distinguishes PRISM-Physics from another 2025 physics-reasoning resource, PRiSM, which is a synthetic, fully dynamic, multimodal benchmark with executable Python ground truth and unit-aware verification (Imani et al., 5 Dec 2025). PRISM-Physics instead centers on causal DAG representations and symbolic derivation scoring. This suggests a division of emphasis: PRiSM targets dynamic multimodal scientific reasoning, while PRISM-Physics targets process-level evaluation of formal derivations.

5. Empirical evaluation and human alignment

PRISM-Physics evaluates a wide set of LLMs in both text-only and multimodal settings with figures, under standardized prompts and decoding settings. Two baselines are used: LLM-as-judge scoring and PSAP-S / PhysReason-style step scoring, which relies on stepwise checks but lacks causal DAG modeling (Zhao et al., 3 Oct 2025).

The main empirical result is that step-level scoring reveals substantially more than final-answer scoring. Across models, performance drops as difficulty rises and response time increases. Final-answer accuracy falls sharply on medium and hard problems, while step-level accuracy remains substantially higher. The paper interprets this as evidence that models often obtain many intermediate formulas correctly before failing later in the derivation. This is a central empirical justification for process-level evaluation.

The reported experiments also indicate that reasoning-oriented models generally achieve better step accuracy than chat-only models, but with large latency costs. Increasing reasoning effort in models such as GPT-5 improves accuracy further, again at the cost of longer response times. Multimodal input can improve intermediate reasoning more than final-answer accuracy, although for smaller models images may hurt performance when the diagram is not essential. The paper further reports domain variation: models perform best in thermodynamics/statistical physics and worst in quantum mechanics.

Human expert comparison is a major validation component. The paper samples 70 problems, 10 from each domain, and uses solutions from DeepSeek-V3. Two human experts score each item independently, with disagreements adjudicated by a third annotator. The annotators include an IPhO gold medalist and a top-tier physics PhD. Agreement is measured by Kendall’s $v$4. PRISM-DAG is reported to outperform both baselines:

• LLM-as-Judge: $v$5
• PSAS-S: $v$6
• PRISM-DAG: $v$7

It also has the smallest p-values, with asymptotic $v$8 and permutation $v$9. The paper’s stated conclusion is that PRISM-DAG is more aligned with human expert grading than outcome-only or earlier process-based baselines.

6. Failure modes, limitations, and significance

PRISM-Physics includes an explicit taxonomy of first incorrect steps. The seven categories are Diagram Analysis Error (DAE), Physics Theorem/Application Error (PTAE), Modeling and Process Understanding Error (MPUE), Condition or Assumption Error (CAE), Variable Relationship Error (VRE), Derivation and Computation Error (DCE), and Unit Dimension Error (UDE). The main failure modes observed are CAE, DCE, and MPUE. In the paper’s interpretation, this indicates that current models struggle both with selecting the correct physical setup and with executing derivations reliably (Zhao et al., 3 Oct 2025).

The framework also acknowledges evaluator failure modes. Context-sensitive equivalence is not fully captured, text-only responses fall outside strict formula matching, and parsing long-tail LaTeX notation remains imperfect. These limitations define the present boundaries of the method. The framework is therefore not a general evaluator of all physics reasoning; it is a structured evaluator for formula-based derivations with canonicalizable intermediate steps.

Its broader significance lies in the claim that physics evaluation should be process-level, causal, and symbolic rather than answer-level alone. The step-level scores are presented as especially useful because they provide dense intermediate reward signals. The paper explicitly suggests that PRISM-Physics can guide reinforcement learning, post-training, and data curation for scientific reasoning models.

A plausible implication is that PRISM-Physics occupies a methodological middle ground between static answer-key benchmarks and fully dynamic scientific-reasoning environments. It does not attempt to simulate the entire scientific workflow, but it imposes structural rigor on derivational correctness. In that sense, its contribution is not only a benchmark but an evaluation doctrine: intermediate formulas are treated as causally organized objects, and scoring is defined over that structure rather than over surface-form sequences or opaque holistic judgments.