---
title: 'PhyJudge-9B: Physics-Aware Video Judge'
url: https://www.emergentmind.com/topics/phyjudge-9b
type: topic
---

# PhyJudge-9B: Physics-Aware Video Judge

Searching arXiv for the PhyGround / PhyJudge paper to ground the article in the primary source.
arXiv search query: "PhyGround Benchmarking Physical Reasoning in Generative World Models"
PhyJudge-9B is an open, physics-specialized vision-language model judge released alongside the PhyGround benchmark for evaluating physical reasoning in video generation. It is designed to score whether generated videos follow expected physical outcomes under a taxonomy of 13 physical laws spanning solid-body mechanics, fluid dynamics, and optics. The model operates on an augmented prompt that includes the original conditioning information, an explicit expected outcome, and a per-law sub-question checklist, and returns a JSON object containing a single integer score from 1 to 5 for the queried dimension. Within the PhyGround framework, PhyJudge-9B is positioned as an audit-friendly automated evaluator intended to complement a large-scale, quality-controlled human study and to reduce evaluator bias in physics-aware video assessment [2605.10806].

## 1. Definition and benchmark context

PhyJudge-9B is the open “judge” model released with PhyGround, a criteria-grounded benchmark for evaluating physical reasoning in video generation [2605.10806]. PhyGround contains 250 curated prompts, each augmented with an expected physical outcome, and organizes evaluation around a taxonomy of 13 physical laws across solid-body mechanics, fluid dynamics, and optics. In this setting, PhyJudge-9B functions as a reproducible automated evaluator for both general and law-specific scoring.

The benchmark is motivated by three stated challenges in prior physics-focused video evaluation: coarse evaluation frameworks that hide law-specific failures, response biases and fatigue that undermine the validity of annotation judgments, and automated evaluators that are insufficiently physics-aware or difficult to audit [2605.10806]. PhyJudge-9B addresses these constraints through law-level decomposition, fixed inference settings, and explicit sub-question rubrics.

The judge is trained and evaluated in a framework that includes eight modern video generation models and a large-scale human annotation study. A total of 459 annotators provided 5,796 complete annotations and over 37.4K fine-grained labels; after quality control, the retained annotations exhibited high split-half model-ranking correlations with Spearman’s $\rho > 0.90$ [2605.10806]. This human study provides the reference signal against which PhyJudge-9B’s automated judgments are compared.

## 2. Architecture and input-output interface

PhyJudge-9B is built on a 9-billion-parameter multimodal Transformer stack combining four components [2605.10806]:

- **A frozen Vision Transformer (ViT)-based video encoder**: it tokenizes sampled frames, with 4 fps used by default.
- **A frozen language Transformer (causal LLM)**: it processes the prompt and scoring rubric.
- **A lightweight trainable vision-language aligner**: it projects visual tokens into the LLM’s modality.
- **LoRA adapters**: these use rank 32, $\alpha = 64$, and dropout $= 0.05$, and are injected into all q/k/v/o and MLP projection layers of the LLM.

The input consists of a short video and an “augmented prompt.” The augmented prompt includes three elements: the original text plus first-frame image condition, an explicit expected outcome, and a per-law sub-question checklist. The output is a JSON object containing a single integer score from 1 to 5 for the queried dimension, whether general or per-law [2605.10806].

The released inference interface is likewise specified as JSON. The input has fields `{"video": [frame tensors], "prompt": text describing initial frame + expected outcome, "law": law-key, "sub_questions": [binary probes]}`, and the inference call returns `{"law": <law_key>, "score": <integer 1–5>}` [2605.10806]. The typical pipeline per video makes 3 calls for general dimensions—SA, PTV, and Persistence—and 2–4 calls for applicable physical laws.

A plausible implication is that the model’s interface is intentionally narrow: it is designed for rubric-conditioned scoring rather than open-ended explanation generation. This is consistent with the stated emphasis on auditability and version-pinned evaluation.

## 3. Training procedure and supervision

PhyJudge-9B is trained on 2,000 videos, each associated with 2–4 applicable laws and scored by 459 annotators, yielding 37.4K individual Likert scores [2605.10806]. All Veo-3.1 videos and their labels are held out for evaluation. Supervision is provided as structured JSON “criterion $\rightarrow$ score” pairs, where each score is the human-aggregated mean on a 1–5 scale.

The optimization objective is token-level negative log-likelihood over the serialized JSON label:

$$
\mathcal{L}(\theta) \;=\; -\mathbb{E}_{(v,q_k,y_k)\sim\mathcal{D}\sum_{t=1}^{|y_k|} \log p_\theta(y_{k,t}\mid y_{k,<t},\,v,\,q_k)\,.
$$

Only the LoRA parameters $\theta$ and the small vision-language aligner are trainable; the base ViT and LLM remain frozen [2605.10806]. This training choice confines adaptation to a relatively small parameter subset while preserving the base multimodal stack.

The reported hyperparameters are AdamW, learning rate $= 1\mathrm{e}{-4}$ for LoRA and $2\mathrm{e}{-6}$ for the aligner, 5% warmup, cosine decay, gradient clipping 1.0, and effective batch size 8 with gradient accumulation. Training runs for 1 epoch, approximately 300 steps, taking 1 h 42 m on 4 $\times$ A100 80 GB [2605.10806].

This training design suggests a parameter-efficient specialization regime rather than full multimodal re-training. The frozen-encoder, frozen-LLM, and trainable-adapter pattern aligns with the model’s intended role as a domain-specific evaluator rather than a general-purpose generator.

## 4. Physical-law taxonomy and rubric grounding

PhyGround organizes physical reasoning into 13 laws grouped into three domains, and PhyJudge-9B evaluates them through explicit sub-question operationalization [2605.10806]. Each law is broken down into 2–3 binary “violation probes” that constrain the judge’s attention. Human annotators see only the main question and a brief note, whereas PhyJudge-9B sees the full sub-question checklist.

| Domain | Laws |
|---|---|
| Solid-Body Mechanics | Gravity, Inertia, Momentum, Impenetrability, Collision, Material |
| Fluid Dynamics | Buoyancy, Displacement, Flow Dynamics, Boundary Interaction, Continuity |
| Optical Physics | Reflection, Shadow |

The solid-body mechanics laws are defined by prompts such as Gravity (“Do unsupported objects/liquids fail to move downward?”), Inertia (“Do objects change motion without an external force?”), Momentum (“Are post-collision directions/speeds implausible?”), Impenetrability (“Do solids intersect?”), Collision (“Is rebound or deformation inconsistent with impact?”), and Material (“Do objects violate the expected behavior of their material?”) [2605.10806].

The fluid-dynamics laws are Buoyancy (“Do objects sink/float against density?”), Displacement (“Does liquid level not rise or overflow correctly?”), Flow Dynamics (“Is bulk motion of liquid implausible?”), Boundary Interaction (“Is splash/deflection wrong upon contact?”), and Continuity (“Does fluid disappear or appear spontaneously?”). The optical-physics laws are Reflection (“Do mirror surfaces display implausible reflections?”) and Shadow (“Are shadows directionally or temporally inconsistent?”) [2605.10806].

For Collision, the reported example sub-questions are: “Does one object pass through another at impact?”, “Is rebound direction opposite to the incident vector?”, and “Is rebound speed grossly mismatched to impact speed?” [2605.10806]. This rubric design operationalizes law-specific failures in observable terms rather than relying on a single undifferentiated physics score.

A plausible implication is that the sub-question checklist serves as an attention-shaping device for the judge. The paper explicitly states that these law-specific checklists steer the judge’s observations and reduce hallucinations [2605.10806].

## 5. Evaluation metrics and comparative performance

PhyJudge-9B is evaluated using aggregate relative bias, signed relative bias, and Spearman’s rank correlation [2605.10806]. For a judge $J$ and human mean $H$ over the same scores, aggregate relative bias is defined as

$$
E \;=\; \frac{|\bar J - \bar H|}{\bar H}\times100\%\,,
$$

where lower $E$ indicates closer agreement with humans. Signed relative bias is

$$
\delta \;=\; \frac{\bar J - \bar H}{\bar H}\times100\%\,,
$$

with $\delta > 0$ indicating systematic over-scoring and $\delta < 0$ under-scoring. Spearman’s rank correlation is given by

$$
\rho \;=\; 1 - \frac{6\sum_i d_i^2}{n(n^2-1)}\,,
$$

where $d_i$ is the rank difference for the $i$-th model among $n$ models.

On a held-out test set of Veo-3.1 videos, with all judges using the same per-law sub-question schema, PhyJudge-9B is reported to have substantially lower aggregate relative bias than Gemini-3.1-Pro [2605.10806].

| Judge | General | Physics | Overall | Signed bias |
|---|---:|---:|---:|---:|
| Gemini-3.1-Pro (no CoT) | 17.3% | 15.9% | 16.6% | -6.3% |
| PhyJudge-9B | 1.0% | 5.5% | 3.3% | -1.1% |

The paper states that PhyJudge-9B’s average score is within $\pm 3.3\%$ of human means, versus $\pm 16.6\%$ for Gemini-3.1-Pro [2605.10806]. It also provides a head-to-head optical example on a rotating teapot-mirror scene: the human mean for Reflection is 4.4/5, PhyJudge-9B gives 4.6, and Gemini-3.1-Pro gives 1.0.

The benchmark-level human study reports split-half model-ranking $\rho > 0.90$ between human subsets [2605.10806]. This indicates that the human reference labels used for supervision and evaluation are internally stable at the level of model ranking.

## 6. Auditability, reproducibility, and release artifacts

PhyJudge-9B is explicitly described as physics-specialized and audit-friendly [2605.10806]. Its design decisions are organized around four elements: open source and fixed weights, per-law decomposition, sub-question rubrics, and version-pinned inference.

Open source and fixed weights mean that the model weights, LoRA adapters, and evaluation code are fully released under a permissive license, with no hidden API calls [2605.10806]. Per-law decomposition means that instead of a single physics score, the judge emits a separate 1–5 score per law. Sub-question rubrics are included to steer observations and reduce hallucinations. Version-pinned inference means that all hyperparameters, including frame rate, prompt templates, and sub-questions, are codified to ensure reproducibility.

The release includes prompts, human annotations, model checkpoints, evaluation code, prompt templates, sub-question definitions, and the LoRA checkpoint for the 9B-parameter model at the project page [2605.10806]. This packaging makes the evaluation stack inspectable at the level of both model parameters and scoring protocol.

A plausible implication is that the model is intended to function not merely as a benchmark artifact but as an infrastructure component for repeated evaluation under fixed conditions. The paper’s phrasing around codified inference settings and audit-friendly release supports that interpretation.

## 7. Findings, scope, and limitations

The benchmark reports that fine-grained, per-law evaluation reveals highly heterogeneous strengths and weaknesses in generative world models [2605.10806]. The stated example is that open-source models match closed-source in “collision” but lag greatly on “fluid continuity.” This finding follows directly from the law-specific evaluation design of PhyGround and the per-law outputs of PhyJudge-9B.

The same source reports that PhyJudge-9B reduces automated evaluation bias by approximately 80% relative to closed-source baselines, enabling reproducible large-scale physics evaluations [2605.10806]. Across eight video generators, no model exceeds 3.3/5 on overall physics realism, and solid-body laws remain the hardest, with mean approximately 2.8/5. These values situate PhyJudge-9B within a broader empirical conclusion: current generative world models remain substantially limited in physical realism under rubric-grounded assessment.

Several boundary conditions are also evident. PhyJudge-9B scores queried dimensions as integers from 1 to 5 rather than producing free-form analyses, and its evaluation pipeline depends on explicit law keys and sub-question checklists [2605.10806]. This indicates that its competence is structured by the taxonomy and rubrics supplied at inference time. A plausible implication is that its strengths are greatest when the target physical phenomenon is well captured by the benchmark’s observable sub-questions.

The release paper states that PhyJudge-9B is expected to accelerate physics-aware model development by providing an open, audit-friendly, law-specific “stress tester” for generative world models [2605.10806]. In encyclopedic terms, its significance lies in combining a fixed multimodal judge, a rubric-grounded taxonomy of physical laws, and a human-validated evaluation protocol into a single reproducible assessment framework for video-generation systems.

Source: https://www.emergentmind.com/topics/phyjudge-9b