PhyJudge-9B: Physics-Aware Video Judge

• The paper demonstrates that PhyJudge-9B significantly reduces evaluation bias and aligns closely with human judgments across 13 physical law dimensions.
• It leverages a frozen ViT encoder and causal LLM with trainable LoRA adapters to process augmented prompts and law-specific sub-question checklists.
• PhyJudge-9B sets a reproducible, audit-friendly standard for assessing physical realism in video generation using a detailed taxonomy of physical laws.

Searching arXiv for the PhyGround / PhyJudge paper to ground the article in the primary source. arXiv search query: "PhyGround Benchmarking Physical Reasoning in Generative World Models" PhyJudge-9B is an open, physics-specialized vision-LLM judge released alongside the PhyGround benchmark for evaluating physical reasoning in video generation. It is designed to score whether generated videos follow expected physical outcomes under a taxonomy of 13 physical laws spanning solid-body mechanics, fluid dynamics, and optics. The model operates on an augmented prompt that includes the original conditioning information, an explicit expected outcome, and a per-law sub-question checklist, and returns a JSON object containing a single integer score from 1 to 5 for the queried dimension. Within the PhyGround framework, PhyJudge-9B is positioned as an audit-friendly automated evaluator intended to complement a large-scale, quality-controlled human study and to reduce evaluator bias in physics-aware video assessment (Lin et al., 11 May 2026).

1. Definition and benchmark context

PhyJudge-9B is the open “judge” model released with PhyGround, a criteria-grounded benchmark for evaluating physical reasoning in video generation (Lin et al., 11 May 2026). PhyGround contains 250 curated prompts, each augmented with an expected physical outcome, and organizes evaluation around a taxonomy of 13 physical laws across solid-body mechanics, fluid dynamics, and optics. In this setting, PhyJudge-9B functions as a reproducible automated evaluator for both general and law-specific scoring.

The benchmark is motivated by three stated challenges in prior physics-focused video evaluation: coarse evaluation frameworks that hide law-specific failures, response biases and fatigue that undermine the validity of annotation judgments, and automated evaluators that are insufficiently physics-aware or difficult to audit (Lin et al., 11 May 2026). PhyJudge-9B addresses these constraints through law-level decomposition, fixed inference settings, and explicit sub-question rubrics.

The judge is trained and evaluated in a framework that includes eight modern video generation models and a large-scale human annotation study. A total of 459 annotators provided 5,796 complete annotations and over 37.4K fine-grained labels; after quality control, the retained annotations exhibited high split-half model-ranking correlations with Spearman’s $\rho > 0.90$ (Lin et al., 11 May 2026). This human study provides the reference signal against which PhyJudge-9B’s automated judgments are compared.

2. Architecture and input-output interface

PhyJudge-9B is built on a 9-billion-parameter multimodal Transformer stack combining four components (Lin et al., 11 May 2026):

• A frozen Vision Transformer (ViT)-based video encoder: it tokenizes sampled frames, with 4 fps used by default.
• A frozen language Transformer (causal LLM): it processes the prompt and scoring rubric.
• A lightweight trainable vision-language aligner: it projects visual tokens into the LLM’s modality.
• LoRA adapters: these use rank 32, $\alpha = 64$, and dropout $= 0.05$, and are injected into all q/k/v/o and MLP projection layers of the LLM.

The input consists of a short video and an “augmented prompt.” The augmented prompt includes three elements: the original text plus first-frame image condition, an explicit expected outcome, and a per-law sub-question checklist. The output is a JSON object containing a single integer score from 1 to 5 for the queried dimension, whether general or per-law (Lin et al., 11 May 2026).

The released inference interface is likewise specified as JSON. The input has fields {"video": [frame tensors], "prompt": text describing initial frame + expected outcome, "law": law-key, "sub_questions": [binary probes]}, and the inference call returns {"law": <law_key>, "score": <integer 1–5>} (Lin et al., 11 May 2026). The typical pipeline per video makes 3 calls for general dimensions—SA, PTV, and Persistence—and 2–4 calls for applicable physical laws.

A plausible implication is that the model’s interface is intentionally narrow: it is designed for rubric-conditioned scoring rather than open-ended explanation generation. This is consistent with the stated emphasis on auditability and version-pinned evaluation.

3. Training procedure and supervision

PhyJudge-9B is trained on 2,000 videos, each associated with 2–4 applicable laws and scored by 459 annotators, yielding 37.4K individual Likert scores (Lin et al., 11 May 2026). All Veo-3.1 videos and their labels are held out for evaluation. Supervision is provided as structured JSON “criterion $\rightarrow$ score” pairs, where each score is the human-aggregated mean on a 1–5 scale.

The optimization objective is token-level negative log-likelihood over the serialized JSON label:

$\mathcal{L}(\theta) \;=\; -\mathbb{E}_{(v,q_k,y_k)\sim\mathcal{D}\sum_{t=1}^{|y_k|} \log p_\theta(y_{k,t}\mid y_{k,<t},\,v,\,q_k)\,.$

Only the LoRA parameters $\theta$ and the small vision-language aligner are trainable; the base ViT and LLM remain frozen (Lin et al., 11 May 2026). This training choice confines adaptation to a relatively small parameter subset while preserving the base multimodal stack.

The reported hyperparameters are AdamW, learning rate $= 1\mathrm{e}{-4}$ for LoRA and $2\mathrm{e}{-6}$ for the aligner, 5% warmup, cosine decay, gradient clipping 1.0, and effective batch size 8 with gradient accumulation. Training runs for 1 epoch, approximately 300 steps, taking 1 h 42 m on 4 $\times$ A100 80 GB (Lin et al., 11 May 2026).

This training design suggests a parameter-efficient specialization regime rather than full multimodal re-training. The frozen-encoder, frozen-LLM, and trainable-adapter pattern aligns with the model’s intended role as a domain-specific evaluator rather than a general-purpose generator.

4. Physical-law taxonomy and rubric grounding

PhyGround organizes physical reasoning into 13 laws grouped into three domains, and PhyJudge-9B evaluates them through explicit sub-question operationalization (Lin et al., 11 May 2026). Each law is broken down into 2–3 binary “violation probes” that constrain the judge’s attention. Human annotators see only the main question and a brief note, whereas PhyJudge-9B sees the full sub-question checklist.

Domain	Laws
Solid-Body Mechanics	Gravity, Inertia, Momentum, Impenetrability, Collision, Material
Fluid Dynamics	Buoyancy, Displacement, Flow Dynamics, Boundary Interaction, Continuity
Optical Physics	Reflection, Shadow

The solid-body mechanics laws are defined by prompts such as Gravity (“Do unsupported objects/liquids fail to move downward?”), Inertia (“Do objects change motion without an external force?”), Momentum (“Are post-collision directions/speeds implausible?”), Impenetrability (“Do solids intersect?”), Collision (“Is rebound or deformation inconsistent with impact?”), and Material (“Do objects violate the expected behavior of their material?”) (Lin et al., 11 May 2026).

The fluid-dynamics laws are Buoyancy (“Do objects sink/float against density?”), Displacement (“Does liquid level not rise or overflow correctly?”), Flow Dynamics (“Is bulk motion of liquid implausible?”), Boundary Interaction (“Is splash/deflection wrong upon contact?”), and Continuity (“Does fluid disappear or appear spontaneously?”). The optical-physics laws are Reflection (“Do mirror surfaces display implausible reflections?”) and Shadow (“Are shadows directionally or temporally inconsistent?”) (Lin et al., 11 May 2026).

For Collision, the reported example sub-questions are: “Does one object pass through another at impact?”, “Is rebound direction opposite to the incident vector?”, and “Is rebound speed grossly mismatched to impact speed?” (Lin et al., 11 May 2026). This rubric design operationalizes law-specific failures in observable terms rather than relying on a single undifferentiated physics score.

A plausible implication is that the sub-question checklist serves as an attention-shaping device for the judge. The paper explicitly states that these law-specific checklists steer the judge’s observations and reduce hallucinations (Lin et al., 11 May 2026).

5. Evaluation metrics and comparative performance

PhyJudge-9B is evaluated using aggregate relative bias, signed relative bias, and Spearman’s rank correlation (Lin et al., 11 May 2026). For a judge $J$ and human mean $\alpha = 64$0 over the same scores, aggregate relative bias is defined as

$\alpha = 64$1

where lower $\alpha = 64$2 indicates closer agreement with humans. Signed relative bias is

$\alpha = 64$3

with $\alpha = 64$4 indicating systematic over-scoring and $\alpha = 64$5 under-scoring. Spearman’s rank correlation is given by

$\alpha = 64$6

where $\alpha = 64$7 is the rank difference for the $\alpha = 64$8-th model among $\alpha = 64$9 models.

On a held-out test set of Veo-3.1 videos, with all judges using the same per-law sub-question schema, PhyJudge-9B is reported to have substantially lower aggregate relative bias than Gemini-3.1-Pro (Lin et al., 11 May 2026).

Judge	General	Physics	Overall	Signed bias
Gemini-3.1-Pro (no CoT)	17.3%	15.9%	16.6%	-6.3%
PhyJudge-9B	1.0%	5.5%	3.3%	-1.1%

The paper states that PhyJudge-9B’s average score is within $= 0.05$0 of human means, versus $= 0.05$1 for Gemini-3.1-Pro (Lin et al., 11 May 2026). It also provides a head-to-head optical example on a rotating teapot-mirror scene: the human mean for Reflection is 4.4/5, PhyJudge-9B gives 4.6, and Gemini-3.1-Pro gives 1.0.

The benchmark-level human study reports split-half model-ranking $= 0.05$2 between human subsets (Lin et al., 11 May 2026). This indicates that the human reference labels used for supervision and evaluation are internally stable at the level of model ranking.

6. Auditability, reproducibility, and release artifacts

PhyJudge-9B is explicitly described as physics-specialized and audit-friendly (Lin et al., 11 May 2026). Its design decisions are organized around four elements: open source and fixed weights, per-law decomposition, sub-question rubrics, and version-pinned inference.

Open source and fixed weights mean that the model weights, LoRA adapters, and evaluation code are fully released under a permissive license, with no hidden API calls (Lin et al., 11 May 2026). Per-law decomposition means that instead of a single physics score, the judge emits a separate 1–5 score per law. Sub-question rubrics are included to steer observations and reduce hallucinations. Version-pinned inference means that all hyperparameters, including frame rate, prompt templates, and sub-questions, are codified to ensure reproducibility.

The release includes prompts, human annotations, model checkpoints, evaluation code, prompt templates, sub-question definitions, and the LoRA checkpoint for the 9B-parameter model at the project page (Lin et al., 11 May 2026). This packaging makes the evaluation stack inspectable at the level of both model parameters and scoring protocol.

A plausible implication is that the model is intended to function not merely as a benchmark artifact but as an infrastructure component for repeated evaluation under fixed conditions. The paper’s phrasing around codified inference settings and audit-friendly release supports that interpretation.

7. Findings, scope, and limitations

The benchmark reports that fine-grained, per-law evaluation reveals highly heterogeneous strengths and weaknesses in generative world models (Lin et al., 11 May 2026). The stated example is that open-source models match closed-source in “collision” but lag greatly on “fluid continuity.” This finding follows directly from the law-specific evaluation design of PhyGround and the per-law outputs of PhyJudge-9B.

The same source reports that PhyJudge-9B reduces automated evaluation bias by approximately 80% relative to closed-source baselines, enabling reproducible large-scale physics evaluations (Lin et al., 11 May 2026). Across eight video generators, no model exceeds 3.3/5 on overall physics realism, and solid-body laws remain the hardest, with mean approximately 2.8/5. These values situate PhyJudge-9B within a broader empirical conclusion: current generative world models remain substantially limited in physical realism under rubric-grounded assessment.

Several boundary conditions are also evident. PhyJudge-9B scores queried dimensions as integers from 1 to 5 rather than producing free-form analyses, and its evaluation pipeline depends on explicit law keys and sub-question checklists (Lin et al., 11 May 2026). This indicates that its competence is structured by the taxonomy and rubrics supplied at inference time. A plausible implication is that its strengths are greatest when the target physical phenomenon is well captured by the benchmark’s observable sub-questions.

The release paper states that PhyJudge-9B is expected to accelerate physics-aware model development by providing an open, audit-friendly, law-specific “stress tester” for generative world models (Lin et al., 11 May 2026). In encyclopedic terms, its significance lies in combining a fixed multimodal judge, a rubric-grounded taxonomy of physical laws, and a human-validated evaluation protocol into a single reproducible assessment framework for video-generation systems.