Multimodal Judge: AI-Driven Stepwise Evaluation

• Multimodal Judge is an automated system using MLLMs to assess multi-step reasoning across text, images, diagrams, and formulas.
• The methodology employs a dynamic dual-phase fine-tuning protocol that synthesizes reference solutions and improves error detection across scientific tasks.
• Empirical findings show that instruction-tuning and specialized benchmarks like ProJudgeBench substantially narrow performance gaps between proprietary and open-source models.

A Multimodal Judge is an automated evaluation system—typically instantiated as a Multimodal LLM (MLLM)—designed to assess, diagnose, and provide feedback on multi-step reasoning or generative outputs spanning text, images, diagrams, and formulas. These models act as automated surrogates for human graders, aiming to deliver fine-grained, instruction-following judgments across scientific domains and diverse problem modalities. The role of a Multimodal Judge is central in applications such as scientific problem solving, educational feedback, auditing model reasoning, and reliability assessment of generative systems (Ai et al., 9 Mar 2025).

1. Rationale for Multimodal Process Judges

Process-level, step-by-step evaluation is critical in scientific domains because metrics focused solely on final-answer correctness conceal reasoning deficiencies, such as flawed logic, poor visual interpretation, or improper knowledge application. Fine-grained process evaluation surfaces error locations, error types, and error frequencies, enabling precise diagnosis and targeted model improvement. As MLLMs generate longer and more complex multi-modal solutions, manual annotation becomes infeasible, underscoring the demand for reliable automated judges (Ai et al., 9 Mar 2025).

Prevailing single-modal or single-discipline benchmarks do not generalize to diverse scientific tasks. Previous approaches commonly assessed only where the first error occurred, utilized artificial error injection rather than natural failure modes, and often relied on proprietary, non-reproducible systems.

2. Benchmarking and Taxonomy in ProJudgeBench

ProJudgeBench is the first benchmark dedicated to evaluating the stepwise judgment capabilities of MLLM-based process judges. It comprises 2,400 test cases and 50,118 step-level annotations distributed evenly across mathematics, physics, chemistry, and biology. Each discipline includes both K–12 (primary to high school) and Olympiad-level problem instances, ensuring comprehensive coverage in both content and difficulty.

Each solution is decomposed into an average of 20.8 steps (maximum 470), with explicit step-level annotations: correctness $\in \{0,1\}$, error type if incorrect, and a free-text explanation. The error taxonomy spans:

1. Reasoning Error (logical flaws)
2. Visual Interpretation Error (misread diagrams/graphs)
3. Numerical Calculation Error (arithmetic mistakes)
4. Symbolic Calculation Error (algebraic manipulation mistakes)
5. Knowledge Error (misapplied or missing domain knowledge)
6. Question Understanding Error (misinterpretation of problem requirements)
7. No Solution Provided (refusal, truncation, nondeterminism)

Empirically, 21.6% of all steps are erroneous, with an average of 6.6 errors per solution (Ai et al., 9 Mar 2025).

3. Model Evaluation, Metrics, and Key Findings

Evaluation of MLLM-based judges in ProJudgeBench is twofold:

• Step Correctness Accuracy: A binary classification task indicating per-step correctness.
• Error-Type Classification Accuracy: A seven-way classification for erroneous steps.

Both proprietary MLLMs (e.g., GPT-4o, Gemini-2.0-flash-exp) and open-source contenders (InternVL2.5, Qwen2.5-VL, MiniCPM-V, LLaVA, QVQ) were benchmarked. Representative step-correctness scores:

Model	Step-Corr
GPT-4o	85.10 %
Gemini-2.0-flash-exp	72.51 %
InternVL2.5-38B	78.85 %
Qwen2.5-VL-72B	77.93 %
InternVL2.5-8B	25.58 %
Qwen2.5-VL-3B	11.47 %

A significant accuracy gap (20–60 percentage points) persists between proprietary and open-source models. Open-source models are substantially weaker in visual interpretation and high-difficulty science tasks, with step-correctness for Olympiad-level math/physics falling below 60%. Error-detection blind spots for visual and question-understanding errors were observed (Ai et al., 9 Mar 2025).

4. Large-Scale Instruction-Tuning: ProJudge-173k and Dynamic Dual-Phase Strategy

ProJudge-173k is a 173,354-sample instruction-tuning dataset paralleling ProJudgeBench in disciplines, modality, and error label granularity. It pools:

• 26,249 controlled-injection samples with known synthetic GPT-4o errors
• 93,184 K–12 problems (GPT-4o as judge)
• 53,921 OlympiadBench problems (GPT-4o as judge)

Each instance is labeled with per-step correctness, error type, and an explanation. Example:

Instruction: "Given problem P, final answer A, and student solution S split into steps, evaluate each step..." Response: [ ["Step 1 text", 1, "", ""], ["Step 2 text", 0, "Visual Interpretation Error", "Misread angle in diagram"], # ... ]

The Dynamic Dual-Phase (DDP) fine-tuning protocol alternates between two modes:

• Direct Evaluate: Standard process-evaluation cross-entropy loss.
• Synthesize-then-Evaluate: The judge model solves the problem itself, generates a reference solution, and then uses it for comparison when evaluating a candidate solution. The joint loss integrates both problem solving and evaluation.

This dual-phase protocol is dynamically alternated probabilistically at each batch step, which empirically enhances robustness and prevents protocol overfitting (Ai et al., 9 Mar 2025).

5. Empirical Impact of Fine-Tuning and DDP

Fine-tuning on ProJudge-173k profoundly improves open-source model performance. For example:

Model	Base Accuracy	+FT (Normal)	+FT + DDP
InternVL2.5-8B	25.58 %	83.37 %	84.50 %
Qwen2.5-VL-3B	11.47 %	80.57 %	81.29 %
Qwen2.5-VL-7B	34.65 %	81.91 %	83.72 %

On the held-out OlympicArena set, DDP consistently adds 3–5 percentage points over standard fine-tuning, demonstrating better generalization. Visual interpretation and reasoning error-type detection improve by 30–45 points post fine-tuning, dramatically narrowing the performance gap with GPT-4o (Ai et al., 9 Mar 2025).

6. Applications, Limitations, and Research Directions

Applications:

• Educational systems: automatic stepwise feedback in math, physics, chemistry, biology
• Scientific QA: chain-of-thought vetting in literature review or automated discovery
• Model auditing: step-level audits in safety-critical pipelines (e.g., engineering, medicine)

Limitations and Challenges:

• High annotation cost and limited domain coverage (extensions to engineering, earth science, languages needed)
• Weaknesses in multi-image understanding and unconventional/creative reasoning patterns
• DDP increases inference cost by requiring the judge to produce reference solutions
• Residual judge bias, such as under-identification of question-understanding errors

All ProJudgeBench, ProJudge-173k, prompts, and fine-tuned models will be released to encourage further research in reliable multimodal process evaluation (Ai et al., 9 Mar 2025).

7. Significance and Broader Impact

The Multimodal Judge paradigm underpins a new class of automated evaluators that move beyond end-to-end accuracy in generative AI, enabling step-level, multi-discipline, and multi-modality assessment at scalable human-expert fidelity. ProJudge establishes a comprehensive testbed and large-scale resource for process-level judgment, validating that instruction tuning plus dynamic dual-phase fine-tuning yield open-source judges with competitive reasoning transparency and error-detection capabilities. This methodological rigor is essential for deployment in education, scientific discovery, and high-stakes critical systems, ensuring that large-scale AI applications are trustworthy, explainable, and aligned with expert expectations (Ai et al., 9 Mar 2025).