LLM-as-a-Judge Protocol

• LLM-as-a-Judge Protocol is an automated evaluation framework that uses large language models as impartial judges to assess video description accuracy in multimodal tasks.
• It employs a dual-checklist framework to separately gauge similarities and differences, yielding fine-grained error analysis with high human–LLM agreement.
• Its applications in content forensics, video editing verification, and change-log generation demonstrate practical insights into dynamic scene analysis.

The LLM-as-a-Judge protocol is an automated evaluation framework that operationalizes LLMs as impartial “judges” for assessing the quality and accuracy of model-generated descriptions in complex multimodal tasks. This protocol was formalized in the context of the ViDiC-1K benchmark for Video Difference Captioning, specifically for evaluating the comparative reasoning abilities of Multimodal LLMs (MLLMs) when analyzing and describing differences and similarities between video pairs (Wu et al., 3 Dec 2025).

1. Motivation and Context

Existing evaluation methods for comparative video analysis—such as simple matching of open-ended captions—are inadequate for capturing nuanced compositional, spatial, and temporal distinctions critical to dynamic scene understanding. While prior work in Image Difference Captioning (IDC) addressed the comparison of static images, these approaches were insufficient for dynamic content due to their inability to account for motion continuity, event evolution, and editing consistency across time. The LLM-as-a-Judge protocol was introduced to address these limitations, enabling reliable, fine-grained, and scalable assessment across multiple annotated dimensions in the ViDiC-1K dataset.

2. Protocol Mechanics and Workflow

The protocol comprises a multi-stage evaluation process involving several system components:

• Model Under Test ($M$): Generates a natural-language, free-form description $D$ of differences and similarities for a given video pair $(A, B)$.
• Checklist Query ($Q$): A set of binary (“yes”/“no”) questions spanning seven comparative dimensions (subject, style, background, camera work, motion, position, playback).
• Judge LLM ($J$): Receives $D$ and $Q$ as input, then outputs an answer set $A_J$ in JSON format, where each response includes “yes” or “no” plus a brief rationale.
• Ground Truth ($A_{GT}$): Derived through expert annotation, providing reference answers for each question.
• Scoring: Results are calculated by measuring agreement between $A_J$ and $A_{GT}$.

A distinctive feature is the dual-checklist framework, which separates similarity scoring from difference detection to enable nuanced error analysis.

3. Evaluation Taxonomy and Accuracy Metrics

ViDiC-1K’s evaluation is structured as follows:

• Similarity Questions ($Q_{\text{sim}}$): Items posed inversely (e.g., “Are these attributes different?” with ground truth “no”). Only hallucinations—erroneous reporting of differences—are penalized; omission of similarities is non-penalizing.
• Difference Questions ($Q_{\text{diff}}$): Items framed as factual assertions of change (e.g., “Does B have X that A lacks?” with ground truth “yes”). Omission or denial of true differences are penalized.

Let $|Q_{\text{sim}}|$ and $|Q_{\text{diff}}|$ denote the counts of similarity and difference items, respectively. The accuracy metrics are:

$$A_{\text{sim}} = \frac{1}{|Q_{\text{sim}}|} \sum_{i\in Q_{\text{sim}}} \mathbf{1}(A_{J,i} = A_{GT,i})$$

$$A_{\text{diff}} = \frac{1}{|Q_{\text{diff}}|} \sum_{j\in Q_{\text{diff}}} \mathbf{1}(A_{J,j} = A_{GT,j})$$

$$A_{\text{all}} = \frac{1}{|Q|} \sum_{k\in Q} \mathbf{1}(A_{J,k} = A_{GT,k})$$

where $Q = Q_{\text{sim}} \cup Q_{\text{diff}}$.

4. Implementation Details and Human–LLM Agreement

The protocol implementation for ViDiC-1K specifies GPT-5-Mini as the judge model. The evaluation pipeline achieves the following human–LLM agreement rates:

• Overall agreement: 95.2%
• Similarity questions: 95.9%
• Difference questions: 94.97%

These figures indicate high reliability as a surrogate for human annotation. Five repeated evaluations on a single description (using Gemini 2.5-Pro as $M$) varied less than 0.6% in accuracy, confirming protocol stability.

The answer format includes both a categorical decision and a succinct rationale, facilitating both quantitative and qualitative analyses of failure modes.

5. Observed Trends, Limitations, and Pathologies

Application of the LLM-as-a-Judge protocol to ViDiC-1K revealed several significant trends and failure modes in state-of-the-art MLLMs:

• Similarity scores consistently exceed difference scores across models (e.g., GPT-4o: Sim 81.12% vs. Diff 39.14%).
• Style recognition is relatively strong ($>75\%$), while detection of camera work and playback technique is notably weaker ($<50\%$ even for leading models).
• Use of “Thinking” mode in certain models increases difference detection accuracy but induces more hallucinations, reducing similarity accuracy.
• Pathologies include hallucinated changes, self-contradictions, omission or vague description of differences, and failure to recognize compositional elements such as camera perspective or depth-of-field.

This suggests that current architectures lack mechanisms for robust comparative spatio-temporal reasoning, especially for subtle motion and compositional differences.

6. Challenges and Future Directions

Several limitations constrain the scalability and generality of the current protocol:

• Dataset scale: ViDiC-1K’s 1,000 curated video pairs suffice for evaluation but are inadequate for large-scale model training.
• Annotation complexity: Human annotation requires significant expertise—only 16.32% of LLM-drafted checklist items were retained verbatim, with extensive manual revision.
• Checklist granularity: Maintaining objectivity with binary questions competes with the richness of open-ended language required for full semantic coverage.

Planned extensions include expansion toward large-scale instruction-tuning corpora, architectural innovation for comparative temporal reasoning, and development of joint pipelines for similarity and difference recognition.

7. Applications and Significance

The LLM-as-a-Judge protocol supports diverse downstream tasks requiring reliable automated evaluation of video comparison tasks:

• Content forensics: Identification of forged or manipulated edits.
• Video editing verification: Ensuring edit fidelity with respect to natural language prompts.
• Automated change-log generation: Supporting transparent post-production pipelines.
• Performance feedback: Fine-grained motion analysis in sports and rehabilitation.
• Intelligent surveillance: Semantic alerting for relevant activity detection.

By leveraging checklist-based, LLM-driven evaluation, the protocol establishes a rigorous, generalizable foundation for benchmarking and advancing multimodal comparative reasoning systems (Wu et al., 3 Dec 2025).