GLaVE-Bench: Multi-Scene Video Caption Benchmark
- GLaVE-Bench is a benchmark for evaluating video captioning systems under complex multi-scene conditions with dense query coverage.
- It employs manually verified scene-hints to disambiguate scene-specific references, ensuring precise assessment of caption details.
- The benchmark measures local detail accuracy and global contextual consistency, guiding improvements in caption generation models.
GLaVE-Bench is a purpose-built benchmark for evaluating video detailed captioning systems under realistic, multi-scene conditions where fine-grained visual details and global contextual consistency both matter. It is introduced alongside the GLaVE-Cap framework to fill a gap the authors identify in current video captioning evaluation: existing benchmarks either contain too few scenes per video, too few questions per video, or lack mechanisms to disambiguate scene-specific references, which makes it hard to fairly assess whether a caption truly captures the detailed content of a video. In this setting, “reliable evaluation” for detailed video captions means testing whether a caption can serve as a proxy for the video to answer diverse, content-grounded questions (Xu et al., 14 Sep 2025).
1. Motivation and evaluation target
As discussed in the paper’s Introduction and Sec. 4.1, most recent captioning pipelines follow a local-to-global paradigm, but they often produce captions that omit fine-grained details, lose consistency across scenes, or suffer information loss in summarization. At the evaluation side, prior benchmarks, including MVBench, Video-MME, VidCapBench, and NEXT-QA, typically offer only a handful of questions per video and often focus on single-scene or short clips. This sparsity leads to shallow assessment of a model’s ability to capture objects, actions, attributes, scenes, relations, events, and text consistently across a multi-scene narrative (Xu et al., 14 Sep 2025).
GLaVE-Bench addresses this by using multi-scene videos, providing dense, diversified question coverage per video across visual dimensions, and introducing manually verified scene-hints that disambiguate the intended scene for each question without leaking answers. The “5X more queries per video” claim refers to the average number of question-option pairs per video in GLaVE-Bench compared to prior caption-based evaluation benchmarks. Concretely, GLaVE-Bench averages 118.02 questions per video, versus 16.55 for VidCapBench, which is more than a fivefold increase.
This design suggests that the benchmark is not limited to measuring surface-level descriptiveness. A plausible implication is that it is intended to diagnose whether a caption preserves both local evidence and cross-scene continuity when the video itself is no longer available at evaluation time.
2. Dataset composition and scope
GLaVE-Bench adopts the Video-MME Short subset as the source of video content because of its high resolution and clear thematic focus. From this pool, the authors manually select 55 multi-scene videos with stable camera motion and well-defined topics. It is a test-only benchmark (Xu et al., 14 Sep 2025).
| Property | Value |
|---|---|
| Number of videos | 55 |
| Average video length | 85.5 seconds |
| Average scenes per video | 8.84 |
| Total number of question-option pairs | 6,491 |
| Average queries per video | 118.02 |
| Annotation | Auto & Manual |
Captions and questions are first generated automatically via the GLaVE-Cap pipeline and refined by GPT-4o, followed by extensive human verification totaling 230 rater-hours. Scene-hints are provided and verified for each scene to reduce ambiguity.
Table 1 in the paper contrasts GLaVE-Bench with NEXT-QA, MVBench, AutoEval-Video, TempCompass, Video-MME-S, and VidCapBench. GLaVE-Bench exceeds prior work in multi-scene coverage, query density, and scene-hint provision. Specifically, it reports average 8.84 scenes per video, 6,491 question-option pairs total, and manually verified hints for each scene to prevent ambiguity due to changes in objects or attributes across scenes. These elements improve evaluation reliability and allow more precise diagnosis of caption quality, including hallucinations and omissions.
3. Query taxonomy and scene-hint design
The benchmark’s queries span diverse dimensions such as objects and attributes, actions, scenes and environment, spatial relations and directions, temporal relations and order, causal and plot understanding, and text recognition and subtle cues. The taxonomy builds on the 16 question types defined in LLaVA-Video [46], augmented with a new Visual-cue type to target subtle but informative elements (Xu et al., 14 Sep 2025).
Questions are partitioned into scene-level queries and global-level queries. Scene-level queries comprise 12 description-oriented categories, including description-object, fine-grained action, count, attributes, and spatial positioning. For each scene , at most one question per category is generated from the scene’s caption, covering almost all scenes and drilling into local details. Global-level queries cover the remaining categories and focus on holistic comprehension and reasoning across the entire video, including time order, temporal dependencies, causal reasoning, and plot understanding. Up to five question-option pairs per category are constructed based on the full video caption.
Scene-hints are central to this design. The benchmark provides manually verified scene-hints that disambiguate the intended scene for each question without leaking answers. Representative examples include “1: A man is doing pull-ups.”, “5: A man is doing sit-ups.”, and “6: A sign is displayed in the center.” These hints uniquely identify scenes without revealing the answer. In the gym workout example, the question “Where is the person in the background located compared to the man who is exercising?” can yield different answers depending on the referenced scene; the hint anchors the question to the intended context.
The examples given in the paper illustrate both local and global probing. Scene-level questions include “What are the characteristics of the cartoon character?”, “How many cacti are in the whimsical desert setting at the start?”, and “Where is the young boy positioned in the frame?” Global-level questions include “What scene follows the radio broadcast scene?” and “How is humor incorporated into the narrative?” This suggests that GLaVE-Bench is designed to evaluate not only recognition of visual entities and attributes but also temporal order and narrative-level interpretation.
4. Annotation pipeline and quality control
The annotation pipeline is described in Sec. 4.1 and Appendix B. First, using GLaVE-Cap, the authors produce fine-grained, scene-aware captions with global-local alignment and derive concise scene-hints per scene. Second, initial QA pairs are generated from these captions: scene-level items are generated from scene captions with at most one per type per scene, and global-level items are generated from the full caption with up to five per type (Xu et al., 14 Sep 2025).
Third, refinement with GPT-4o is applied. Non-atomic questions are split into atomic ones, and wording is neutralized to reduce answer leakage. GPT-4o then constructs 4-option multiple choice answers with one correct option and three stylistically plausible distractors. Fourth, human verification is conducted for 230 rater-hours. Annotators validate scene segmentation and hints for clarity and minimal leakage, check question-option correctness, eliminate ambiguity, ensure scene alignment, and revise or remove defective items. Protocols and full guidelines are provided as “Annotation.md” in the supplementary material.
The benchmark characterizes its annotation as “Auto & Manual,” but the verification stage is extensive rather than nominal. A plausible implication is that the benchmark is intended to preserve the scale advantages of automatic generation while using human review to control ambiguity and answer leakage, both of which are especially problematic in multi-scene evaluation.
5. Evaluation protocol and reported metrics
The benchmark evaluates a captioning system by replacing the original video with the system’s generated caption and then answering questions from GLaVE-Bench using GPT-4o as the judge. Each evaluation item includes the caption under evaluation, a question with four options labeled A–D, and a scene-hint for scene-level questions. Hints from adjacent scenes may optionally be provided to further reduce ambiguity. If the caption does not contain enough information to answer a question, GPT-4o must select “E. Not mentioned” (Xu et al., 14 Sep 2025).
The benchmark reports three metrics. Let , , and denote the number of correct selections, wrong selections, and “E-type” responses, respectively. Then:
High N.M. indicates insufficient coverage in the caption. High Hall. indicates the caption contains fabricated or incorrect details relative to the video.
To mitigate variability from GPT-based judging, each question is evaluated three times, and the authors report minimal fluctuation across runs. The benchmark also includes prompt templates for evaluation to standardize the judge’s behavior. Preprocessing, tokenization, and inference settings such as beam search are not central in this protocol because judging is handled by GPT-4o over textual captions.
One common misconception is that the benchmark directly evaluates video-question answering models from raw video. It does not: the protocol explicitly replaces the video with the system’s caption. The object of evaluation is therefore the informational sufficiency, correctness, and coherence of the caption as a proxy for the video.
6. Relationship to GLaVE-Cap, GLaVE-1.2M, and empirical findings
GLaVE-Bench was designed to stress-test the capabilities that GLaVE-Cap explicitly targets: capturing fine-grained local details, including objects, attributes, relations, and text, while preserving global contextual consistency across scenes. The benchmark’s multi-scene nature, dense query coverage, and scene-hints align with GLaVE-Cap’s TrackFusion and CaptionBridge modules (Xu et al., 14 Sep 2025).
TrackFusion enables comprehensive local caption generation by leveraging vision experts to acquire cross-frame visual prompts, coupled with a dual-stream structure. The dual-stream local captioning is described as a differential stream for dynamics and a detailed stream for static attributes. CaptionBridge injects global context into local captioning via an overview caption, performs adaptive scene segmentation, and summarizes scene-level captions into a coherent video-level description.
The authors also construct GLaVE-1.2M, a large-scale training corpus of 15,814 videos (376 hours) with 1,176,410 QA pairs, using GLaVE-Cap and the same QA generation methodology. Although GLaVE-1.2M is a training set and GLaVE-Bench is an evaluation benchmark, their alignment in query design and visual dimension coverage helps demonstrate how fine-grained supervision can improve student models tuned for detailed video understanding.
On GLaVE-Bench, GLaVE-Cap achieves state-of-the-art performance across all backbones tested. With GPT-4o backbone, it reports Acc. 67.68, Hall. 16.04, and N.M. 19.40. With Qwen2.5-VL-72B backbone, it reports Acc. 63.67, Hall. 17.42, and N.M. 22.90. These results surpass LVD-2M, AuroraCap, ShareGPT4Video (DiffSW), LLaVA-Video (multi-level captioning), and Vript (scene-wise summarization). The paper states that GLaVE-Cap’s advantage is particularly strong on GLaVE-Bench, which emphasizes fine-grained description and multi-scene consistency.
Ablation results on GLaVE-Bench further tie benchmark difficulty to the proposed architecture. Removing the dual-stream structure reduces accuracy by 5.01%. Removing adaptive scene split reduces accuracy by 21.70%. Removing the overview caption severely harms temporal coherence qualitatively, even if aggregate accuracy drops less. Removing visual prompts has limited impact on GLaVE-Bench headline metrics but significantly hurts tasks involving counting across datasets. Student model analysis shows that a student model, GLaVE-7B, fine-tuned on GLaVE-1.2M achieves the best performance on GLaVE-Bench compared to open-source baselines including LLaVA-OneVision-7B, LLaVA-Video-7B, and ShareGPT4Video-8B.
7. Practical usage, limitations, and bias considerations
For fair comparison, the evaluation scripts and protocol specify four steps: replace the video with the system’s caption; supply the GLaVE-Bench question-option pairs and scene-hints; use GPT-4o to select among A–D, or “E. Not mentioned” if the caption lacks the needed information; and run each question three times while reporting Acc., Hall., and N.M. as defined above (Xu et al., 14 Sep 2025).
The authors state that the source code, model weights, benchmark, and dataset will be open-sourced. Licensing for assets used in construction is detailed in Appendix P. The test benchmark follows Video-MME’s license, specified as academic use only, no commercial use, and no redistribution. Code is released under Apache 2.0, with examples including PySceneDetect under BSD-3-Clause and Grounded-SAM2 under Apache 2.0. The training dataset, GLaVE-1.2M, is built on LLaVA-Video-178K under Apache 2.0, and the pretrained model Qwen2.5-VL is also Apache 2.0.
The paper also states several limitations. While GLaVE-Bench improves coverage and reliability, it remains constrained by the upstream capacity of VLMs and object tracking experts. In crowded scenes with many objects, even the GLaVE-Cap pipeline may fail to capture all elements. The authors plan to move from keyframe-based to object-centric local captioning, combining tracking with multi-round expert–VLM interactions to further improve detail fidelity.
Biases and societal impacts are addressed in Appendix O and P. Video data is sourced from public datasets, specifically Video-MME and LLaVA-Video-178K, which may include personally identifiable information. Users must adhere to original licensing and ethical standards. As with any VLM-based pipeline, there is risk of misuse, including generating misleading content, so usage guidelines and safety considerations are emphasized.
Taken together, these details characterize GLaVE-Bench as a test-only, multi-scene caption evaluation benchmark built to measure both fine-grained coverage and global consistency. Its combination of dense query coverage, manually verified scene-hints, explicit omission and hallucination metrics, and caption-as-proxy evaluation protocol makes it a benchmark for detailed video captioning rather than a generic video understanding suite.