GLaVE-Cap: Global-Local Video Captioning

• GLaVE-Cap is a video detailed captioning framework designed to generate comprehensive descriptions in multi-object and multi-scene settings by coupling local and global reasoning.
• It features two key modules—TrackFusion for dual-stream local captioning and CaptionBridge for adaptive scene segmentation—to enhance detail and context alignment.
• The framework sets new performance benchmarks with the GLaVE-Bench and GLaVE-1.2M dataset, achieving significant improvements in temporal coherence and descriptive accuracy.

Searching arXiv for the specified paper and a small set of closely related video captioning works to ground the article. GLaVE-Cap is a video detailed captioning framework introduced in "GLaVE-Cap: Global-Local Aligned Video Captioning with Vision Expert Integration" (Xu et al., 14 Sep 2025). It is designed for comprehensive video description in multi-object and multi-scene settings, where standard local-to-global pipelines often produce captions that are less detailed and contextually inconsistent. The framework addresses two issues identified as central to that degradation: the lack of a mechanism to ensure fine-grained local captions, and weak interaction between local and global descriptions. Its architecture combines vision expert integration with explicit global-local coupling through two modules, TrackFusion and CaptionBridge, and is accompanied by a new benchmark, GLaVE-Bench, and a large-scale training dataset, GLaVE-1.2M (Xu et al., 14 Sep 2025).

1. Problem setting and motivating diagnosis

Video detailed captioning in the formulation used by GLaVE-Cap seeks to generate comprehensive descriptions that facilitate video understanding. The paper characterizes most recent approaches as following a local-to-global paradigm: local captions are first generated from clips or keyframes and then summarized into a global caption. According to the paper, this design underperforms in complex videos because it induces three recurrent failure modes: insufficient object attribute and relation description, poor contextual consistency across segments, and information loss during summarization (Xu et al., 14 Sep 2025).

The diagnosis is technically specific. Local captions tend to emphasize inter-frame dynamics and therefore miss static fine-grained details such as attributes, quantities, spatial relations, and text. In addition, local captions produced without global guidance may misinterpret entities or events, and those inconsistencies accumulate across segments. Finally, one-step or naive scene-wise summarization compresses rich local content and omits details; even multi-step strategies may still fail to merge intermediate outputs into a coherent global narrative.

The root causes are identified as reliance on text-only prompts, which limits control over skills such as counting and spatial reasoning, and weak local-global interaction, in which local captions are not guided by global context and global summarization ignores local semantic structure. GLaVE-Cap is therefore positioned not as a simple refinement of local-to-global summarization, but as a framework that explicitly couples local and global reasoning while introducing cross-frame visual prompting.

2. Framework design and end-to-end pipeline

The framework consists of two core modules. TrackFusion enables comprehensive local caption generation by leveraging vision experts to acquire cross-frame visual prompts and by using a dual-stream structure. CaptionBridge establishes local-global interaction by using global context to guide local captioning and by adaptively summarizing local captions into a coherent global caption (Xu et al., 14 Sep 2025).

The pipeline begins with keyframe extraction from a raw video $V$, where CLIP is used to select keyframes $K$. Vision expert prompting and tracking are then applied. Grounding DINO detects objects, while SAM 2 segments them per keyframe with consistent IDs; when a newly detected object has negligible overlap with tracked instances, it receives a new ID, otherwise it inherits the best match. These masks update the SAM 2 video predictor for stable tracking. Visual prompts $M$ are constructed by drawing high-contrast boundaries and numeric IDs at object centers following Set-of-Mark, and textual supplementary information $S$ is produced from object tags and boxes.

Local caption generation is performed in two streams. A differential caption, $C^{\mathrm{diff}}$, captures dynamic changes across adjacent frames, while a detailed caption, $C^{\mathrm{detail}}$, distills static attributes and scene context from individual keyframes and their prompts. These are merged into a local caption, $C^{\mathrm{local}}$, with object IDs retained to avoid reference ambiguity. In the global stage, an overview caption, $C^{\mathrm{overview}}$, is generated from all keyframes with sentence-level frame range annotations, adaptive scene segmentation $SS$ is derived from PySceneDetect initialization plus VLM-guided semantic merging, and scene captions $C^{\mathrm{scene}}$ are concatenated in temporal order into the final video caption $K$0 (Xu et al., 14 Sep 2025).

This organization is significant because the paper frames it as a bidirectional design rather than a unidirectional summarization chain. Global context influences local captioning, and local semantics influence global scene partitioning and summarization.

3. TrackFusion: vision expert integration and dual-stream local captioning

TrackFusion is the module responsible for fine-grained, temporally consistent local descriptions. Its first component is vision expert integration. Grounding DINO performs open-set detection on keyframes, and SAM 2 segments the detected instances. IDs are propagated across frames by overlap-based matching, and the resulting masks and IDs are used to update the SAM 2 video predictor, which enforces long-range consistency under motion, appearance changes, and disappearance (Xu et al., 14 Sep 2025).

The prompts created from these expert outputs are not merely textual annotations. The visual prompt $K$1 draws high-contrast boundaries and numeric IDs at object centers, while $K$2 encodes bounding boxes per ID. The paper states that these prompts significantly improve counting, spatial relations, and ID-consistent references in captions. This suggests that the framework treats object-centric visual grounding as a control interface for the VLM rather than as a post hoc verification mechanism.

TrackFusion then separates local description into two streams. The differential stream, $K$3, focuses on actions, changes, and camera motion between adjacent keyframes, using $K$4 together with overview context. The detail stream, $K$5, compares the original keyframe $K$6 with prompted inputs $K$7 to elicit static attributes, scene context, text, logos, and intra-frame details. Both streams instruct the VLM to attach unique IDs when referring to labeled objects. The merge strategy preserves change descriptions from $K$8 and replaces vague or incorrect static details with $K$9, yielding $M$0 (Xu et al., 14 Sep 2025).

A notable point is what TrackFusion does not introduce. The paper explicitly states that no specialized scoring functions, fusion weights, cross-attention layers, or optimization objectives are defined inside the module beyond prompt-driven VLM generation. The mechanism is therefore architectural and prompt-mediated rather than based on a new learned fusion objective.

4. CaptionBridge: overview-guided captioning and adaptive summarization

CaptionBridge is the module that establishes local-global interaction. It first generates a global overview caption, $M$1, from all keyframes $M$2. This overview summarizes key visual elements, events, and timeline, and includes sentence-level frame range annotations. The overview is then injected into both the differential and detail streams so that local captioning is globally informed (Xu et al., 14 Sep 2025).

The second component is adaptive scene segmentation and scene-wise summarization. PySceneDetect provides initial shot boundaries, but frequent viewpoint shifts can over-segment a video. CaptionBridge therefore prompts the VLM to merge redundant segments using local captions $M$3 and $M$4 into semantically coherent scenes with precise boundaries. For each resulting segment, scene caption generation uses the local captions within the segment, the previous scene caption $M$5 for temporal continuity, and the overview for global consistency. Concatenating scene captions then yields the final global caption $M$6 (Xu et al., 14 Sep 2025).

This module is central to the paper’s argument that information loss is not only a matter of caption quality but also of segmentation policy. A plausible implication is that summarization quality depends on whether scene partitions align with semantic continuity rather than merely with shot transitions. The paper’s formulation of CaptionBridge supports that interpretation by making segmentation itself part of the captioning logic.

An important misconception the paper implicitly rejects is that GLaVE-Cap is simply a longer-context summarizer. The framework instead uses global context to guide local generation and local semantics to refine global structure, which is a stronger coupling than one-step or multi-step caption compression.

5. Benchmarks, datasets, and evaluation protocol

The paper introduces GLaVE-Bench as a benchmark for fine-grained video captioning and GLaVE-1.2M as a large-scale training dataset (Xu et al., 14 Sep 2025). GLaVE-Bench focuses on multi-scene videos and supplies 5× more queries per video than existing benchmarks, with manually curated scene hints intended to reduce ambiguity. GLaVE-1.2M is derived from LLaVA-Video-178K after duration, scene-count, and quality filtering.

Resource	Scope and size	Notable properties
GLaVE-Bench	55 multi-scene videos; average length 85.5 s; ≈8.84 scenes/video; 6,491 question-option pairs; ≈118.02 per video	5× more queries per video than prior benchmarks; adds “Visual-cue” question type; scene hints manually curated
GLaVE-1.2M	15,814 videos; 376 hours; 15,814 fine-grained video captions; 1,176,410 QA pairs	Source: LLaVA-Video-178K; filtering by duration, scene count, and GPT-4o screening; Apache 2.0

GLaVE-Bench uses an evaluation protocol in which videos are replaced with captions and GPT-4o judges answers with scene hints; when caption information is insufficient, the output is “E. Not mentioned.” The benchmark defines the metrics

$M$7

where $M$8, $M$9, and $S$0 denote counts of correct, wrong, and “Not mentioned” responses (Xu et al., 14 Sep 2025).

The paper also specifies implementation details for a student model, GLaVE-7B. Qwen2.5-VL-7B serves as the base model, training uses MindSpeed-MM across mixed image and video QA data, and for the GLaVE-1.2M configuration the data include 1,146,818 open-ended QA items while captions are excluded for length concerns. Training is conducted on 64 Ascend 910B2 devices, with GRAD_ACC_STEP=16, a global batch size of 1,024, and 2,000 iterations over approximately 60 hours. The paper does not define a bespoke loss for TrackFusion or CaptionBridge and does not report explicit hyperparameters beyond the MindSpeed-MM default configuration.

6. Empirical performance, analysis, and limitations

Across four benchmarks, the paper reports that GLaVE-Cap achieves state-of-the-art performance (Xu et al., 14 Sep 2025). On GLaVE-Bench, it reaches $S$1 with GPT-4o, together with $S$2 and $S$3; with Qwen2.5-VL-72B, the reported accuracy is 63.67. The method is described as particularly strong on multi-scene benchmarks such as GLaVE-Bench and Video-MME-S, where adaptive summarization and overview guidance are presented as decisive factors.

The student model analysis further supports the role of the dataset. GLaVE-7B attains the highest reported GLaVE-Bench accuracy, 79.28, and also reaches top results in other reported columns, including 62.96, 67.78, and 66.33. The paper notes a slight decline on Video-MME and MVBench relative to the base model, attributing it to catastrophic forgetting, but states that the decline is smaller than in other fine-tuned models, which it interprets as evidence of richer supervision from GLaVE-1.2M.

The ablation studies isolate the major mechanisms. Removing the dual-stream design reduces GLaVE-Bench accuracy by 5.01%. Removing adaptive scene-split causes a 21.70% accuracy drop, which the paper presents as confirmation that summarization loss is a major bottleneck. Removing the overview caption harms temporal coherence, although with a smaller accuracy drop. Removing visual prompts has modest overall effect on GLaVE-Bench but substantially degrades counting-related subtasks, including MVBench Number Count from 51.33 to 44.17 and NEXT-QA Moving Count from 62.94 to 58.07. Additional analyses report that TrackFusion enriches detailed words by +30% at the local level and +18% at the video level relative to an ablated dual-stream setting, that overview captions reduce inconsistencies per 1k words, and that adaptive scene-split improves preservation of local visual descriptions in the final video caption by approximately +20%.

Qualitative examples reinforce the same pattern. In the multi-scene cartoon “Pony Express,” the framework is reported to generate an approximately 241-word coherent scene-level description that retains rider attire, horse motion, mailbag labeling, expressive visual style, on-screen text, and timeline transitions that baselines omit. In the grocery store example, it identifies the celebrity Jyoti with context, tracks multiple people by ID, and captures text and spatial relations. In the film example, it tracks helicopter scenes, text, weather, lighting, object manipulations, and action transitions.

The paper also states several limitations. The current framework depends on the VLM’s capacity to process many visual inputs in a single inference, and in scenes with numerous objects some elements may still be missed. A stated future direction is to shift from keyframe-based to object-centric local captioning with tracked objects and multi-round interactions between experts and the VLM. Reproducibility resources are to be open-sourced, including source code, model weights, benchmark, and dataset; the reported licenses include Apache 2.0 for code and dataset components, BSD-3-Clause for PySceneDetect, and academic-use constraints for benchmark videos derived from Video-MME (Xu et al., 14 Sep 2025).