Grounded Video Caption Generation

• Grounded video caption generation is the process of producing captions that explicitly link noun phrases to corresponding spatio-temporal regions in videos.
• It leverages attention, transformer, and graph-based models to align visual evidence with linguistic output, thereby reducing hallucinations.
• Evaluation employs benchmarks like ActivityNet-Entities and metrics such as CIDEr and AP50 to assess both caption quality and grounding accuracy.

Grounded video caption generation refers to the task of generating natural language descriptions of video content such that specific noun phrases or entities in the caption are explicitly and accurately linked (grounded) to spatio-temporal regions in the video. This approach enables not only descriptive coverage of a scene but also fine-grained localization of entities (objects, people, roles) beyond simple global video summarization. Grounded captioning increases interpretability, reduces hallucinations, and supports downstream tasks (such as video question answering, situation recognition, and video retrieval) that require alignment between language and visual evidence.

1. Task Definition, Scope, and Benchmarks

In grounded video caption generation, the input is a raw or pre-processed video sequence (typically untrimmed or segment-level). The output is a caption (or paragraph) in natural language, together with a mapping from a subset of noun phrases (NPs) in the caption to spatial regions or spatio-temporal tubes in the video. Some task and dataset definitions extend this grounding to verb-role interactions, providing a structured mapping from semantic roles (e.g., "who," "what," "where") to video cuboids or tracks (Kazakos et al., 2024, Zhou et al., 2018, Khan et al., 2022).

Benchmark datasets:

• ActivityNet-Entities: Over 157.8k NP-to-bounding-box annotations for ∼52k video segments (Zhou et al., 2018).
• GROC (GROunded Video Caption Generation) test set: 1,100 clips with per-frame, per-NP grounding for dense evaluation (Kazakos et al., 2024).
• HowToGround / HowToGround1M: Large pseudo-labeled training sets for pre-training, derived from instructional video corpora (Kazakos et al., 13 Mar 2025, Kazakos et al., 2024).
• iGround: 3,500 videos, ≈420k manually annotated bounding boxes for high-quality evaluation (Kazakos et al., 13 Mar 2025).

Typical metrics include METEOR, CIDEr for caption quality, and AP50, mIoU, or F1-based scores for grounding accuracy. Grounding is measured as the proportion of entities correctly linked to boxes/tubes (IoU > 0.5) and the fraction of generated noun phrases that can be located in the visual stream (Zhou et al., 2018, Kazakos et al., 2024).

2. Model Architectures and Grounding Mechanisms

Grounded video captioning models combine video region/object proposal extraction, region feature encoding, and language modeling with an explicit or implicit alignment mechanism.

Classic attention-based models:

• Spatio-temporal attention LSTMs: At every decoding step, the captioner softly attends over region or tube proposals and conditions the next word prediction on a context vector $z_t$ that represents attended regions. Grounding is available as a by-product of attention scores $\{\beta_{ti}\}$ without explicit supervision (Zanfir et al., 2016).
• Scene-graph or object-interaction architectures: Higher-order relations among detected ROI features (objects/relations/attributes) are explicitly modeled (e.g., graph convolutions or interaction modules) before decoding; these modules support relational grounding and allow the LLM to disambiguate entity references, actions, and their visual support (Zhang et al., 2021, Ma et al., 2017).

Transformer-based and LLM-fused models:

• Two-stream architectures (GROVE, VideoGround): Feature tokens from multiple frames are encoded by frozen ViT backbones (often CLIP-L), pooled, and projected into the embedding space of a LLM. Detection/grounding queries are injected by special tokens (e.g., <DET>) produced by the LLM, and a frozen SAM encoder decodes these tokens into per-frame bounding box predictions via cross-attention (Kazakos et al., 13 Mar 2025, Kazakos et al., 2024).
• Object-region query models: Transformers fuse global video and localized object tokens and allow cross-attention from semantic role queries to object features, yielding fine-grained, compositional region-word grounding for each semantic role (Khan et al., 2022).

Semi-parametric and scalable models:

• Frame retriever + generator split: To handle long videos, models such as SeViT treat the video as an external data store, retrieving only a query-relevant subset of frames for detailed language grounding and fusion via marginalization or FiD ("fusion-in-decoder") schemes. This enables explicit selection of visual support for long-form captioning (Kim et al., 2023).

3. Datasets, Annotation Protocols, and Evaluation

Explicit evaluation of grounding requires datasets in which (at least) certain noun phrases or semantic roles are annotated with bounding boxes (per frame) or temporally consistent tubes.

Dataset Properties