Hierarchical Visual-Grounded Captioning

• HVGC is a paradigm that integrates multi-level visual representations with hierarchical language generation to produce detailed, semantically aligned captions.
• It employs architectures such as graph-based encoders and hierarchical decoders with adaptive attention to align objects, attributes, and actions with linguistic elements.
• Empirical results show that hierarchical grounding significantly improves captioning performance, enhancing metrics and reducing compositional errors.

Hierarchical Visual-Grounded Captioning (HVGC) encompasses a family of models and annotation frameworks that generate descriptive natural language grounded in structured, multi-level visual representations. Unlike flat, purely sequential captioning approaches, HVGC leverages hierarchical structures—across both the visual encoder and the language decoder—to align linguistic units with entities, attributes, relations, and high-level semantic topics. This paradigm enables richer semantic fidelity, improved compositionality, and granular visual-linguistic grounding across images, video, and multimodal data.

1. Core Principles and Formal Models

The HVGC paradigm is founded on aligning the inherent hierarchies of visual scenes and natural language. Core tenets include:

• Hierarchical visual representation: Images or videos are decomposed into multi-level structures such as bounding boxes (objects), regions, fine-grained instances, relationships, and semantic topics. Visual backbones (e.g., Faster R-CNN, Mask R-CNN, ViT) generate region- or instance-level features, while higher-level groupings (composition, relations, scene) are constructed through graph-based or tree-based parsing (Yao et al., 2019, Hsieh et al., 2024, Fei, 2019).
• Hierarchical linguistic generation: Captioning is organized in stages or layers—spanning from phrase (or entity) generation, through predicate/action grounding, to sentence- or paragraph-level coherence. Decoders may employ stacked LSTMs (Song et al., 2018, Tan et al., 2017), Transformer stacks, or Markov Decision Process planners with hierarchical decision-making (Zhang et al., 25 Oct 2025).
• Explicit visual grounding: Attentional or gating mechanisms route each generated word, phrase, or topic to the visual representation most semantically aligned with its linguistic function—nouns to objects, adjectives to attributes, verbs/prepositions to interactions, and global narrative to the holistic scene (Song et al., 2018, Guo et al., 2019, Hsieh et al., 2024).

2. Architectures and Methodologies

HVGC spans a diverse set of encoder-decoder architectures incorporating attention, gating, and graph-based inference. Notable representative designs include:

• Hierarchical LSTMs with Adaptive Attention (hLSTMat):
  • A two-layer LSTM decoder where the bottom layer processes local visual inputs and the top layer refines high-level linguistic context. An adaptive scalar gate $\beta_t$ determines whether the caption generator relies more on visual context $c_t$ (for visual words) or linguistic context $\bar{h}_t$ (for function words) at each time step:

  $\bar c_t = \beta_t\,c_t + (1-\beta_t)\,\bar h_t$ - This gating explicitly “grounds” visual words and bypasses the visual channel for purely syntactic components (Song et al., 2018).

• Graph-Based and Tree-LSTM Encoders:

  • Multi-level features are refined using tree-structured LSTMs (Yao et al., 2019) or Graph Convolutional Networks (GCN) encoding both spatial and semantic relations among regions and instances (Fei, 2019, Guo et al., 2019):

  $$H^{(l+1)} = \sigma\bigl(\widehat{A} H^{(l)} W^{(l)}\bigr),\quad \widehat{A} = D^{-\frac{1}{2}}(A+I)D^{-\frac{1}{2}}$$ - These encoders output multi-scale embeddings, fused into LSTM or Transformer decoders.

• Hierarchical Modular Networks:

  • For video, modular encoders extract per-level representations: entity features (principal objects), predicate/action features (conditioned on objects), and sentence-level semantic embeddings. Each level is explicitly supervised using linguistic projections and contrastive losses (Ye et al., 2021).
• Phrase-based and Planning-Based Decoders:
  • Decoding proceeds from bottom-level noun phrases, through abbreviated sentence decoding, to final assembly (Tan et al., 2017), or through planning in a Markov Decision Process, where tree search and region-guided expansions iteratively refine the caption (Zhang et al., 25 Oct 2025).

3. Visual-Linguistic Grounding and Attention Mechanisms

A defining trait of HVGC is the explicit alignment between visual perceptual elements and words or phrases:

• Category-wise Attentional Gating: Context-gated attention modules first perform soft attention within each VSU (visual semantic unit) category (object, attribute, relation), then use inter-category gates $\beta^c_t$ to select which type is relevant for each word (Guo et al., 2019). The final context vector is a concatenation of weighted attended features across categories.
• Semantic-Graph Alignment: Hierarchical graphs (either tree-structured or DAGs) structure the flow of visual information and encode explicit spatial, compositional, and relational dependencies between regions (Hsieh et al., 2024, Fei, 2019).
• Adaptivity: Gating or soft-attention mechanisms are trained to maximize grounding of content words on visual representations and minimize spurious attention for grammatical tokens (Song et al., 2018, Guo et al., 2019).

4. Multi-Grained Supervision, Losses, and Optimization

HVGC frameworks often incorporate multi-level supervision and loss structures:

• Entity, Predicate, and Sentence Losses: Independent module-level losses are imposed, e.g., Hungarian-matched entity alignment, predicate-level and sentence-level cosine similarity, and cross-entropy for caption generation (Ye et al., 2021).
• Contrastive Multi-Positive Learning: Graph-based captioning uses multiple-positive contrastive objectives, where each node caption and compositional/relation node in the graph serves as a positive instance for the image (Hsieh et al., 2024).
• Topic-Model and ELBO Losses: In topic-guided frameworks, evidence lower bounds jointly regularize semantic topic inference (via variational approximations) and sequence likelihood under the LLM, integrating BoW textual statistics, visual features, and hierarchical topic priors (Guo et al., 2021).
• Reinforcement Learning: Policy optimization with CIDEr or other reward metrics is common in the final training phase, with hierarchical structures shown to increase robustness and performance (Yao et al., 2019, Guo et al., 2019).

5. Applications: Video, Image, and Multimodal Tasks

HVGC principles are instantiated across image captioning, video captioning, paragraph generation, and multimodal synthesis:

• Image Captioning: Models employing hierarchical region- and instance-based parsing, GCNs, or graph-based captions achieve state-of-the-art CIDEr, BLEU, and SPICE scores (e.g., GCN-T: CIDEr 129.7 (Fei, 2019), HIP+GCN: CIDEr-D 130.6 (Yao et al., 2019), GBC-CLIP: Recall@1 60.6 on Flickr-1k (Hsieh et al., 2024)).
• Video Captioning: Hierarchical modular networks align entity, predicate, and sentence-level video semantics, achieving strong performance on MSVD and MSR-VTT (CIDEr 104.0%/51.5%) (Ye et al., 2021).
• Paragraph and Topic-Level Captioning: Coupling deep topic models with visual extractors enables generation of globally coherent, multi-sentence paragraphs grounded at multiple semantic levels (Guo et al., 2021).
• Text-to-Sounding Video (T2SV): HVGC is used to generate disentangled, modality-pure video and audio captions to eliminate modal interference and optimize dual-tower diffusion models, resulting in significant improvements across FVD, FAD, and AV-Align metrics (Guan et al., 3 Oct 2025).

6. Empirical Performance and Ablation

Empirical ablations demonstrate the necessity of hierarchical structures and grounding:

• Component-level ablations: Removal of any hierarchy level (entities, predicates, relations) or gating/attention mechanisms typically yields degradations on CIDEr or related metrics of 5–15 points (Ye et al., 2021, Song et al., 2018, Yao et al., 2019, Guo et al., 2019).
• Composition and Relation Nodes: Explicit modeling of composition and relation nodes in GBC delivers substantial boosts over region-only or flat-captions, with ablation confirming orthogonal semantic gains from hierarchical and relational annotations (Hsieh et al., 2024).
• Planning-based Generation: Top-down semantic refinement with MCTS planners outperforms single-step VLM captioners on compositional and hallucination suppression benchmarks, with ablations confirming the importance of visual-guided parallel expansion and adaptive early stopping (Zhang et al., 25 Oct 2025).

7. Extensions and Future Directions

HVGC’s systematic modeling of visual and linguistic hierarchies opens several frontiers:

• Deeper stacks and heterogeneous hierarchies: Incorporating multiple granularity levels (e.g., pixels → regions → objects → scene → discourse) and modality-specific branches (vision, audio, text) (Song et al., 2018, Guan et al., 3 Oct 2025).
• Graph-Structured and Multi-relation Attention: Fusion with GCNs or structure-aware hierarchical attention enables grounding of complex relational and compositional semantics (Hsieh et al., 2024, Guo et al., 2019, Fei, 2019).
• Plug-and-Play Modularization: Architectures such as HIP and TDSR are “pluggable” into a wide range of encoders and decoders, improving both interpretability and empirical performance (Yao et al., 2019, Zhang et al., 25 Oct 2025).
• Structured Annotation at Web Scale: Automated hierarchical caption datasets (e.g., GBC10M with >10M images) serve as high-fidelity pretraining corpora, improving retrieval, classification, and dense prediction (Hsieh et al., 2024).
• Multimodal and Multi-agent Grounding: HVGC pipelines for disentangling video and audio language conditioning, or extending to cross-sentence and discourse-level grounding, are active research directions (Guan et al., 3 Oct 2025, Song et al., 2018).

In summary, Hierarchical Visual-Grounded Captioning unifies architectural, representational, and annotation strategies to bridge the gap between compositional visual structure and natural-language semantics. Empirical results consistently demonstrate that multi-level grounding and hierarchical modeling deliver substantial improvements across captioning benchmarks, semantic retrieval, and multimodal generation tasks (Song et al., 2018, Ye et al., 2021, Yao et al., 2019, Hsieh et al., 2024, Guan et al., 3 Oct 2025, Zhang et al., 25 Oct 2025, Fei, 2019, Guo et al., 2019, Guo et al., 2021, Tan et al., 2017).