---
title: E-commerce Hierarchical Video Captioning
url: https://www.emergentmind.com/topics/e-commerce-hierarchical-video-captioning-e-hvc
type: topic
---

# E-commerce Hierarchical Video Captioning

E-commerce Hierarchical Video Captioning (E-HVC) is a research paradigm for generating structured, product-aware textual narrations from consumer-driven e-commerce videos. E-HVC addresses the challenges of dense, multimodal video content, requiring models to recognize fine-grained product actions and attributes, align them with spoken or written narrative evidence, and synthesize multi-level descriptions or titles adapted for retrieval and recommendation scenarios. E-HVC systems systematically integrate visual, linguistic, and metadata signals through hierarchical architectures, and operate at both event-level granularity and broader narrative abstractions tailored to e-commerce domains [2006.13608][2601.07366].

## 1. Problem Formulation and Dataset Construction

E-HVC distinguishes itself from generic video captioning by targeting the e-commerce context, where consumer-generated videos are highly information-dense and product-centric. The central task involves generating structured textual outputs—ranging from single-sentence titles to temporally grounded narratives—by leveraging multimodal evidence inherent to e-commerce data.

The E-HVC dataset introduced in [2601.07366] comprises \(N = 146,000\) videos (mean length ≈ 60 s, total ≈ 2,433 hours), and a dedicated benchmark (E-HVC-Bench) of 1,852 videos spanning 13 product categories. The annotation schema provides dual levels of granularity:

- **Temporal Chain-of-Thought (TCoT):** Ordered sequences of event annotations \(E_i = (t_i^{\text{start}}, t_i^{\text{end}}, d_i)\) anchored to video intervals, capturing host actions, product demonstrations, or scene transitions.
- **Chapter Summaries:** Partitions of events into contiguous chapters \(C_j = ([c_j^{\text{start}}, c_j^{\text{end}}], \tau_j, s_j)\) with precise time boundaries, concise titles \(\tau_j\), and thematic summaries \(s_j\).

Annotation involves a three-stage process: (i) Automatic Speech Recognition (ASR) text enhancement and adaptation to product domains, (ii) frame-level visual description via advanced vision-language models, and (iii) hierarchical reasoning using large-scale multimodal transformers to generate and refine event and chapter annotations. Data curation applies quality filters and manual corrections for the benchmark validation [2601.07366].

## 2. Hierarchical Modeling Approaches

E-HVC architectures universally adopt hierarchical processing:

- **Granular Interaction Modeling:** At the base level, video content, comments, and attributes are processed as separate graphs. For instance, the approach in [2006.13608] constructs three dedicated graphs—video landmark graphs \(G_v\), narrative comment graphs \(G_c\), and attribute graphs \(G_a\)—with nodes encoding frame landmarks, syntactic word tokens, and product attribute entries respectively. Edges are established to capture spatial-temporal or syntactic relations, with learnable edge weights.
- **Cross-Modal Aggregation:** Global-local aggregation modules integrate information across graphs, leveraging node-level and graph-level attention to form holistic representations (e.g., fusing \(G_a\) and \(G_c\) to yield \(G_{ac}\), then integrating with \(G_v\) to produce \(G_{vac}\)).
- **Abstraction/Narrative Layer:** At the upper level, frame features and graph-aggregated node embeddings are fused. Typical architectures employ gated recurrent units (GRUs) or transformer decoders, with attentive mechanisms that reference both frame-wise dynamics and cross-modal context. Generation modules produce titles or multi-paragraph summaries, with explicit mechanisms for temporal alignment and hierarchical structuring [2006.13608][2601.07366].

## 3. Scene-Primed ASR-Anchored Compression (SPA-Compressor)

E-commerce videos tend to be fast-paced with long sequences of dense visual tokens, challenging conventional MLLM (Multimodal Large Language Model) architectures due to self-attention bottlenecks. The SPA-Compressor [2601.07366] introduces a hierarchical token compression strategy:

- **Vision–ASR Fusion:** ASR outputs and visual features are cross-attended, integrating text and visual evidence.
- **SceneFusionAggregator:** Learnable scene queries summarize global, scene-level representations for each sequence segment.
- **EventDetailExtractor:** Event-level queries extract temporally grounded event details via cross-attention to ASR and scene tokens.

Mathematically, given input \(\mathcal{X} = \{\mathbf{V}, \mathbf{A}, \mathbf{T}\}\), SPA-Compressor constructs hierarchical representations \(\mathbf{H} \in \mathbb{R}^{B \times (S + N(1+E)) \times D}\) with configurable token budgets \((S, E)\) controlling compression. For a recommended setting \((S=64, E=32)\), the compression ratio is \(\rho \approx 0.174\), resulting in ≈ 82.6% reduction in token count and a theoretical \(33\times\) speedup in self-attention cost.

## 4. Training Objectives and Optimizations

Both graph-based and transformer-based E-HVC models are optimized using multi-task objectives:

- **Token-Level Cross-Entropy:** For each ground-truth event description \(d_i\), chapter title \(\tau_j\), and chapter summary \(s_j\), the model minimizes the cross-entropy loss with predicted outputs:
  \[
  \mathcal{L} = \lambda_1 \sum_i \mathrm{CE}(d_i, \hat d_i) + \lambda_2 \sum_j [\mathrm{CE}(\tau_j, \hat\tau_j) + \mathrm{CE}(s_j, \hat s_j)]
  \]
- **Coverage Loss and Regularization:** Models may include terms to penalize repeated mass in word probabilities or apply \(\ell_2\) regularization to stabilise training [2006.13608][2601.07366].
- **Optimization Schedules:** State-of-the-art training employs AdamW, staged fine-tuning phases (task adaptation, compression-aware retraining), and large-scale distributed setups (e.g., batch size 64 across 16 A100 GPUs).

## 5. Evaluation Protocols and Results

Performance is evaluated using reference-based and retrieval-based metrics:

- Reference-based: BLEU-4, METEOR, ROUGE-L, and CIDEr for captioning quality, BERTScore and SODA_c for narrative factuality and semantic alignment [2601.07366].
- Retrieval-based: Recall@1/5/10, Median Rank, and MRR are used to assess the match between generated and ground-truth narrations.
- Human evaluation: Fluency, diversity, and grounding assessed on standardized sample sets.
- Online AB testing: CTR improvements observed in real e-commerce platforms (e.g., +9.9% in Mobile Taobao) [2006.13608].

The HiVid-Narrator framework sets state-of-the-art results on E-HVC-Bench (e.g., SODA_c 14.48, CIDEr 1.45, METEOR 32.01, BERTScore 74.25), outperforming prior MLLMs such as InternVL3, Qwen2.5-VL, and Keye-VL. Token compression via SPA-Compressor achieves efficiency gains (≈ 82.6% reduction) with no loss in narrative quality [2601.07366].

| Model                       | SODA_c | CIDEr | METEOR | BERTScore |
|-----------------------------|--------|-------|--------|-----------|
| InternVL3 (8B)              | 11.98  | 1.04  | 27.38  | 67.75     |
| Qwen2.5-VL (7B)             | 12.27  | 1.12  | 27.74  | 68.83     |
| Keye-VL (8B)                | 12.83  | 1.34  | 28.29  | 69.23     |
| HiVid-Narrator (w/ SPA)     | 14.48  | 1.45  | 32.01  | 74.25     |

Ablation indicates the importance of TCoT annotation and token compression: removing TCoT or reducing scene/event tokens both degrade all metrics.

## 6. Insights, Limitations, and Prospects

Ablation studies across both frameworks confirm that hierarchical modeling—explicitly capturing event temporal structure, chapter-level narrative, and cross-modal evidence—is integral to performance. For example, dropping abstraction-level summarization or global-local aggregation reduces CIDEr and overall retrieval accuracy [2006.13608]. SPA-Compressor provides a scalable solution to the quadratic token cost of MLLMs without sacrificing factuality or coherence.

Documented limitations include dependency on ASR quality (sensitivity to accents/noise), difficulty of handling ultra-rapid scene transitions, and time-consuming annotation procedures [2601.07366]. Future extensions are proposed: integrating on-screen OCR, differentiable chapter-boundary detectors, incorporating user feedback for iterative narrative refinement, and domain transfer to other dense video genres (e.g., sports, medical).

A plausible implication is that the E-HVC methodology can serve as a general foundation for structured video understanding in high-density environments, wherever fine-grained product or procedural semantics must be coupled with robust narrative abstraction. High-quality, temporally grounded annotation remains a key determinant of downstream performance.

## 7. Related Research Directions

E-HVC situates at the intersection of graph-based multimodal fusion, hierarchical sequence modeling, and multimodal large language model (MLLM) adaptation. Early work ("Comprehensive Information Integration Modeling Framework for Video Titling" [2006.13608]) established graph neural network approaches for integrating structured modalities. Later frameworks (e.g., HiVid-Narrator [2601.07366]) emphasize hierarchical transformer-based reasoning, cross-modal temporal alignment, and efficient token compression. Both lines motivate future research in scalable e-commerce video understanding, dataset curation with dual-level subjective/objective annotations, and systematic benchmarking for dense-multimodal sequence modeling.

Source: https://www.emergentmind.com/topics/e-commerce-hierarchical-video-captioning-e-hvc