Multi-Granular Video Understanding

• Multi-granular video understanding is a framework that processes content at multiple semantic, spatial, and temporal scales, capturing both fine details and global context.
• It employs hierarchical encoders, multi-granular feature integration, and graph-based architectures to fuse short-term and long-term dependencies for improved performance.
• These techniques drive advances in action recognition, segmentation, and explainable AI by providing robust, scalable, and interpretable analysis across diverse video datasets.

Multi-granular video understanding encompasses models, data representations, and computational frameworks that explicitly handle and reason about video content at multiple semantic, spatial, and temporal granularities. This paradigm is motivated by the inherent multi-scale structure of real-world videos: events span from sub-second object motions to minute-long activities; objects appear as coarse whole entities and as fine-grained parts; narrative understanding requires simultaneous awareness of global context and local detail. Multi-granular approaches have become foundational in modern video analysis, driving advances in action recognition, temporal anticipation, segmentation, video-language retrieval, explainable AI, and agent-driven video reasoning.

1. Foundations and Motivations

The central premise of multi-granular video understanding is that information at different scales—short vs. long-term temporal dynamics, coarse action groups vs. fine-grained categories, global event scenes vs. object-level detail—is complementary and essential for robust modeling. Early work established that behavior recognition in video fundamentally differs from static image tasks, requiring joint spatiotemporal aggregation across both short-range (adjacent-frame) and long-range (distant-frame) dependencies (Zhang et al., 2022). Hierarchical models demonstrated that learning with multi-level targets—coarse groups, fine categories, and free-form captions—systematically improves performance at all levels due to shared representations and semantic priors (Mahdisoltani et al., 2018).

Subsequent research extended these principles to temporal segmentation (Sener et al., 2020), egocentric activity understanding (Peirone et al., 4 Feb 2025), agentic video reasoning (Gao et al., 18 Nov 2025, Zhang et al., 23 May 2025, Li et al., 29 Sep 2025), multi-modal LLM alignment (Pan et al., 12 Dec 2025, Li et al., 9 Jan 2026, Shi et al., 14 Apr 2025), and dense video object segmentation (Lim et al., 2024). The outcome is a landscape where multi-granular analysis is now the default regime for state-of-the-art video understanding systems.

2. Model Designs and Architectural Strategies

Multi-granular video models incorporate explicit mechanisms for decomposing, aggregating, and aligning information across scales. The principal strategies include:

• Hierarchical Encoders: Architectures stack modules or graph layers to process video at progressively coarser temporal or semantic resolutions. For example, the hierarchical video understanding model encodes frames with 3D-CNN + LSTM, then cascades group, category, and caption heads, each dependent on prior-level predictions (Mahdisoltani et al., 2018). Similarly, Hier-EgoPack constructs a temporal hierarchy of graphs, where each higher stage pools and abstracts over temporally adjacent nodes from the previous finer stage, enabling both segment-level and clip-level reasoning (Peirone et al., 4 Feb 2025).
• Multi-Granular Feature Integration: The Integration of Multigranular Motion Features (IMG) framework employs parallel submodules for channel-attentive short-term (adjacent-frame) motion enhancement and cascaded long-term (distant-frame) motion aggregation, both inserted into a Res2Net backbone. These outputs are fused to build unified representations that are sensitive to both temporal extremes (Zhang et al., 2022).
• Graph and Hypergraph Formulations: Multi-Granular Hypergraphs (MGH) build per-scale spatial graphs by partitioning frames and interconnect part-based nodes with hyperedges spanning multiple temporal ranges. The resulting hypergraphs propagate information to align misaligned parts and recover under occlusion, with each scale contributing a pooled embedding. Mutual information penalties are used to decorrelate redundancy across scales (Yan et al., 2021). Hierarchical conditional graph models, as in QGA, interleave object-, frame-, and clip-level graph attention, each conditioned on text queries, yielding interpretable, multi-granular compositionality for video question answering (Xiao et al., 2021).
• Chunked and Rotational Encoding in Transformers: In Mavors, an intra-chunk vision encoder leverages 3D convolutions and ViTs to preserve spatial detail inside temporal chunks, while an inter-chunk aggregator applies rotary-encoded transformer attention to model long-range coherence without loss of spatial fidelity (Shi et al., 14 Apr 2025).

3. Data Representations and Multi-Granular Annotation

The move to multi-granular models has necessitated parallel advances in dataset construction and representation:

• Hierarchical Labels: Datasets like Something-Something v2 expose annotation hierarchies—action groups, categories, free-form captions—enabling hierarchical loss formulations and analysis of cross-level transfer (Mahdisoltani et al., 2018).
• Fragment- and Object-level Influence: For explainability in video summarization, fragment-level (shot) and object-level (mask) perturbation-based explanations reveal which temporal and spatial elements most strongly drive the summarizer’s decisions (Tsigos et al., 2024).
• Multi-granularity Video Object Segmentation (VOS): MUG-VOS densely annotates videos with masks at different object and part granularities—including both salient foreground, non-salient objects, and object parts—supporting fine-grained segmentation and robust memory-based mask propagation (Lim et al., 2024).
• Expanding Data Granularity via Synthesis: The GEXIA framework introduces “granularity expansion” by systematically synthesizing long-video/long-text and long-video/short-text pairs from single-grained corpora, and proposes a model that iteratively approximates variable-length, multi-granularity inputs to fixed semantic vectors for scalable contrastive alignment (Wang et al., 2024).

4. Algorithms for Multi-Scale Integration and Aggregation

Computational strategies for cross-granular integration are diverse:

• Temporal Aggregation and Pooling: Temporal Aggregation Blocks (TABs) combine max-pooling and attention over snippets at distinct temporal scales, coupled via non-local blocks, achieving state-of-the-art anticipation by fusing short-term and spanning context (Sener et al., 2020).
• Multi-granular Spatio-Temporal Token Pruning: To accelerate video LLMs, multi-granular spatio-temporal token merging (STTM) generates spatial tokens via a coarse-to-fine quadtree and merges temporally redundant tokens across frames, reducing inference time without retraining or significant accuracy loss (Hyun et al., 10 Jul 2025).
• Contrastive and Multi-Task Losses: Multi-granular encoders are often trained with multi-level cross-entropy or contrastive objectives, sometimes with joint regularization (e.g., information-theoretic decorrelation or dynamic weighting among granularities) (Yan et al., 2021, Li et al., 9 Jan 2026, Wang et al., 2024).
• Agent-Based Search and Iterative Reasoning: Agentic frameworks orchestrate a small set of search-centric tools (global browse, clip-level retrieval, frame-level inspection) and use LLM agents to iteratively refine multi-granular search and inspection in long video reasoning (Zhang et al., 23 May 2025, Gao et al., 18 Nov 2025, Li et al., 29 Sep 2025). These approaches prioritize completeness and efficiency by traversing the video from global summaries down to precise frame or object detail as required by the task.

5. Applications and Empirical Outcomes

Multi-granular video understanding underpins a broad spectrum of SOTA tasks:

• Action Recognition & Anticipation: Models integrating multi-level motion or temporal context achieve significant accuracy improvements over single-scale baselines on benchmarks such as Something-Something, HMDB51, UCF101, Breakfast, and EPIC-Kitchens (Zhang et al., 2022, Sener et al., 2020).
• QA, Summarization, and Retrieval: Multi-granularity retrieval and memory architectures exhibit notable accuracy and computational gains in hour-long video QA, summarization (ROUGE-2/METEOR), and frame-level precision (Ego4D, HourVideo, MovieChat-1K) when compared to monolithic or fine-only representations (Li et al., 9 Jan 2026).
• Video-Language Pretraining: GEXIA’s enlarged corpus and iterative approximation achieve strong retrieval, classification, and transfer performance on ActivityNet, LVU, COIN, and Charades-Ego, without explicit multi-granular benchmarks (Wang et al., 2024).
• Video-Language Large Models (Video-LLMs): Video LLMs such as UFVideo explicitly link global, pixel, and temporal grounding via a unified token interface and modular mask decoder, yielding consistent SOTA across global, pixel, and temporal QA tasks (MVBench, VideoRefer, ReVOS, Charades-STA, UFVideo-Bench) (Pan et al., 12 Dec 2025).
• Agentic and Explainable Systems: Agentic frameworks leveraging multi-granular databases and tools (AVI, DVD) demonstrate competitive or superior performance to RL-trained or proprietary LLM systems, while offering transparent, interpretable reasoning trajectories across all granularities (Gao et al., 18 Nov 2025, Zhang et al., 23 May 2025, Li et al., 29 Sep 2025, Tsigos et al., 2024).

6. Limitations, Variants, and Open Challenges

Identified limitations and areas for future research include:

• Dataset Bottlenecks: Real-world, richly annotated multi-granular video-language datasets remain scarce. Most benchmarks remain single-granular. Synthesis-based expansion (GEX) may address coverage, but manual annotation for dense segmentation or QA remains labor-intensive (Wang et al., 2024, Lim et al., 2024).
• Computational Efficiency: Hierarchical, multi-granular architectures are inherently more complex, often requiring strategies like memory-efficient branching, token merging, or staged computation to be practical for long videos (Hyun et al., 10 Jul 2025, Shi et al., 14 Apr 2025).
• Dynamic and Adaptive Granularity: Current systems often operate at fixed, pre-defined scales. Adaptive, data-driven, or task-driven granularity selection is a topic of active exploration (Hyun et al., 10 Jul 2025, Wang et al., 2024, Li et al., 29 Sep 2025).
• Cross-Modal and Open-Vocabulary Fusion: Extending multi-granularity to integrate audio, transcripts, and other modalities, as well as supporting open-text, object, and event schemas, is not yet fully realized. Promising directions include unified multi-modal LLMs and language-guided mask propagation (Pan et al., 12 Dec 2025, Lim et al., 2024).
• Interpretability and Explainability: While fragment and object-level explanations are now feasible (Tsigos et al., 2024), causal understanding and feedback to model design or system users (e.g., media editors) is not yet fully integrated into video LLM pipelines.

7. Outlook and Future Directions

Progress in multi-granular video understanding has enabled significant advances in fine- and coarse-grained reasoning, efficient long-video analysis, and generalization across downstream tasks. Emerging lines of inquiry include developing adaptive multi-granular modeling policies, learning task- and input-dependent granularity schedules, designing unified video-language pretraining objectives for arbitrary time scales, and integrating active agentic planning with fully differentiable multi-granular representations. The confluence of multi-granular modeling, LLMs, and agentic search is poised to drive further breakthroughs in comprehensive, scalable, and explainable video understanding systems.

References

• Behavior Recognition Based on the Integration of Multigranular Motion Features (Zhang et al., 2022)
• Hierarchical Video Understanding (Mahdisoltani et al., 2018)
• Learning Multi-Granular Hypergraphs for Video-Based Person Re-Identification (Yan et al., 2021)
• Video as Conditional Graph Hierarchy for Multi-Granular Question Answering (Xiao et al., 2021)
• Multi-Granularity Video Object Segmentation (Lim et al., 2024)
• Temporal Aggregate Representations for Long-Range Video Understanding (Sener et al., 2020)
• MMViR: A Multi-Modal and Multi-Granularity Representation for Long-range Video Understanding (Li et al., 9 Jan 2026)
• UFVideo: Towards Unified Fine-Grained Video Cooperative Understanding with LLMs (Pan et al., 12 Dec 2025)
• GEXIA: Granularity Expansion and Iterative Approximation for Scalable Multi-grained Video-language Learning (Wang et al., 2024)
• Mavors: Multi-granularity Video Representation for Multimodal LLM (Shi et al., 14 Apr 2025)
• Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMs (Hyun et al., 10 Jul 2025)
• Hier-EgoPack: Hierarchical Egocentric Video Understanding with Diverse Task Perspectives (Peirone et al., 4 Feb 2025)
• Agentic Video Intelligence: A Flexible Framework for Advanced Video Exploration and Understanding (Gao et al., 18 Nov 2025)
• Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understanding (Zhang et al., 23 May 2025)
• Perceive, Reflect and Understand Long Video: Progressive Multi-Granular Clue Exploration with Interactive Agents (Li et al., 29 Sep 2025)
• An Integrated Framework for Multi-Granular Explanation of Video Summarization (Tsigos et al., 2024)