VideoForest: Hierarchical Cross-Video QA

• VideoForest is a cross-video QA framework based on person-anchored hierarchical reasoning that connects multi-camera video streams.
• It employs dual-stream encoding and adaptive segmentation to accurately extract person-level features and semantic video segments.
• The modular system uses a multi-agent, tree-structured approach to achieve superior performance on the CrossVideoQA benchmark.

VideoForest, introduced in "VideoForest: Person-Anchored Hierarchical Reasoning for Cross-Video Question Answering" (Meng et al., 5 Aug 2025), is a framework for cross-video question answering built around person-anchored hierarchical reasoning. It addresses the problem of establishing meaningful connections across video streams and managing the complexity of multi-source information retrieval by using person-level features as natural bridge points between videos, organizing content through a multi-granularity spanning tree structure, and answering queries through a multi-agent reasoning framework. The system is modular, does not require end-to-end training, and is evaluated on CrossVideoQA, where it reports 71.93% accuracy in person recognition, 83.75% in behavior analysis, and 51.67% in summarization and reasoning (Meng et al., 5 Aug 2025).

1. Problem setting and conceptual basis

Cross-video question answering is defined over multiple video sources rather than a single temporally contiguous stream. In VideoForest, the task is formalized as

$$\mathcal{Q}: \mathcal{V} \times \mathcal{T} \times \mathcal{L} \rightarrow \mathcal{A},$$

where the formulation explicitly couples video content with temporal and location-related structure. The central premise is that cross-video reasoning requires more than sequential aggregation of independently processed videos; it requires explicit mechanisms for linking events, identities, and temporal structure across streams (Meng et al., 5 Aug 2025).

The framework adopts a person-centric view of this problem. Person-level trajectories are treated as the primary anchors for connecting disjoint camera sources, times, and locations. This is not merely a retrieval heuristic but the organizing principle of the entire system: feature extraction, segmentation, indexing, traversal, and reasoning are all constructed around persistent person identity and motion cues. The reported results indicate that this organization is particularly effective for person recognition, behavior analysis, and summarization and reasoning in multi-video settings.

A common simplification is to regard cross-video QA as a straightforward extension of single-video QA. The experimental framing of VideoForest argues against that view: the benchmark explicitly separates single-video, cross-spatial, cross-temporal, and cross-spatiotemporal conditions, and the comparison to prior methods shows that general-purpose VideoQA models degrade sharply in multi-video, person-tracking tasks (Meng et al., 5 Aug 2025).

2. Person-anchored feature extraction and adaptive segmentation

VideoForest uses dual-stream encoding. The visual stream applies the ViCLIP encoder for frame-level semantic features:

$$\mathbf{v}(f_{i,j}) = \phi(f_{i,j}; \theta_v) \in \mathbb{R}^d.$$

In parallel, the person stream uses tracking and Re-Identification models to detect, track, and assign unique IDs to persons in each frame:

$$\mathbf{p}(f_{i,j}) = \psi(f_{i,j}; \theta_p) = \{(t_k, \mathbf{x}_k, \mathrm{id}_k)\}_{k=1}^{K_{i,j}}.$$

The summary associates this stage with ReID and tracking algorithms, including YOLO-SlowFast and STD/STA techniques (Meng et al., 5 Aug 2025).

Segmentation is adaptive rather than fixed-window. A frame boundary is introduced through a hybrid criterion:

$$S(f_{i,j}) = \mathbb{1}[C_1(f_{i,j}) \vee C_2(f_{i,j}) \vee C_3(f_{i,j})],$$

where $C_1$ denotes local feature transitions, $C_2$ denotes global content deviation, and $C_3$ denotes person-set changes, specifically identities entering or leaving. This mechanism ties semantic change to both appearance and population dynamics, which is important in surveillance-style and distributed multi-camera settings.

Each segment is then encoded by combining a keyframe appearance representation with aggregated person trajectory features:

$$\mathbf{C}(S_{i,k}) = \eta(\mathbf{v}(f_{i,j}^{\text{key}}), \mathbf{P}(S_{i,k}); \theta_{\eta}).$$

The significance of this construction is that segment representations are not purely scene-level embeddings. They are explicitly conditioned on person trajectories, which makes subsequent cross-video linking possible through identity continuity rather than through global similarity alone (Meng et al., 5 Aug 2025).

3. Multi-granularity spanning tree representation

The core structural abstraction in VideoForest is the multi-granularity spanning tree structure. Each video is represented as a tree $\mathcal{T}_i$ whose root corresponds to the entire video duration and whose leaves correspond to fine-grained semantic segments. A node is represented as

$$v = (t_{\text{start}}, t_{\text{end}}, \mathcal{R}_v, \mathbf{C}_v, \Gamma_v),$$

where $$\mathbf{v}(f_{i,j}) = \phi(f_{i,j}; \theta_v) \in \mathbb{R}^d.$$0 stores person ID and trajectory information, $$\mathbf{v}(f_{i,j}) = \phi(f_{i,j}; \theta_v) \in \mathbb{R}^d.$$1 stores the segment or aggregate representation, and $$\mathbf{v}(f_{i,j}) = \phi(f_{i,j}; \theta_v) \in \mathbb{R}^d.$$2 denotes child nodes (Meng et al., 5 Aug 2025).

The hierarchy satisfies a disjoint cover principle:

$$\mathbf{v}(f_{i,j}) = \phi(f_{i,j}; \theta_v) \in \mathbb{R}^d.$$3

This means that the children of a node cover the parent interval without overlap. The effect is to provide a recursively navigable temporal partition in which the system can move from coarse summaries to fine segments while preserving temporal completeness.

Person-centric indexing is the mechanism that turns a collection of trees into a cross-video structure. ReID metadata stored at each node makes it possible to link nodes across trees by identity continuity. In practice, the trees remain video-specific, but person identity provides the cross-tree edge semantics. This yields a hybrid organization: temporally nested trees within videos and person-indexed bridges across videos. The paper characterizes this as supporting top-down, granular navigation and enabling efficient retrieval and aggregation under temporal, spatial, or person constraints (Meng et al., 5 Aug 2025).

This tree design differs from flat memory schemes. Rather than storing all retrieved evidence in a uniform buffer, VideoForest uses hierarchical abstraction levels. That organization is central to the claim that it can answer complex cross-video queries while maintaining computational efficiency.

4. Multi-agent reasoning and non-end-to-end orchestration

Reasoning in VideoForest is implemented as a multi-agent system using four agent types, implemented with CrewAI: $$\mathbf{v}(f_{i,j}) = \phi(f_{i,j}; \theta_v) \in \mathbb{R}^d.$$4, $$\mathbf{v}(f_{i,j}) = \phi(f_{i,j}; \theta_v) \in \mathbb{R}^d.$$5, $$\mathbf{v}(f_{i,j}) = \phi(f_{i,j}; \theta_v) \in \mathbb{R}^d.$$6, and $$\mathbf{v}(f_{i,j}) = \phi(f_{i,j}; \theta_v) \in \mathbb{R}^d.$$7 (Meng et al., 5 Aug 2025). The filter agent extracts constraints from the query and selects relevant video trees. The retrieval agent acts as a knowledge base manager with confidence-weighted storage. The navigate agent traverses the tree hierarchies. The integrate agent synthesizes information across trees and forms the final answer.

The navigation stage is governed by an adaptive relevance-based search:

$$\mathbf{v}(f_{i,j}) = \phi(f_{i,j}; \theta_v) \in \mathbb{R}^d.$$8

This search rule implements hierarchical descent only when a coarse node is not sufficiently relevant. As a result, the system can stop at coarse granularity when the query is broad and descend to leaves when the query requires detailed evidence.

The knowledge base stores entries of the form $$\mathbf{v}(f_{i,j}) = \phi(f_{i,j}; \theta_v) \in \mathbb{R}^d.$$9, where $$\mathbf{p}(f_{i,j}) = \psi(f_{i,j}; \theta_p) = \{(t_k, \mathbf{x}_k, \mathrm{id}_k)\}_{k=1}^{K_{i,j}}.$$0 is a dynamic confidence that self-corrects via reinforcement and revision on future queries. This introduces persistence across interactions without requiring joint retraining of the perception and reasoning stack.

A key property of the framework is that it does not require end-to-end training. Each module is standalone; feature extractors and LLMs are pre-trained and frozen; orchestration rather than gradient-based joint optimization produces the final reasoning behavior (Meng et al., 5 Aug 2025). This point is frequently misunderstood. The absence of end-to-end training does not imply the absence of learned components. It means that the system composes pre-trained components through structured reasoning, retrieval, and control rather than through expensive multi-video joint training.

5. CrossVideoQA benchmark and empirical performance

VideoForest is evaluated on CrossVideoQA, described as the first comprehensive benchmark for person-centric cross-video QA. The benchmark combines Edinburgh Office Surveillance and HACS. The Edinburgh Office Surveillance component contains approximately 450,000 frames, 18 cameras, and 12 days, while HACS contains 50,000 videos and 1.55M action clips (Meng et al., 5 Aug 2025). The task set is divided into person recognition, behavior analysis, and summarization and reasoning, and the evaluation conditions include $$\mathbf{p}(f_{i,j}) = \psi(f_{i,j}; \theta_p) = \{(t_k, \mathbf{x}_k, \mathrm{id}_k)\}_{k=1}^{K_{i,j}}.$$1, $$\mathbf{p}(f_{i,j}) = \psi(f_{i,j}; \theta_p) = \{(t_k, \mathbf{x}_k, \mathrm{id}_k)\}_{k=1}^{K_{i,j}}.$$2, $$\mathbf{p}(f_{i,j}) = \psi(f_{i,j}; \theta_p) = \{(t_k, \mathbf{x}_k, \mathrm{id}_k)\}_{k=1}^{K_{i,j}}.$$3, and $$\mathbf{p}(f_{i,j}) = \psi(f_{i,j}; \theta_p) = \{(t_k, \mathbf{x}_k, \mathrm{id}_k)\}_{k=1}^{K_{i,j}}.$$4.

The headline task results are as follows.

Task	VideoForest	Best Baseline
Person Recognition	71.93%	58.93%
Behavior Analysis	83.75%	66.67%
Summarization and Reasoning	51.67%	46.67%
Overall	69.12%	58.38%

The best baseline reported in the summary is InternVL2.5. The comparison indicates that VideoForest improves on all three task groups and on the overall score (Meng et al., 5 Aug 2025).

The ablation study clarifies which components drive these gains. Removing ReID in search causes up to a 20% performance drop in cross-temporal settings. Removing the video filter causes up to a 26.92% drop in cross-spatiotemporal settings. Removing deep tree traversal causes a 6–7% average drop. These results are consistent with the intended design logic: identity continuity, selective candidate filtering, and hierarchical traversal are all essential, not auxiliary.

Performance is also reported across spatiotemporal modalities. VideoForest is stated to be substantially better across all modalities, including 72.00% in cross-temporal and 69.23% in cross-spatial settings. This is important because the most difficult cross-video conditions are precisely those in which temporal disjunction, camera disjunction, or both disrupt naive sequential reasoning.

6. Relation to prior approaches, adjacent work, and scope

The paper’s comparison to prior approaches frames VideoForest as a departure from monolithic or weakly linked multi-video processing. Prior systems such as InternVL and VideoCCAM are characterized as having limited or absent cross-video reasoning, weak or absent person-centric tracking, flat memory or single-video hierarchies, and training paradigms that are end-to-end, large-scale, data-hungry, and single-stream-based (Meng et al., 5 Aug 2025). VideoForest instead uses explicit person-anchored bridges, multi-granularity trees, collaborative agents, and modular orchestration over pre-trained models.

Within the broader video literature, tree-structured or forest-structured representations have appeared for different purposes. StreamForest organizes streaming video into event-level tree structures through a Persistent Event Memory Forest and a Fine-grained Spatiotemporal Window for online video understanding (Zeng et al., 29 Sep 2025). ForestPrune constructs token forests across frames under semantic, spatial, and temporal constraints for training-free visual token compression in video MLLMs (Ju et al., 24 Mar 2026). These systems pursue memory retention and token pruning rather than cross-video QA, but they show that forest-like structures have become a recurring design pattern for long-horizon video reasoning, compression, and retrieval. A plausible implication is that VideoForest belongs to a broader methodological shift toward explicit hierarchical structure in video models.

The scope of VideoForest is explicitly person-centric. Its benchmark, indexing scheme, and strongest empirical gains are tied to scenarios in which people serve as the stable anchors across cameras and times. This does not diminish the framework’s contribution, but it does delimit the evidence currently provided. The reported ablations show that when ReID-based identity linkage is weakened, performance drops sharply, especially in cross-temporal settings. This suggests that the framework’s current strengths are most directly established for distributed visual information in which person identity is the primary cross-video connective tissue.

As a result, VideoForest is best understood not simply as another VideoQA model, but as a structured cross-video reasoning system: human-anchored in representation, hierarchical in storage and traversal, multi-agent in inference, and modular in training strategy. In the benchmarked person-centric regime, the evidence indicates that this combination yields a new baseline for cross-video understanding (Meng et al., 5 Aug 2025).