CrossVideoQA: Multi-Stream Video QA
- CrossVideoQA is a research area that synthesizes evidence across distinct video sources to answer queries requiring inter-stream, spatiotemporal reasoning.
- It leverages person-anchored hierarchical reasoning and object-centric models to track entities and actions across different locations and times.
- Benchmark studies reveal significant performance gaps in multimodal large language models, highlighting challenges in context retention and cross-video disambiguation.
Searching arXiv for papers on CrossVideoQA and closely related multi-video/video QA work. Searching arXiv for "CrossVideoQA". CrossVideoQA denotes question answering that requires integrating evidence across multiple, distinct video streams rather than reasoning within a single clip. In its most explicit formulation, the problem centers on establishing meaningful connections across video streams and managing the complexity of multi-source information retrieval, often through queries that span locations, time periods, or both (Meng et al., 5 Aug 2025). Closely related benchmark work frames the task as cross-video relational reasoning over cross-video object association, cross-video event association, and cross-video complex reasoning, and reports that current multimodal LLMs remain far from human performance on these settings (Zhu et al., 27 Aug 2025). As a research topic, CrossVideoQA inherits the central concerns of VideoQA—spatio-temporal reasoning, multimodal fusion, and answer grounding—but elevates inter-video context retention, entity disambiguation, and cross-stream causal linkage to primary requirements.
1. Definition and task structure
CrossVideoQA differs from conventional VideoQA by requiring synthesis across multiple video sources. The defining challenge is not merely longer visual context, but the need to connect fragmented evidence across streams that may be separated by camera, location, or time. VideoForest characterizes this setting as one in which person identity, actions, and contextual cues must be reliably tracked and integrated across space and time, and develops a benchmark with four evaluation modalities: Single, Cross-spatial, Cross-temporal, and Cross-spatiotemporal (Meng et al., 5 Aug 2025).
A broader formulation appears in CVBench, which organizes cross-video reasoning into three hierarchical tiers. The first tier, cross-video object association, covers tasks such as cross-video object recognition, multi-video attribute recognition, joint-video counting, and cross-video entity matching. The second tier, cross-video event association, includes cross-video anomaly detection, scene recognition, multi-video key action recognition, event retrieval, multi-view scene understanding, and temporal reasoning. The third tier, cross-video complex reasoning, includes joint-video spatial navigation, video difference captioning, cross-video counterfactual reasoning, joint-video summarization, and procedural transfer (Zhu et al., 27 Aug 2025).
A useful distinction is between CrossVideoQA as a task family and CrossVideoQA as a dataset name. VideoForest introduces CrossVideoQA as a benchmark dataset specifically designed for person-centric cross-video analysis, with three reasoning categories: Person Recognition, Behavior Analysis, and Summarization & Reasoning (Meng et al., 5 Aug 2025). This suggests that the field currently combines a general research direction with benchmark-specific operationalizations.
2. Antecedents in VideoQA
CrossVideoQA emerged from a broader VideoQA literature that progressively expanded what counts as relevant evidence. Early hierarchical and relational models such as Hierarchical Conditional Relation Networks introduced the Conditional Relation Network as a reusable neural unit for jointly modeling content selection and relation construction, then stacked these units hierarchically to capture near- and far-term relations in video-length structures (Le et al., 2020, Le et al., 2020). These models established a template in which question-conditioned relational reasoning is performed at multiple temporal scales.
Object-centric approaches pushed this idea toward explicit entity structure. “Object-Centric Representation Learning for Video Question Answering” turns a video into an evolving relational graph of objects, using Faster R-CNN, DeepSort, query-guided temporal attention, GCN-based interaction modeling, and object résumés that summarize each object’s lifetime and interactions (Dang et al., 2021). This object-level abstraction is directly relevant to CrossVideoQA because cross-video reasoning frequently depends on persistent entities rather than holistic clip descriptors.
Graph-transformer and event-centric approaches further enriched the representation space. CoVGT reframed VideoQA as contrastive matching rather than simple classification, combining a dynamic graph transformer with supervised and self-supervised contrastive objectives (Xiao et al., 2023). EC-GNNs introduced dense caption modality as an auxiliary source and performed cross-modal reasoning over caption, video, and question via Event-Correlated Graph Neural Networks and question-guided self-adaptive multi-modal fusion (Yin et al., 2023). Long-term VideoQA work then moved toward end-to-end generative formulations: MCG combines Joint Unimodal Modeling, Multi-granularity Contrastive Learning, and Cross-modal Collaborative Generation to reformulate VideoQA as answer generation rather than classification (Yu et al., 2024).
Another important antecedent is the extension of VideoQA beyond purely visual appearance. “Watching the News: Towards VideoQA Models that can Read” introduced scene-text-aware VideoQA and the NewsVideoQA dataset, arguing that textual information embedded within video frames is complementary to action and provides essential contextualisation cues (Jahagirdar et al., 2022). The dataset comprises 3,083 clips and 8,672 human-annotated question-answer pairs, and only 17% of questions can be answered using subtitles, underscoring that transcripts alone do not exhaust the evidence space (Jahagirdar et al., 2022). For CrossVideoQA, this is consequential because cross-stream reasoning may require aligning entities, events, and textual overlays simultaneously.
3. Benchmark landscape
The current empirical landscape combines direct cross-video benchmarks with adjacent resources that stress robustness, generalization, and complex reasoning. The most directly relevant benchmarks are CrossVideoQA and CVBench; CVRR-ES and EgoCross are not cross-video benchmarks, but they expose reasoning and generalization failures that plausibly transfer into cross-video settings.
| Resource | Focus | Reported statistics |
|---|---|---|
| CrossVideoQA (Meng et al., 5 Aug 2025) | Person-centric cross-video analysis | EOSD: 18 videos, 3 locations, 12 dates, 450,000 frames; HACS: 50,000 videos, 1.55 million clips; 3 reasoning categories; 4 evaluation modalities |
| CVBench (Zhu et al., 27 Aug 2025) | Cross-video relational reasoning | 1,315 videos, 1,000 QA pairs, 2 ~ 4 videos per QA, average duration 106.6 s; 3 tiers; 15 fine-grained task categories |
| CVRR-ES (Xie et al., 18 Jul 2025) | Open-ended, context-heavy video QA | 11 challenging categories including multi-action, fine-grained actions, partial actions, time order, actions not present, continuity/object counting, physical anomalies, and social/emotional/visual context |
| EgoCross (Li et al., 14 Aug 2025) | Cross-domain EgocentricQA | 798 video clips, 957 QA pairs, 4 domains, 4 key QA tasks, OpenQA and CloseQA formats |
CVBench is the first comprehensive benchmark designed to assess cross-video relational reasoning rigorously (Zhu et al., 27 Aug 2025). It evaluates 10+ leading MLLMs under zero-shot or chain-of-thought prompting paradigms and reports stark performance gaps. Human performance reaches 91.3% average accuracy across all categories, whereas the best model reaches 69.4% on complex cross-video reasoning; for causal reasoning, even top models such as GPT-4o achieve only 60% accuracy, compared to 91.0% human accuracy (Zhu et al., 27 Aug 2025).
VideoForest’s CrossVideoQA benchmark is narrower but more structured around person-centric reasoning. It is built from the Edinburgh Office Surveillance Dataset and HACS, and its three reasoning categories are evaluated under Single, Cross-spatial, Cross-temporal, and Cross-spatiotemporal settings (Meng et al., 5 Aug 2025). This person-anchored design makes identity linkage a first-class problem rather than an incidental subroutine.
CVRR-ES, used in “Team of One,” is a demanding open-ended benchmark for complex real-world scenarios. Although it is not cross-video, its 11 categories emphasize reasoning depth and robustness, and the training-free model-synergy framework achieves 88.04% average accuracy, compared with 70.78% for GPT-4V and 96.67% for human annotators (Xie et al., 18 Jul 2025). EgoCross similarly shows that even strong MLLMs struggle to generalize beyond familiar daily-life egocentric domains, with top CloseQA performance below 55% and top OpenQA performance below 35% (Li et al., 14 Aug 2025). A plausible implication is that CrossVideoQA will remain sensitive not only to reasoning complexity but also to domain shift.
4. Core modeling strategies
Existing CrossVideoQA systems and closely related VideoQA models fall into several methodological families: hierarchical relational encoders, object- or person-anchored representations, contrastive cross-modal matching, grounding-oriented causal models, and modular multi-expert reasoning.
VideoForest is the most explicit CrossVideoQA architecture in the cited literature. Its central idea is person-anchored hierarchical reasoning: persons serve as natural anchor points that connect semantic trajectories across multiple videos (Meng et al., 5 Aug 2025). The framework consists of three elements. First, person-anchored feature extraction uses person detection, tracking, and ReID to establish spatiotemporal relationships across sources. Second, a multi-granularity spanning tree hierarchically organizes visual content around person-level trajectories. Third, a multi-agent reasoning framework, implemented atop CrewAI, decomposes query resolution into Filter Agent, Retrieval Agent, Navigate Agent, and Integrate Agent. Its top-down search is defined through a relevance function
which enables coarse-to-fine traversal of the hierarchy (Meng et al., 5 Aug 2025).
Hierarchical reasoning before CrossVideoQA was developed in HCRN. There, CRN blocks transform arrays of objects conditioned on contextual features, and the hierarchy composes frame-level and clip-level relations for scalable VideoQA (Le et al., 2020, Le et al., 2020). This logic reappears in CrossVideoQA as multi-granularity organization, except that the “objects” become cross-stream entities or segments rather than only within-video units.
Object-centric modeling provides a second line of continuity. OCRL represents videos as query-guided evolving relational graphs of objects and summarizes object lives into résumés (Dang et al., 2021). CrossVideoQA generalizes this principle by using person-level trajectories across videos as bridge points. The shift is from object persistence within a video to identity persistence across videos.
Contrastive methods offer a third strategy. CoVGT learns separate video and text transformers, a dynamic graph transformer for object relations and dynamics, and joint supervised plus self-supervised contrastive objectives (Xiao et al., 2023). MCG extends contrastive alignment to long-term end-to-end VideoQA through instance-level and token-level objectives:
before generating free-form answers with Cross-modal Collaborative Generation (Yu et al., 2024). This suggests that cross-video systems may benefit from explicit alignment at both global and local granularities, especially when evidence is distributed across clips or cameras.
A fourth strategy is modular model synergy. “Team of One” coordinates multiple heterogeneous VLMs through structured chain-of-thought prompting and uses an external Multimodal LLM as evaluator and integrator:
(Xie et al., 18 Jul 2025). Although demonstrated on CVRR-ES rather than cross-video data, the framework is lightweight, extensible, and training-free. This suggests a plausible route for CrossVideoQA when end-to-end retraining over multiple video streams is impractical.
5. Grounding, faithfulness, and multimodal evidence
A persistent difficulty in CrossVideoQA is that a correct answer is not sufficient if the supporting evidence is poorly localized or spuriously correlated. This issue is addressed directly in Video Question Grounding research. CRA targets spurious cross-modal correlations and unfaithful generalization by coupling Gaussian Smoothing Grounding, Cross-Modal Alignment, and Explicit Causal Intervention (Chen et al., 5 Mar 2025). Its causal formulation combines front-door intervention for vision and back-door intervention for language, with the aim of improving causal consistency between question answering and video temporal grounding. The framework reports superior performance on NextGQA and STAR in discovering visually grounded content and achieving robust question reasoning (Chen et al., 5 Mar 2025).
For CrossVideoQA, grounding is more complex because relevant evidence may reside in different videos, and the dominant scene in one source may be irrelevant without corroboration from another. CVBench explicitly identifies deficient inter-video context retention and poor disambiguation of overlapping entities as fundamental bottlenecks in current MLLM architectures (Zhu et al., 27 Aug 2025). VideoForest addresses this by storing person ReID tuples in tree nodes as explicit bridge points for cross-video traversal (Meng et al., 5 Aug 2025). The contrast between these two works is instructive: CVBench diagnoses model-level failure modes, while VideoForest builds an explicit retrieval and reasoning scaffold around identity continuity.
Multimodal evidence extends beyond RGB frames. NewsVideoQA shows that scene text can be essential contextualisation that transcripts do not capture; OCR-aware SINGULARITY modifies pretraining from pairs to pairs and adds Vision-OCR Contrastive Loss, Masked Language Modeling over OCR, and Vision-OCR Matching (Jahagirdar et al., 2022). EC-GNNs add dense captions as a new auxiliary modality for event correlation (Yin et al., 2023). A plausible implication is that future CrossVideoQA systems will need multimodal retrieval over identities, events, and embedded text rather than only over raw video embeddings.
6. Empirical findings, misconceptions, and open directions
Two common misconceptions are not supported by the current evidence. The first is that CrossVideoQA is simply single-video QA with more frames. CVBench reports poor inter-video context retention, entity disambiguation failures, causal and temporal reasoning limitations, and sensitivity to input order (Zhu et al., 27 Aug 2025). VideoForest likewise shows that models that perform competitively on single-video analysis drop sharply when multi-video reasoning is required, whereas VideoForest maintains 72.00% on Cross-Temporal, 69.23% on Cross-Spatial, 65.38% on Cross-Spatiotemporal, and 61.54% on Single-video settings (Meng et al., 5 Aug 2025).
The second misconception is that stronger generic Video-LMMs automatically solve cross-video reasoning. On CrossVideoQA, VideoForest reports 71.93% in Person Recognition, 83.75% in Behavior Analysis, and 51.67% in Summarization & Reasoning, outperforming baselines such as mPLUG-Owl3, InternVL-2.5, and BIMBA-LLaVA (Meng et al., 5 Aug 2025). Its ablations further report that removing person ReID from search causes a 20% accuracy drop in cross-temporal QA, removing video filtering causes up to 26.9% loss in complex multi-hop reasoning, and removing deep traversal causes an approximately 6% average drop (Meng et al., 5 Aug 2025). These results indicate that explicit cross-video structure and retrieval policy remain decisive.
The broader evaluation literature points to several open directions. CVBench recommends enhanced spatiotemporal context modeling, robust entity grounding and disambiguation, temporal and causal chain modeling, input encoding optimizations for video identity and order, and commonsense or domain knowledge synthesis (Zhu et al., 27 Aug 2025). EgoCross shows that reinforcement learning can produce the largest overall improvement in cross-domain EgocentricQA, raising Qwen2.5-VL-7B from 37.80% baseline CloseQA accuracy to 60.12% with RL in pilot studies (Li et al., 14 Aug 2025). “Team of One” suggests that training-free coordination of heterogeneous models can yield large gains on complex reasoning benchmarks (Xie et al., 18 Jul 2025). CRA suggests that causal deconfounding and grounding consistency are necessary for faithful reasoning (Chen et al., 5 Mar 2025). Taken together, these findings imply that next-generation CrossVideoQA is likely to combine explicit entity anchors, hierarchical memory, multimodal evidence integration, and modular or causal reasoning mechanisms rather than relying on single-stream scaling alone.