Multi-Video Collaborative Framework

• Multi-video collaborative framework is a paradigm that treats multiple video units as interdependent components, ensuring global coherence across views.
• It employs techniques like cross-shot synthesis, shared memory, and graph-based fusion to preserve temporal, geometric, and semantic consistency.
• Applications span joint video generation, dynamic multi-camera control, and distributed video caching, enhancing both performance and efficiency.

A multi-video collaborative framework is a class of video systems in which multiple videos, shots, camera views, temporal segments, modalities, or edge nodes are treated as interdependent units rather than as isolated sequences. In current research, collaboration is instantiated in several distinct ways: step-by-step cross-shot generation from a one-sentence story, joint denoising of multiple camera-conditioned videos, latent communication among locally budgeted agents for long-video understanding, graph-level fusion of related videos for video LLMs, topic-conditioned summarization, and cooperative caching, transcoding, and fast-forwarding in networked multi-camera systems (Zheng et al., 2024, Kuang et al., 2024, Chen et al., 1 May 2026, He et al., 16 Sep 2025, Panda et al., 2017, Tran et al., 2016, Lan et al., 2023). The common premise is that temporal coherence, geometric consistency, semantic completeness, and system efficiency often depend on information that is distributed across video units and therefore cannot be recovered reliably by single-stream processing alone.

1. Conceptual scope and meanings of collaboration

The term collaborative has no single fixed meaning in the literature. In "VideoGen-of-Thought" (Zheng et al., 2024), collaboration denotes a staged pipeline in which script generation, keyframe synthesis, shot-level video generation, and smoothing are explicitly linked, and in which outputs of earlier shots influence later shots through textual continuity and identity or latent propagation. In "Collaborative Video Diffusion" (Kuang et al., 2024), collaboration refers to jointly generating multiple videos of the same scene from different camera trajectories while enforcing cross-view synchronization during denoising. In "Collaborative Multi-bitrate Video Caching and Processing in Mobile-Edge Computing Networks" (Tran et al., 2016), collaboration means that MEC servers are not independent silos: they share cached content and distribute transcoding responsibility across neighboring base stations.

This heterogeneity suggests that a multi-video collaborative framework is better understood as an architectural principle than as a single algorithmic template. The collaborative unit may be a shot, a camera view, a temporal segment, a modality stream, a topic-related companion video, or an edge node. What remains stable across these settings is the explicit modeling of dependency. A video unit is not processed solely for its local objective; it is processed in anticipation that its representation, latent state, or service decision will be consumed by another unit or by a coordinator.

A plausible implication is that collaboration becomes necessary whenever the target property is global rather than local. Movie-like narrative continuity, cross-view geometric agreement, long-range semantic reasoning, topic-level summarization, and bandwidth-aware delivery all require decisions that are conditioned on more than one clip or stream. Much of the recent literature can therefore be read as an attempt to formalize where these dependencies arise and how they should be represented.

2. Recurrent architectural patterns

One recurrent pattern is decomposition followed by fusion. In MACF, the input video is partitioned into disjoint temporal segments,

$V = \{V^{(1)}, V^{(2)}, \dots, V^{(M)}\},$

and each local agent $A_m$ produces compact communication tokens $\mathbf{c}^{(m)} \in \mathbb{R}^{K \times d}$. A central coordinator then computes the final prediction as

$$\hat{y} = A_0(q, \mathbf{c}^{(1)}, \dots, \mathbf{c}^{(M)}),$$

under the communication constraint $M \times K + \mathrm{token}(q) \le B^{com}$ (Chen et al., 1 May 2026). The same structural logic appears in graph-based multi-video reasoning, where each video is first converted into a spatio-temporal graph, then refined within-video, and only afterward fused across videos through Cross-Graph Attention (He et al., 16 Sep 2025).

A second pattern is shared memory or shared latent state. In collaborative memory for video-level learning, multiple clips from the same video are sampled jointly, a global memory $\mathcal{M}$ is constructed by

$\mathcal{M} = Push(\{X_n\}_{n=0}^{N-1}),$

and clip-specific context is read back by

$M_n = Pop(\mathcal{M}, X_n).$

The associative-memory variant writes correlations through outer products and retrieves context by query-based interaction, so each clip attends to other clips from the same video (Yang et al., 2021). Although this framework is not framed as multi-video reasoning in the narrow sense, it exemplifies a core collaborative idea: local observations are upgraded by a representation built from sibling observations.

A third pattern is coordination through constraints rather than concatenation. Some frameworks communicate via continuous latent tokens rather than text, as in MACF (Chen et al., 1 May 2026). Others impose geometric admissibility, as in CVD’s epipolar attention mask (Kuang et al., 2024). Others use graph structure, such as grounded object nodes, intra-frame predicate edges, and inter-frame tracking edges (He et al., 16 Sep 2025). The shift away from naive aggregation is central. Collaboration is rarely implemented as raw token accumulation; it is usually mediated by a structured interface that limits redundancy and specifies what kinds of information transfer are allowed.

3. Joint generation, restoration, and production pipelines

In generative systems, collaboration is often introduced to repair failures of single-clip or single-view synthesis. VGoT is a collaborative, training-free multi-shot video generation framework that starts from a user’s one-sentence story prompt $S$ and a target number of shots $N$, converts the story into shot descriptions $A_m$0, expands them into detailed prompts $A_m$1, generates identity-conditioned keyframes, synthesizes shot-level clips, and then applies a smoothing mechanism at shot boundaries (Zheng et al., 2024). Its collaborative core is organized around dynamic storyline modeling, identity-aware cross-shot propagation, and adjacent latent transition mechanisms. Quantitatively, the framework reports $A_m$2 improvement in within-shot face consistency, $A_m$3 improvement in style consistency, over $A_m$4 better cross-shot consistency, and $A_m$5 fewer manual adjustments than alternatives, with FC (cross-shot) $A_m$6, SC (cross-shot) $A_m$7, FC $A_m$8, and SC $A_m$9 in Table 1 (Zheng et al., 2024).

CVD addresses a different generative problem: multiple videos of the same scene rendered from different camera trajectories. The collaborative denoising objective is written as

$\mathbf{c}^{(m)} \in \mathbb{R}^{K \times d}$0

and the Cross-View Synchronization Module restricts cross-video attention by an epipolar mask based on the fundamental matrix $\mathbf{c}^{(m)} \in \mathbb{R}^{K \times d}$1 with threshold $\mathbf{c}^{(m)} \in \mathbb{R}^{K \times d}$2 (Kuang et al., 2024). The reported identical-trajectory case yields an MSE of $\mathbf{c}^{(m)} \in \mathbb{R}^{K \times d}$3, compared with $\mathbf{c}^{(m)} \in \mathbb{R}^{K \times d}$4 for CameraCtrl and $\mathbf{c}^{(m)} \in \mathbb{R}^{K \times d}$5 for CameraCtrl+SparseCtrl, while RE10K geometric metrics improve to rotation AUC $\mathbf{c}^{(m)} \in \mathbb{R}^{K \times d}$6, translation AUC $\mathbf{c}^{(m)} \in \mathbb{R}^{K \times d}$7, precision $\mathbf{c}^{(m)} \in \mathbb{R}^{K \times d}$8, and matching score $\mathbf{c}^{(m)} \in \mathbb{R}^{K \times d}$9 (Kuang et al., 2024).

Several frameworks generalize collaboration beyond paired video generation. DreamFactory treats long multi-scene synthesis as an AI virtual film production company with roles including CEO, Movie Director, Film Producer, Screenwriter, Filmmaker / Creator / Painter / Art Director, Reviewer, and Monitor, and uses a Key Frames Iteration Design Method in which a Base Description $$\hat{y} = A_0(q, \mathbf{c}^{(1)}, \dots, \mathbf{c}^{(M)}),$$0 extracted from the first keyframe anchors later generation through contextual propagation $$\hat{y} = A_0(q, \mathbf{c}^{(1)}, \dots, \mathbf{c}^{(M)}),$$1 (Xie et al., 2024). RoboMaster models robotic manipulation video generation as a collaborative trajectory problem by decomposing interaction into pre-interaction, interaction, and post-interaction phases and conditioning each phase on the dominant subject; on Bridge V2 it reports FVD $$\hat{y} = A_0(q, \mathbf{c}^{(1)}, \dots, \mathbf{c}^{(M)}),$$2, PSNR $$\hat{y} = A_0(q, \mathbf{c}^{(1)}, \dots, \mathbf{c}^{(M)}),$$3, SSIM $$\hat{y} = A_0(q, \mathbf{c}^{(1)}, \dots, \mathbf{c}^{(M)}),$$4, Robot TrajError $$\hat{y} = A_0(q, \mathbf{c}^{(1)}, \dots, \mathbf{c}^{(M)}),$$5, Object TrajError $$\hat{y} = A_0(q, \mathbf{c}^{(1)}, \dots, \mathbf{c}^{(M)}),$$6, and user preference $$\hat{y} = A_0(q, \mathbf{c}^{(1)}, \dots, \mathbf{c}^{(M)}),$$7 (Fu et al., 2 Jun 2025). VideoFusion, by contrast, is a U-shaped spatio-temporal collaborative network for infrared-visible video fusion and restoration, trained on M3SVD with $$\hat{y} = A_0(q, \mathbf{c}^{(1)}, \dots, \mathbf{c}^{(M)}),$$8 temporally synchronized and spatially registered infrared-visible video pairs comprising $$\hat{y} = A_0(q, \mathbf{c}^{(1)}, \dots, \mathbf{c}^{(M)}),$$9 frames across $M \times K + \mathrm{token}(q) \le B^{com}$0 distinct scenes; it combines CmDRM, CMGF, and BiCAM to generate a fused video $M \times K + \mathrm{token}(q) \le B^{com}$1 as well as restored modality streams (Tang et al., 30 Mar 2025).

Collaboration can also be production-operational rather than purely model-internal. Sima 1.0 organizes long-form documentary creation as an 11-step pipeline distributed across a hybrid workforce in which the human operator determines the topic, reviews source material, drafts the script, records the A-roll, and makes final judgment calls, while junior and senior AI agents handle caption polishing, asset collection, split-level editing, and asset exportation (Song, 9 Apr 2026). The framework formalizes A-roll, B-roll, Type A and Type B assets, and script splits, and reports concrete workload estimates such as about $M \times K + \mathrm{token}(q) \le B^{com}$2 hours of actual recording for a $M \times K + \mathrm{token}(q) \le B^{com}$3-hour video, about $M \times K + \mathrm{token}(q) \le B^{com}$4 hours for manual caption cleanup, about $M \times K + \mathrm{token}(q) \le B^{com}$5 minutes to source and annotate B-roll for a $M \times K + \mathrm{token}(q) \le B^{com}$6-minute segment, and about $M \times K + \mathrm{token}(q) \le B^{com}$7 to $M \times K + \mathrm{token}(q) \le B^{com}$8 minutes to edit a $M \times K + \mathrm{token}(q) \le B^{com}$9-minute segment depending on asset availability (Song, 9 Apr 2026). This suggests that, in practical media pipelines, collaboration may refer as much to task orchestration and interface design as to latent fusion.

4. Long-video understanding and structured multi-video reasoning

In video understanding, collaboration is frequently motivated by bounded context windows and the incompleteness of single-video evidence. MACF formalizes two bottlenecks: a perception budget $\mathcal{M}$0, constraining each agent to inputs satisfying $\mathcal{M}$1, and a communication budget $\mathcal{M}$2, constraining how many latent tokens the coordinator can consume (Chen et al., 1 May 2026). Its solution is an end-to-end Multi-Agent Collaboration Framework in which each agent encodes local observations into compact, task-sufficient latent tokens rather than textual summaries. With Qwen3-VL-8B as backbone, MACF reports gains of $\mathcal{M}$3 on Video-MME, $\mathcal{M}$4 on LongVideoBench, $\mathcal{M}$5 on LVBench, and $\mathcal{M}$6 on MLVU-Test over the plain MLLM under identical perception-budget constraints, and it exceeds text-based communication by $\mathcal{M}$7, $\mathcal{M}$8, $\mathcal{M}$9, and $\mathcal{M} = Push(\{X_n\}_{n=0}^{N-1}),$0 on the same benchmarks (Chen et al., 1 May 2026).

A related but distinct line of work augments a target video with retrieved related videos. The structured multi-video framework of (He et al., 16 Sep 2025) first builds a Video Structuring Module using scene detection, dense captioning, textual scene graph parsing into triplets $\mathcal{M} = Push(\{X_n\}_{n=0}^{N-1}),$1, object grounding, and tracking, then applies a Graph Fusion Module with Hierarchical Frame Graph Attention Network and Cross-Graph Attention, and finally constructs a prompt that keeps the target video’s <VIDEO_TOKENS>, the target graph’s <GRAPH_TOKENS>, and only <GRAPH_TOKENS> for related videos. The paper’s strongest cautionary result is that naive multi-video token concatenation is harmful: single-video reasoning yields $\mathcal{M} = Push(\{X_n\}_{n=0}^{N-1}),$2 accuracy with $\mathcal{M} = Push(\{X_n\}_{n=0}^{N-1}),$3K context, whereas multi-video tokens yield $\mathcal{M} = Push(\{X_n\}_{n=0}^{N-1}),$4 accuracy with $\mathcal{M} = Push(\{X_n\}_{n=0}^{N-1}),$5K context; by contrast, the full structured method reaches $\mathcal{M} = Push(\{X_n\}_{n=0}^{N-1}),$6 accuracy on InternVid-QA, improving over Video-LLaVA by $\mathcal{M} = Push(\{X_n\}_{n=0}^{N-1}),$7 on MSRVTT-QA, $\mathcal{M} = Push(\{X_n\}_{n=0}^{N-1}),$8 on ActivityNet-QA, and $\mathcal{M} = Push(\{X_n\}_{n=0}^{N-1}),$9 on InternVid-QA (He et al., 16 Sep 2025).

VideoChat-M1 extends collaboration from representation fusion to policy planning. Its Collaborative Policy Planning paradigm consists of Policy Generation, Policy Execution, and Policy Communication, with multiple policy agents $M_n = Pop(\mathcal{M}, X_n).$0, a tool set $M_n = Pop(\mathcal{M}, X_n).$1, and a shared memory buffer $M_n = Pop(\mathcal{M}, X_n).$2 (Chen et al., 24 Nov 2025). Each agent produces an initial tool-use plan $M_n = Pop(\mathcal{M}, X_n).$3, executes it step by step, writes intermediate evidence into memory, and may revise the remaining plan according to peer outputs; the system is then optimized by Multi-Agent Reinforcement Learning using $M_n = Pop(\mathcal{M}, X_n).$4, $M_n = Pop(\mathcal{M}, X_n).$5, and $M_n = Pop(\mathcal{M}, X_n).$6. On LongVideoBench, the reported score is $M_n = Pop(\mathcal{M}, X_n).$7, outperforming Gemini 2.5 Pro by $M_n = Pop(\mathcal{M}, X_n).$8 points and GPT-4o by $M_n = Pop(\mathcal{M}, X_n).$9 points, with an average of $S$0 frames per video and $S$1 seconds inference time (Chen et al., 24 Nov 2025). A plausible interpretation is that collaboration in long-video understanding is shifting from evidence selection alone toward jointly learnable planning over tools, memory, and communication.

5. Representative selection, topic context, and video-level memory

Collaborative frameworks predate recent multi-agent video systems in the form of topic-conditioned summarization. Collaborative Video Summarization assumes that a target video $S$2 is accompanied by topic-related companion videos $S$3, and that a good summary should capture both target-specific particularities and topic-level generalities (Panda et al., 2017). Its optimization jointly learns sparse coefficients $S$4 and $S$5 for reconstructing the target and related videos from the target video dictionary, then adds diversity and consensus regularization through the consensus matrix $S$6. On CoSum, CVS achieves mean top-5 AP $S$7 versus $S$8 for CoSum and top-15 AP $S$9 versus $N$0; on TVSum50, top-5 AP is $N$1 versus $N$2, and top-15 AP is $N$3 versus $N$4 (Panda et al., 2017). The ablation CVS-Neighborhood drops to $N$5, indicating that neighborhood context is more important than the diversity penalty in this formulation (Panda et al., 2017).

Collaborative memory addresses a different but structurally related problem: weak temporal coverage during training. Instead of summarizing one video with other videos, the method samples multiple clips from the same video, aggregates them into a memory, and optimizes both per-clip predictions and the averaged video-level logit

$N$6

with loss

$N$7

The method reports gains across Kinetics-400, Kinetics-700, Charades, Something-Something-V1, and AVA, including Slow-only-50 $N$8 improving from $N$9 to $A_m$00, SlowFast-101 $A_m$01 improving from $A_m$02 to $A_m$03, and Charades SlowFast-101+NL $A_m$04 improving from $A_m$05 to $A_m$06, with about $A_m$07–$A_m$08 FLOPs overhead (Yang et al., 2021). Although this framework is clip-centric rather than multi-video in the strict sense, it clarifies a broader principle: collaboration can be introduced by making multiple partial observations jointly responsible for a video-level objective.

6. Distributed streaming, caching, and multi-camera control

In networking and edge systems, multi-video collaboration is driven by resource allocation rather than semantic reasoning. The MEC framework of (Tran et al., 2016) models a network of $A_m$09 MEC cache servers, a library of $A_m$10 videos, and $A_m$11 bitrate versions per video. Cache placement is version-aware through binary variables $A_m$12, constrained by

$A_m$13

and requests can be served through direct local cache hit, local cache plus local transcode, remote cache retrieval, remote cache plus remote transcode plus transfer, or remote cache retrieval plus transcode at the home BS (Tran et al., 2016). The joint caching and processing problem is formulated as an ILP that minimizes backhaul cost under cache and processing constraints, and the proposed online JCCP algorithm, built on LRU replacement, is reported to significantly outperform CachePro and CoCache in cache hit ratio, average access delay, and external backhaul traffic load under a setup with $A_m$14 MEC servers, $A_m$15 videos, $A_m$16 bitrate versions, Zipf popularity with $A_m$17, and $A_m$18 requests per BS (Tran et al., 2016).

Multi-camera fast-forwarding introduces a more explicitly perceptual form of system-level collaboration. In MFFNet and DMVF, each camera agent dynamically chooses slow, normal, or fast pace and skips frames based on reinforcement learning, while inter-view overlap determines which views should be processed more carefully (Lan et al., 2023). MFFNet’s heuristic controller selects main views by maximizing a coverage objective over match counts $A_m$19, assigns slow pace to main views, and chooses normal or fast pace for the remaining views according to their redundancy. Compared with FFNet, the centralized method reduces processing rate by $A_m$20 on VideoWeb 3-view, $A_m$21 on VideoWeb 6-view, and $A_m$22 on CarlaSim, and reaches $A_m$23 FPS on VideoWeb 6-view in the embedded deployment (Lan et al., 2023).

DMVF is the distributed counterpart. Each agent fast-forwards its stream in adaptation periods, exchanges only selected frames with neighbors in a connected graph $A_m$24, computes local importance scores by frame similarity $A_m$25, and reaches a shared global vector $A_m$26 through maximal consensus (Lan et al., 2020). On VideoWeb, DMVF reports coverage $A_m$27 at processing rate $A_m$28, compared with FFNet’s $A_m$29 coverage at $A_m$30 processing rate, Random’s $A_m$31 at $A_m$32, Uniform’s $A_m$33 at $A_m$34, and $A_m$35-processing baselines OK, SC, and SMRS (Lan et al., 2020). Communication overhead ranges from $A_m$36 GB to $A_m$37 GB, or about $A_m$38 to $A_m$39 of raw input data, and average frame rate reaches $A_m$40 FPS overall and $A_m$41 FPS on Jetson TX2 using only ARM cores (Lan et al., 2020). These results show that collaboration can be formalized as distributed control over who processes which video evidence, not only as joint content representation.

7. Evaluation regimes, limitations, and open problems

Evaluation remains heterogeneous because different collaborative objectives induce different failure modes. DreamFactory introduced Cross-Scene Face Distance Score and Cross-Scene Style Consistency Score, and contributed the Multi-Scene Videos Dataset with over $A_m$42 human-rated videos, specifically to measure long-video consistency beyond standard FID, IS, CLIP Score, FVD, and KVD (Xie et al., 2024). VGoT reports within-shot face consistency, style consistency, cross-shot consistency, and human evaluation, but also notes that evaluation still relies mostly on face and style consistency and that there is a lack of robust metrics that fully capture narrative flow and multi-shot coherence (Zheng et al., 2024). VideoFusion further notes that a variant without CMGF can score well on SF/SD despite obvious artifacts and distortions, implying that conventional metrics alone may not capture true sequence quality (Tang et al., 30 Mar 2025).

A second recurrent issue is constraint mismatch between modules. VGoT explicitly reports that richer scripts improve keyframes and narrative structure, but overly detailed prompts can make the video model less dynamic because the underlying video generator was not trained on such detailed scripts (Zheng et al., 2024). MACF shows a related budget trade-off: $A_m$43 hurts performance noticeably, $A_m$44 improves significantly, and $A_m$45 gives only small additional gains, indicating saturation; increasing the number of agents $A_m$46 improves performance by increasing temporal coverage, but communication and coordinator limits remain explicit (Chen et al., 1 May 2026). In graph-based multi-video reasoning, performance rises as more related videos are added up to $A_m$47, then falls when too many are added, and lower-similarity retrieved videos reduce accuracy, even though structured fusion is more robust than raw token concatenation (He et al., 16 Sep 2025).

A third issue is structural assumption. CVD assumes synchronized videos and, during training, mostly uses video pairs rather than true large multi-view dynamic sequences; it also does not inherently fix internal inconsistencies inherited from the base generators AnimateDiff and CameraCtrl (Kuang et al., 2024). VGoT states that each shot currently receives only a single identity-preserving embedding, making complex multi-character scenes harder to represent (Zheng et al., 2024). These limitations suggest that collaboration does not remove the need for stronger base models, better retrieval, or richer supervision; rather, it reorganizes where the system attempts to recover missing structure.

Taken together, the literature indicates that multi-video collaborative frameworks are converging on three broad claims. First, isolated video processing is frequently inadequate when coherence, reasoning completeness, or resource efficiency is a global property. Second, collaboration is most effective when it is implemented through structured interfaces—latent tokens, graphs, memories, geometric masks, or constrained service paths—rather than through unrestricted concatenation. Third, the principal open problems are no longer only generative fidelity or benchmark accuracy, but also metric validity, retrieval quality, multi-entity coordination, and the design of collaboration protocols that remain stable under strict compute, context, and bandwidth constraints.