Scaling Video Understanding via Compact Latent Multi-Agent Collaboration

Abstract: Multi-modal LLMs (MLLMs) advance vision language understanding but face inherent limitations in long-video tasks due to bounded perception context budgets. Existing agentic methods mitigate this via rule-based preprocessing, yet often suffer from information loss, high cost, and reliance on textual intermediates. We propose MACF, an end-to-end Multi-Agent Collaboration Framework that decouples per-agent perception budgets from global video complexity, enabling scalable video understanding while preserving visual fidelity. MACF partitions videos into segments for locally budgeted agents and enables holistic reasoning via an agent-native latent communication protocol. Each agent encodes partial observations into compact, task-sufficient tokens in a shared embedding space, allowing efficient and information-preserving collaboration by a central coordinator. We introduce a curriculum training strategy that progressively enforces semantic alignment, evidence summarization, and cross-agent coordination. Extensive experiments on diverse video understanding benchmarks show that MACF consistently outperforms state-of-the-art MLLMs and multi-agent systems under identical budget constraints, demonstrating the effectiveness of our latent collaboration for scalable video understanding.

• The paper introduces MACF, an end-to-end multi-agent system that partitions video inputs and uses latent token communication for efficient long-video processing.
• MACF achieves significant performance gains (up to 7.7% increase) on benchmarks by hierarchically aggregating observations and reducing information loss.
• A three-stage curriculum training protocol ensures semantic alignment, compact evidence summarization, and robust cross-agent collaboration under strict resource constraints.

Compact Latent Multi-Agent Collaboration for Scaling Video Understanding

Motivation and Problem Formulation

The paper "Scaling Video Understanding via Compact Latent Multi-Agent Collaboration" (2605.00444) addresses a core limitation of contemporary multimodal LLMs (MLLMs): their inability to process long-form video inputs due to fixed context budgets. While current agentic methods, such as rule-based perceptual sampling or frame retrieval, alleviate input bottlenecks, they incur significant information loss, excessive computational cost, and either depend on lossy text intermediates or lack end-to-end optimization.

To resolve these challenges, the authors introduce the Multi-Agent Collaboration Framework (MACF), designed as an end-to-end solution for scalable video understanding under explicit perceptual and communication budget constraints. MACF hierarchically partitions a video into disjoint temporal segments, each processed by a locally budgeted agent. These agents encode partial video observations and communicate with a coordinator agent via compact latent tokens embedded in a joint representation space, facilitating efficient aggregation and holistic reasoning.

Figure 1: Limitations of perceptual sampling and retrieval versus the MACF latent multi-agent framework.

This decouples per-agent processing capacity from global video complexity and substantially improves efficiency and visual fidelity retention compared to traditional approaches relying on perceptual downsampling or text-based summaries.

MACF: Design and Optimization

Framework Architecture

MACF employs a star-topology multi-agent architecture:

• Local Agents with Budgeted Perception: Each agent processes a video segment under a fixed pixel budget, uniformly samples frames, and encodes them into compact continuous communication tokens.
• Latent Collaboration: Information across agents is exchanged via fixed-capacity continuous tokens, facilitating bandwidth-efficient, loss-minimized communication.
• Coordinator Agent: Aggregates communication tokens from all local agents and the task query to produce a global output.

  Figure 2: MACF pipeline: local agent perception, latent communication token exchange, and coordinator-based fusion. Also shown is the curriculum training protocol.

This enables the system to process much longer videos by linearly increasing the number of agents, while retaining high spatial and temporal fidelity per agent, as opposed to suffering context reductions through global frame thinning.

Curriculum Training Protocol

Direct end-to-end optimization for this asymmetric, bandwidth-constrained collaboration is non-trivial. To achieve stable and effective training, the authors propose a three-stage curriculum:

1. Semantic Alignment: Agents learn to anchor communication tokens in a unified latent space using caption supervision.
2. Evidence Summarization: Agents are fine-tuned for query-aware compact evidence extraction using high-quality Image-QA data.
3. Cross-Agent Collaboration: The coordinator is trained to fuse distributed evidence from several local agents and answer Video-QA tasks, optimizing over global context under realistic information bottlenecks.

This staged optimization ensures the communication channel is semantically grounded, task-relevant, and tuned for cross-agent coordination, mitigating the drift and instability typical of naive multi-agent latent protocol learning.

Quantitative and Qualitative Evaluation

The MACF framework is empirically validated on four demanding benchmarks: Video-MME, LongVideoBench, LVBench, and MLVU-Test. The experimental setup enforces strict per-agent perception and end-to-end communication budgets. Qwen3-VL-8B is the principal backbone, with validation across several other MLLM bases, confirming the generality of MACF.

Numerical Results: MACF consistently outperforms both state-of-the-art single-model MLLMs and agentic baselines under identical perception budgets—often by notable margins. Compared to strong MLLM baselines, MACF produces gains of +4.5% (Video-MME), +6.1% (LongVideoBench), +7.0% (LVBench), and +7.7% (MLVU-Test). Furthermore, MACF's agent-native latent communication outperforms text-based (MapReduce-style) and KV-cache-sharing (LatentMAS) paradigms, yielding relative improvements as high as 20.3% on Video-MME.

Ablation on Latent Communication:

Figure 3: Comparison of text-based (MapReduce) and MACF's latent communication. Textual loss of detail propagates cross-agent error, while latent tokens preserve fine-grained evidence.

Ablations confirm the indispensable role of each curriculum stage, with removal of any stage leading to double-digit performance loss—highlighting the necessity of progressive alignment, evidence selection, and collaboration learning.

Analysis of Information Bottlenecks

The study analyzes the dual bottlenecks—perception and communication—that regulate the scalable capacity of video understanding systems.

1. Perception Bottleneck: Defined by the number of agents, frames per agent, and spatial resolution. MACF achieves linear scaling in temporal coverage with growing agent pool size (Figure 4).

   Figure 4: Increasing the local agent count $M$ directly scales temporal perception capacity and performance.

2. Spatial Fidelity: Larger input resolutions boost local evidence fidelity up to the limit of communication capacity (Figure 5).

   Figure 5: Increasing spatial capacity improves performance, with diminishing returns past the communication bottleneck.

3. Communication Bottleneck: Evaluated via the number of latent tokens communicated per agent. Performance monotonically increases up to the point where additional bandwidth yields minimal further gains, aligning with rate-distortion theory.

Overall, MACF exposes a coupled trade-off between per-agent perception volume and aggregate communication bandwidth, revealing a regime where task-relevant information is preserved with minimal communication redundancy.

Model Generality and Efficiency

MACF demonstrably generalizes across heterogeneous local agents and backbones due to a learnable adaptor module that projects all representations into a shared latent space. The framework is model-agnostic, evidenced by stable performance in mixed-backbone deployments. Additionally, MACF yields the lowest communication cost and inference latency among compared approaches, affirming its practical efficiency.

Implications and Future Prospects

The principal theoretical implication is that latent-space multi-agent collaboration can fundamentally decouple scaling constraints imposed by MLLM fixed context sizes, providing a direct pathway for tractable long-form video understanding with high evidence fidelity. Practically, MACF enables deployment in compute- and bandwidth-limited settings, such as edge devices, surveillance analysis, long-duration event QA, and multi-modal content analytics.

Future research directions involve:

• Adaptive segment allocation and dynamic perception-communication trade-offs.
• Extension to multi-modal evidence aggregation with agent heterogeneity.
• Exploration of non-star topologies and more advanced latent communication protocols.
• Transfer and adaptation of MACF-style curriculum training for other tasks (e.g., audio-visual reasoning, action planning).

Conclusion

MACF establishes a robust latent multi-agent paradigm for video understanding under explicit resource constraints, outperforming both current monolithic and agentic baselines via its efficient, bandwidth-aligned, and information-preserving architecture. The proposal of an end-to-end differentiable, curriculum-trained system marks a significant advance in practical scalable video perception and sets a foundation for future work in large-scale multi-agent multimodal reasoning under real-world constraints.