Mosaic: Cross-Modal Clustering for Efficient Video Understanding

Abstract: Large vision-LLMs (VLMs) are enabling interactive video reasoning, giving rise to streaming long-video understanding. In this setting, frames arrive continuously, while the system preserves long-term context and generates responses under strict latency constraints. A central challenge is KVCache management: as video streams grow, KVCache expands rapidly, increasing computation and memory overhead. Existing retrieval-based approaches exploit attention sparsity and offload inactive KVCache from GPU to CPU memory, but their token-level design causes high management overhead and fragmented data movement. We present Mosaic, the first cluster-driven VLM inference system for streaming long-video understanding. Our key insight is that VLM KVCache exhibits an implicit cross-modal clustering structure: retrieved KV states form groups jointly shaped by visual coherence and semantic relevance. Based on this observation, Mosaic uses cross-modal clusters as the basic unit of KVCache organization, maintenance, and retrieval. Evaluations show that Mosaic outperforms state-of-the-art baselines, achieving up to 1.38x speedup.

• The paper introduces a novel cross-modal clustering approach that organizes KVCache at a cluster level, enhancing computational and memory efficiency.
• It employs a hierarchical clustering strategy using cosine-distance normalized K-Means to capture both visual and semantic similarities in video frames.
• Empirical results show up to 2.5% accuracy gains, 1.38x lower latency, and 2.22x reduced GPU memory usage compared to token-level methods.

Mosaic: Cross-Modal Clustering for Efficient Streaming Long Video Understanding

Introduction and Motivation

The proliferation of large vision-LLMs (VLMs) has catalyzed a shift towards interactive, streaming long-video understanding. Operating under stringent latency and resource constraints, modern VLM-based systems must efficiently manage exponentially growing key-value caches (KVCache) as videos are ingested frame-by-frame. Existing retrieval-based solutions rely almost exclusively on token-level granularity for KVCache management, which incurs substantial systems overhead and leads to fragmented data movement due to the fine-grained and disjoint selection of memory segments.

To address these inefficiencies, this work introduces Mosaic, an inference system that reconceptualizes KVCache management by leveraging the observation that VLM retrieval inherently exhibits cross-modal clustering—the system generally activates visually and semantically coherent groups of frames rather than isolated tokens. This enables cluster-level, rather than token-level, organization and retrieval of KVCache entries, which dramatically improves computational, memory, and I/O efficiency while preserving or exceeding state-of-the-art accuracy.

Figure 1: Comparison of token-level and cluster-level KVCache management.

Cross-Modal Clustering: Abstraction and Construction

Mosaic formalizes KVCache retrieval as a nested cross-modal clustering problem. At the first stage, incoming video frames are partitioned by visual embedding similarity, capturing coarse scene and object coherence. Within each visual cluster, a secondary, semantic layer-specific clustering is performed using transformer key embeddings, yielding fine-grained segments that more precisely match the actual retrieval and attention patterns observed during decoder execution.

Figure 3: Visualization of KVCache clustering in visual space (left) and semantic space (right) across models and layers on MLVU.

This hierarchical clustering is implemented with a cosine-distance, normalized K-Means algorithm adapted to both embedding spaces. Rather than producing static clusters, which quickly become obsolete in streaming settings, Mosaic tracks intra-cluster variance to dynamically maintain cluster quality, triggering deferred splits only when necessary, thereby amortizing maintenance overhead along the natural retrieval pathway.

Figure 5: Nested cross-modal clustering and two-stage retrieval indexing.

Clusters are represented by centroids (cluster representatives), forming a GPU-resident index that enables fast two-stage retrieval. Queries first select relevant visual clusters; only within these partitions is layer-specific semantic selection performed. This not only aligns the retrieval process with actual attention patterns, but also maximizes memory and I/O efficiency by ensuring that large, semantically coherent blocks are moved as units between CPU and GPU.

Systems Optimizations: Maintenance and Execution

To further enhance efficiency, Mosaic integrates several key system-level optimizations:

• Self-Adaptive Maintainer: The clustering structure is dynamically updated using a cohesion-aware, variance-guided assignment. Deferred splitting avoids unnecessary I/O by postponing cluster refinement unless an invalidated cluster is actively needed for retrieval.
• Batch-Oriented Execution: By identifying pipeline components that can be decoupled from strict temporal constraints, Mosaic processes visual encoding, cluster retrieval, and feed-forward network operations in large batches, reducing GPU kernel-launch and data-transfer overhead.
• Overlap-Aware Prefetch: The strong residual similarity between adjacent transformer layers is exploited to prefetch probable cluster candidates for subsequent layers, overlapping computation and I/O to further mask latency.

  Figure 2: Impact of deferred splitting on maintenance overhead on MLVU.

  

  Figure 8: (a) Normalized frame encoding time with and without batched execution. (b) Normalized response latency with and without cross-layer prefetch.

Empirical Results

Mosaic is evaluated on three VLM architectures and five long-video understanding benchmarks (including real-world streaming video from Kuaishou). Across the board, Mosaic achieves superior performance in terms of accuracy, latency, memory efficiency, and scalability relative to prominent baselines such as NoCache, ReKV, LiveVLM, and StreamMem.

Key findings include:

• Accuracy: Mosaic consistently outperforms token-level methods by up to 2.5% across benchmarks and models, with especially significant gains under restricted retrieval budgets.
• Latency: Mosaic reduces retrieval latency up to $1.38\times$ compared to the next-best method, with system optimizations (batch execution, overlap-aware prefetch) yielding an additional 26% (frame encoding) and 14.5% (latency) improvements, respectively.
• Memory Efficiency: Cluster-level management results in up to $2.22\times$ reduction in GPU memory usage for long video sequences.
• Scalability: Frame encoding throughput scales linearly with GPU count due to embarrassingly parallel clustering and retrieval computations.

  Figure 4: Performance comparison (normalized) between ReKV, LiveVLM, and Mosaic across models and datasets.

  

  Figure 9: Performance of LLaVA-OV-7B on MLVU across retrieval frames.

  

  Figure 6: GPU memory usage of LLaVA-OV-7B across different video lengths.

  

  Figure 7: Performance comparison across models and datasets with different video lengths (Qwen denotes Qwen2.5-VL-7B).

  

  Figure 10: Strong scaling evaluation across different GPU counts.

Analysis and Theoretical Implications

The central contribution of Mosaic—using cross-modal clustering as the atomic unit for KVCache management—suggests a paradigm shift in the design of scalable, streaming multimodal systems. The approach is orthogonal to changes in model architecture or training, and can be incorporated into existing VLM systems as a purely inference-time optimization.

This work demonstrates that modeling the actual cross-modal access patterns, rather than treating each token or frame as an independent retrieval entity, unlocks significant computational and memory resource savings. The deferred splitting and batch-oriented design further reflect a systematic alignment between algorithmic abstractions and underlying hardware execution characteristics.

Future systems exploiting even finer temporal and hierarchical structure (e.g., segmenting based on detected events or action boundaries), or leveraging scheduled speculative prefetch based on global query distributions, could extend the generality and efficiency gains pioneered by Mosaic's design.

Conclusion

Mosaic substantiates the efficacy of cross-modal, cluster-driven KVCache management for streaming long-video understanding in VLMs. Its algorithm-system co-design delivers strong and robust improvements in accuracy, memory, latency, and scalability over prevailing token-level methods, and establishes a compelling template for future research on efficient online multimodal inference (2604.10060).