VideoNSA: Scalable Sparse Video Attention

• VideoNSA is a sparse attention framework that dynamically gates and selects salient tokens, enabling robust global and local context preservation in ultra-long video streams.
• Its hardware-aware hybrid design achieves up to 128K token processing with only 3.6% of the computation required by full dense attention, ensuring efficient scalability.
• Empirical evaluations demonstrate enhanced long-video understanding and temporal reasoning on benchmarks, mitigating issues like attention sinks with its dynamic branch selection.

VideoNSA is a sparse attention framework for video–LLMs that addresses the challenge of scaling video understanding to ultra-long video contexts in multimodal LLMs. Standard transformer attention is resource-limited by quadratic complexity in context length, which impedes the ability of vision–LLMs to maintain temporal coherence, capture key transitions, and reason over hundreds of thousands of video tokens. VideoNSA introduces a hardware-aware, dynamically gated sparse attention mechanism that allows scaling to input lengths well beyond typical dense attention regimes, preserving both global and local context critical for complex video–language tasks.

1. Motivation and Problem Formulation

Contemporary vision–LLMs (VLMs), even with advanced architectures, are severely restricted by context size due to quadratic attention cost, leading to information loss for long videos—such as missing transition frames or failing to establish long-range dependencies. Token-compression strategies and naive sparsity often lead to irreversible information loss. VideoNSA remedies these issues by adapting native sparse attention (NSA) to the video token stream, enabling both scalable context handling and robust temporal reasoning by selectively attending to salient temporal and spatial regions without lossy compression.

2. Native Sparse Attention Architecture

VideoNSA modifies the attention backbone of a strong base model (Qwen2.5-VL-7B) to employ a specialized, hardware-aware hybrid attention scheme:

• Sparse Attention for Video Tokens: All vision tokens extracted from video frames are routed through the NSA mechanism, which partitions the input into blocks and provides three complementary attention branches:
  • Compression branch ("cmp"): Aggregates tokens within blocks using a learnable MLP $\phi$, emphasizing block-level abstraction.
  • Selection branch ("slc"): Scores blocks and selects the top-k most salient tokens, retaining only key information.
  • Sliding Window branch ("win"): Attends locally within a fixed-size temporal window ($w=256$).
• Dynamic Gating: For each token, a two-layer MLP (with sigmoid activation) produces modulation weights $g_t^c$ (for $c \in \{\mathrm{cmp}, \mathrm{slc}, \mathrm{win}\}$), dynamically fusing the outputs:

$$o_t = \sum_{c \in \{\mathrm{cmp}, \mathrm{slc}, \mathrm{win}\}} g_t^c \cdot \mathrm{Attn}(q_t, \widetilde{K}_t^c, \widetilde{V}_t^c)$$

where each branch provides candidate key and value sets.

• Dense Attention for Text Tokens: Dense (grouped-query) attention is applied to natural language tokens to maintain high-quality instruction-following and text understanding.

This hybrid attention assignment is tuned for both accuracy and computational efficiency—video tokens experience sparsity for scale; text tokens maintain dense, language-driven interactions.

3. Training Regime and Data Curation

VideoNSA is trained end-to-end on a curated instruction dataset derived from LLaVA-Video-178K, filtered to form a 216K long-form video instruction dataset. Content consists of video–question–answer pairs with 350–550 frames sampled at 4 fps, designed to induce both temporal redundancy and the challenges of long-term context. Training is performed over 4600 H100 GPU hours using the SWIFT infrastructure, with up to 36K token contexts during training and extrapolation to 128K tokens during evaluation. The NSA parameters used are block size $s=64$, stride $b=32$, and window size $w=256$.

This setup ensures exposure to temporal redundancies, spatial transitions, and the full spectrum of long-video linguistic instructions, encouraging the model to exploit both block-level global structure and local continuity.

4. Empirical Evaluation and Benchmark Performance

VideoNSA is benchmarked against both token-compression (which typically drops token information) and training-free sparse baselines. Evaluations span multiple video understanding benchmarks:

• LongVideoBench
• MLVU
• TimeScope
• LongTimeScope
• Tomato (temporal reasoning)
• VSIBench (spatial reasoning)

Results show:

• Consistently improved accuracy in long-video understanding and temporal ordering.
• High spatial and local fidelity, preserving fine-grained details often lost in compression-based schemes.
• Reliable scaling to 128K token input lengths, with performance maintained even when evaluation context greatly exceeds training context.
• Strong performance at roughly 3.6% of the computational budget required by full dense attention.

These results validate that NSA-based dynamic sparsification exploits both redundancy in the video signal and the need for non-local context to drive robust video–language reasoning.

5. Ablation Studies and Analytical Findings

A series of systematic ablations provide critical insights:

1. Scaling Reliability: VideoNSA generalizes to longer contexts (up to 128K tokens) without significant performance loss, even beyond the training horizon.
2. Global–Local Attention Budget: Under fixed compute, allocating more of the sparse attention budget to global (block-level) attention yields superior results over simply expanding local sliding windows.
3. Branch Usage Patterns: Gating weights ($g_t^c$) reveal that different branches are exploited in a depth- and task-dependent manner: selection and window dominate in early layers (capturing local events and salient frames), while compression takes precedence in late layers (promoting global abstraction and redundancy reduction).
4. Dynamic Attention Sinks: NSA's learnable gating mechanism mitigates the risk of attention sinks (i.e., tokens attracting disproportionate attention, especially at the sequence start). The compression branch is more prone to attention sinks, but the selection branch suppresses them, and the window branch introduces regular, low-frequency sink patterns. The aggregate sink ratio is maintained at approximately 0.3%, aiding in preserving the temporal spread.

6. Hardware-Efficient and Practical Considerations

VideoNSA is designed for hardware efficiency—by combining block-sparse operations, custom gating, and dense attention for short texts, the method achieves significant memory and compute savings. NSA's parallelism aligns with accelerator architecture. The compression branch, while a computational bottleneck, is isolated and optimizable.

• Contexts up to 128K tokens can be processed without memory overflow on top-tier accelerators.
• Selective block and window strategies make VideoNSA deployable in resource-constrained or latency-aware settings.

Planned improvements include further kernel optimization of the compression branch and adaptation to additional domains with diverse temporal structure.

7. Implications and Future Directions

VideoNSA demonstrates that learnable, hardware-aware sparse attention enables multimodal LLMs to reason over ultra-long videos without irreversible token loss. It establishes that dynamic global-local hybrid attention is essential for maintaining both high-fidelity local structure and holistic temporal coherence. Its findings suggest:

• Allocation strategies between global and local attention should be task- and scale-dependent.
• Dynamic branch selection can mitigate classic attention artifacts such as attention sinks.
• Efficient ultra-long context vision–LLMs become feasible, paving the way for systems that can, for example, summarize entire movies, analyze sports games for decisive transitions, or support surveillance scenarios requiring minute-scale temporal integration.

In sum, VideoNSA establishes a new paradigm for scalable video–language modeling, balancing context, resource, and information preservation through dynamic learnable sparse attention mechanisms (Song et al., 2 Oct 2025).