MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generation (2510.18692v1)

Abstract: Long video generation with Diffusion Transformers (DiTs) is bottlenecked by the quadratic scaling of full attention with sequence length. Since attention is highly redundant, outputs are dominated by a small subset of query-key pairs. Existing sparse methods rely on blockwise coarse estimation, whose accuracy-efficiency trade-offs are constrained by block size. This paper introduces Mixture-of-Groups Attention (MoGA), an efficient sparse attention that uses a lightweight, learnable token router to precisely match tokens without blockwise estimation. Through semantic-aware routing, MoGA enables effective long-range interactions. As a kernel-free method, MoGA integrates seamlessly with modern attention stacks, including FlashAttention and sequence parallelism. Building on MoGA, we develop an efficient long video generation model that end-to-end produces minute-level, multi-shot, 480p videos at 24 fps, with a context length of approximately 580k. Comprehensive experiments on various video generation tasks validate the effectiveness of our approach.

• The paper introduces MoGA, a novel attention mechanism that uses dynamic, learnable token routing to group semantically similar tokens for efficient long video generation.
• It integrates MoGA with local spatiotemporal attention and cross-modal conditioning to ensure both global coherence and fine-grained detail across multi-shot videos.
• Experimental results demonstrate significant improvements in subject consistency, motion smoothness, and computational efficiency, achieving up to 1.7× speedup over full attention.

MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generation

Introduction and Motivation

Long video generation with diffusion transformers (DiTs) is fundamentally constrained by the quadratic computational complexity of full self-attention, which becomes prohibitive as context length increases. Existing approaches to mitigate this challenge—such as block-sparse attention and context compression—either introduce coarse approximations that degrade performance or incur information loss. The Mixture-of-Groups Attention (MoGA) framework addresses these limitations by introducing a dynamic, learnable token routing mechanism that enables precise, semantically-aware grouping of tokens, facilitating efficient and scalable attention for minute-level, multi-shot video generation.

Figure 1: Illustration of the motivation for MoGA, contrasting the inefficiency of full attention and the limitations of block-sparse attention with the proposed groupwise routing approach.

MoGA Architecture and Mechanism

MoGA replaces block-level scoring with a lightweight, trainable router—a single linear layer—that assigns each token to one of $M$ groups based on learned semantic affinities. Full self-attention is then performed independently within each group, reducing the computational complexity from $\mathcal{O}(N^2)$ to approximately $\mathcal{O}(N^2/M)$, where $N$ is the sequence length. This approach is kernel-free and integrates seamlessly with high-performance attention kernels such as FlashAttention, as well as with sequence parallelism frameworks.

The overall architecture interleaves visual attention (using MoGA and local spatiotemporal group attention, STGA) with cross-modal attention blocks for shot-level text conditioning. This design enables both global coherence and local fidelity, and supports multi-shot, long-form video generation with dense, shot-level textual prompts.

Figure 2: The DiT-based architecture with interleaved visual and cross-modal attention, highlighting the integration of MoGA and STGA for global-local consistency.

The dynamic router's group assignments are semantically meaningful, as evidenced by visualizations showing that tokens corresponding to coherent entities (e.g., a person's head and hands) are grouped together, even across shot boundaries.

Figure 3: Visualization of dynamic router grouping, demonstrating semantically coherent token assignments.

Data Pipeline for Long Video Generation

A robust data pipeline is constructed to support training on minute-level, multi-shot videos. The pipeline includes video-level quality assessment, shot segmentation using AutoShot and PySceneDetect, OCR-based cropping to remove watermarks and subtitles, and dense captioning with a multimodal LLM. Clean, multi-shot training samples are assembled by merging temporally adjacent, high-quality single-shot clips.

Figure 4: Multi-shot long video data pipeline, detailing the stages from raw video filtering to multi-shot sample assembly.

Experimental Results

Quantitative Evaluation

MoGA is evaluated on multiple video generation tasks, including single-shot, multi-shot, and long-form video synthesis. Across all settings, MoGA consistently outperforms both full attention and state-of-the-art sparse attention baselines in subject consistency, background consistency, motion smoothness, aesthetic quality, and image quality. Notably, MoGA achieves these results with significantly higher sparsity (e.g., 71.25% for 5-second videos) and substantial reductions in computational cost.

Qualitative Evaluation

Qualitative results demonstrate MoGA's ability to maintain long-range content coherence, character consistency, and high visual quality across extended video durations and multiple shots. The model preserves fine-grained details and avoids identity confusion, even in one-minute, multi-shot videos.

Figure 5: Qualitative results of MoGA and other methods, illustrating long-content coherence and visual quality across multiple shots.

Figure 6: One-minute video generated by MoGA, showing strong long-range contextual consistency and detail preservation.

MoGA also exhibits emergent background consistency and supports multi-style video generation, including high-quality, temporally coherent animation.

Figure 7: Emergence of background consistency across shots in long videos.

Figure 8: Animation style generation for MoGA, demonstrating versatility across visual domains.

Computational Efficiency

MoGA achieves substantial computational savings compared to full attention. For example, generating a 30-second video with $M=5$ groups reduces PFLOPs from 6.94 (full attention) to 2.26, with a $1.7\times$ speedup in both training and inference. Increasing the number of groups further reduces FLOPs, with minimal impact on consistency metrics up to a moderate group count.

Figure 9: Computational efficiency of MoGA as a function of group number and video duration.

Ablation Studies

Ablation studies confirm that the group balancing loss is essential for preventing router collapse and ensuring uniform token distribution across groups, which is critical for maintaining both efficiency and performance.

Figure 10: Group balancing loss curves of MoGA, showing the effect of the auxiliary loss on token assignment balance.

Further analysis demonstrates that MoGA and STGA are complementary: MoGA enables long-range, cross-shot consistency, while STGA ensures local continuity. Their combination is necessary for optimal performance in long video generation.

Figure 11: Visual ablation of MoGA and STGA, highlighting their complementary roles in context-consistent video generation.

Implementation Considerations

• Router Design: The router is a single linear layer with softmax gating, enabling efficient, differentiable group assignment.
• Integration: MoGA is kernel-agnostic and compatible with FlashAttention and sequence parallelism, facilitating deployment in large-scale, high-throughput settings.
• Group Number Selection: There is a trade-off between computational efficiency and global consistency; moderate group counts (e.g., $M=5$–$20$) yield near-optimal results.
• Auxiliary Loss: The group balancing loss is critical for preventing degenerate group assignments and should be included during training.
• Data Requirements: High-quality, densely annotated, multi-shot video data is necessary to fully exploit MoGA's capabilities.

Implications and Future Directions

MoGA demonstrates that precise, learnable token routing can overcome the limitations of both full and block-sparse attention in long-context video generation. The approach is scalable, efficient, and generalizable, with potential applications in other domains requiring long-range sequence modeling (e.g., audio, text, multimodal tasks). Future work may explore hierarchical or adaptive group assignment, integration with memory-augmented architectures, and further scaling to even longer contexts and higher resolutions.

Conclusion

MoGA introduces a principled, efficient sparse attention mechanism for end-to-end long video generation, leveraging dynamic, semantically-aware token grouping to achieve state-of-the-art performance at scale. The framework sets a new standard for computational efficiency and fidelity in long-form video synthesis, with broad implications for the design of scalable generative models.