OmniDance: Multimodal Driven Dance Video Generation with Large-scale Internet Data

Abstract: Music-driven dance video generation aims to synthesize expressive human motion that is temporally aligned with music while maintaining high visual fidelity. Despite recent progress, existing methods still face two key limitations: the lack of large-scale, high-quality dance video datasets, and the absence of principled frameworks for integrating music as a complementary conditioning signal into Video Generation Foundation Models. To address these limitations, we introduce CIPE-Dance, a large-scale Internet-sourced dance video dataset with choreography-informed text annotations, constructed via a progressive expert pipeline. To the best of our knowledge, CIPE-Dance is the largest dataset for dance video generation to date, comprising 300k high-quality clips over 400 hours and covering diverse dancers, environments, and dance genres. We further propose OmniDance, a framework-level recipe for integrating music into a TI2V foundation model without sacrificing its original controllability or visual fidelity. Motivated by the complementary roles of text as low-frequency semantics and music as high-frequency temporal dynamics, OmniDance co-designs a depth-aware specialization architecture, an anchored easy-to-hard curriculum learning strategy, and a modality-specialized time-dependent CFG strategy, enabling unified TI2V, MI2V, and MTI2V generation. Extensive experiments on CIPE-Dance demonstrate that OmniDance achieves state-of-the-art performance across all three tasks and exhibits robust multimodal integration capability. Project is available at https://github.com/AMAP-ML/OmniDance.

• The paper introduces a unified framework that integrates text and music conditioning to generate high-fidelity dance videos.
• It demonstrates strong performance across various modalities using a scalable dataset covering over 30 dance genres.
• The study employs curriculum learning and depth-aware specialization to enhance motion synchronization and visual fidelity.

OmniDance: Multimodal-Driven Dance Video Synthesis at Scale

Introduction

OmniDance introduces a unified framework for large-scale, multimodal dance video generation, addressing limitations in dataset availability and model design that have long constrained progress in the field. By co-integrating textual and musical conditioning within a single video diffusion model architecture, supported by a bespoke web-scale dance dataset (CIPE-Dance), OmniDance advances the state of the art in music- and text-driven video generation. The framework demonstrates robust performance under pure-text, pure-music, and combined text-music driving modalities, while maintaining strong controllability and visual fidelity.

CIPE-Dance: Data Acquisition & Annotation

The lack of comprehensive, high-quality datasets tailored to the complexity of dance video synthesis has been a persistent bottleneck. The CIPE-Dance dataset directly addresses this gap through a progressive expert-based curation pipeline, leveraging lightweight and heavyweight LLM-based filters for multi-stage video quality assessment, performer isolation, and scene stability assurance. The resulting 300k-clip corpus covers more than 30 dance genres and a vast array of environments and performer demographics.

Figure 1: Overview of the Progressive Expert-Based Data Collection Pipeline.

Annotations are automatically generated from five choreography-informed perspectives (Body Dynamics, Choreographic Content, Expressiveness, Camera Presentation, Overall Look) using multi-aspect LLMs, yielding fine-grained, semantically rich textual supervision.

Figure 2: Dataset analysis: annotation display, genre coverage, and semantic caption statistics.

OmniDance Framework: Multimodal Integration

The core methodological contribution is a three-level co-design that injects music conditioning into a pre-trained text/image-to-video (TI2V) DiT backbone (WAN2.2-TI2V-5B), maintaining fidelity and segmentation control while unlocking music-driven choreography. OmniDance features:

• Depth-aware specialization: A progressive audio residual scaling scheme, where shallow layers favor text semantics and deeper layers emphasize music features for fine-grained motion detail.
• Curriculum learning: A staged "easy-to-hard" training regime that first aligns dance-specific semantics via text, then incrementally introduces joint text-music conditioning (MTI2V), followed by isolated music conditioning for genuine MI2V specialization.
• Modality-specialized CFG: Classifier-free guidance scheduling that emphasizes text during low-frequency structure imposition (early diffusion steps), shifting toward music for high-frequency, rhythmic motion refinement in later steps.

  Figure 3: Overview of OmniDance multimodal specialization and multi-stage curriculum strategy.

Experimental Results

Extensive experiments on CIPE-Dance benchmark OmniDance against recent SOTA TI2V and MI2V baselines. OmniDance achieves the highest or near-highest scores across video quality (IQ, AQ, SC, BC, MS, TF), motion fidelity/diversity (FID$_{k/g}$, DIV$_{k/g}$), and modality-alignment (BAS, OC) metrics. Notably, for MTI2V generation, the model demonstrates superior multimodal synergy, with state-of-the-art results on ID consistency, background stability, and rhythmic beat alignment, without degradation in modality-specific performance.

Figure 4: Comparison with SOTAs on Text-Image-to-Video (TI2V) and Music-Image-to-Video (MI2V) tasks on the CIPE-Dance dataset.

Qualitative Assessment

Generated videos via OmniDance display consistent subject identity, high-resolution visual detail, and expressive, music-synchronous choreography—capabilities that have eluded prior methods, which suffer from trade-offs between motion complexity and appearance fidelity. The music-text specialization mechanism is critical to these results; ablation shows rhythmicity and semantic alignment are severely compromised if this architectural strategy is removed or inverted.

Figure 5: Example of OmniDance multimodal-driven dance video generation for various conditioning modalities.

Implications and Future Directions

OmniDance establishes a methodological blueprint for integrating temporally structured audio signals into high-capacity generative models, offering a unified solution that avoids modality-specific silos. Practically, it enables controllable dance synthesis driven by textual intent, musical features, or both, suitable for digital content creation, virtual avatar animation, and AI-art applications.

Theoretically, the findings underscore the benefits of frequency-aligned specialization in multi-branch architectures for temporal generation tasks and highlight the effectiveness of curriculum-based multimodal training. Moreover, the scalable, web-crawled data acquisition pipeline and LLM-based annotation unlock widespread accessibility to complex motion domains previously stymied by scarce supervision.

Looking forward, two primary research directions arise:

1. Real-time Generation: Although current inference is not real-time, model distillation and efficient flow-matching designs (e.g., single-step generation) could bridge this gap, opening interaction-rich applications.
2. Further Multimodality: Expansion to additional modalities (e.g., video–pose–music joint generation), finer-grained control, and co-creation with humans in the loop may be facilitated by the infrastructure established here.

Conclusion

OmniDance presents the first large-scale, multimodal dance video generation framework with web-scale data support and unified model design. By decoupling text and music control via depth-aware specialization, staged curriculum training, and modality-specific inference, it achieves state-of-the-art performance and robust controllability for TI2V, MI2V, and MTI2V tasks, setting a new standard for expressive, high-fidelity AI dance generation (2606.30019).