CIPE-Dance: Large-Scale Dance Video Dataset

Updated 6 July 2026

CIPE-Dance is a large-scale, Internet-sourced dance video dataset with choreography-informed text annotations, enabling multimodal generation tasks.
The dataset uses a progressive expert pipeline to filter 300k high-quality 5-second clips, ensuring single-dancer clarity, scene stability, and synchronized audio.
CIPE-Dance serves as the benchmark for OmniDance, supporting TI2V, MI2V, and MTI2V training with detailed evaluations and metrics.

Searching arXiv for the cited works to ground the article in current papers. Search query: "OmniDance Multimodal Driven Dance Video Generation with Large-scale Internet Data arXiv" CIPE-Dance is a large-scale Internet-sourced dance video dataset introduced together with the OmniDance framework for multimodal dance video generation. In that setting, the term denotes both a concrete data resource and a data-construction paradigm: a corpus of approximately 300k high-quality dance clips totaling more than 400 hours, paired with synchronized music and choreography-informed text annotations, and assembled through a progressive expert pipeline designed to filter for quality, single-dancer clarity, and choreography relevance (Yang et al., 29 Jun 2026). CIPE-Dance is explicitly curated for three conditional generation regimes—text–image-to-video (TI2V), music–image-to-video (MI2V), and music–text–image-to-video (MTI2V)—and serves as the principal training and evaluation substrate for OmniDance (Yang et al., 29 Jun 2026).

1. Definition, scope, and nomenclature

CIPE-Dance is defined as a large-scale Internet-sourced dance video dataset with Choreography-Informed text annotations, constructed via a Progressive Expert-based pipeline. In the OmniDance formulation, the acronym CIPE expands to Choreography-Informed, Progressive Experts rather than to a motion encoder or a graph positional encoding scheme (Yang et al., 29 Jun 2026).

Its stated purpose is to address three deficits in prior dance-generation resources: lack of scale, lack of in-the-wild diversity, and lack of multimodal, choreography-aware annotation. The dataset is therefore organized around clips that contain a clear single dancer, stable camera conditions, a usable reference frame for identity control, and synchronized audio suitable for music-conditioned generation (Yang et al., 29 Jun 2026).

A common source of ambiguity is the acronym itself. In a separate graph-Transformer literature, CIPE refers to Communicability-Inspired Positional Encoding, a graph positional encoding derived from communicability and heat-kernel geometry (Zhang et al., 24 Jun 2026). That work is unrelated in domain and objective to CIPE-Dance. In the dance-video context, CIPE-Dance names a dataset and collection pipeline rather than a graph positional encoding method.

2. Progressive expert pipeline and data curation

CIPE-Dance is built through a six-stage Progressive Expert-Based Data Collection Pipeline applied to web-scraped short-form video from Douyin and TikTok. The guiding principle is that simpler checks are assigned to lightweight experts and more difficult checks to heavier experts. Source videos are taken from about 500 dance content creators with >50K followers; only videos after 2018 are retained; and videos are segmented into 5-second clips at 16 FPS, yielding an initial pool of 630k clips (Yang et al., 29 Jun 2026).

The filtering stages are summarized below.

Stage	Tool	Retention
Visual Quality Verification	VBench	96.76%
Reference Clarity Verification	Qwen3-VL-2B	81.45%
Dance Video Verification	Qwen3-VL-2B	89.24%
Single-Dancer Filtering	Qwen3-VL-2B	74.45%
Scene Stability Filtering	two Qwen3-VL-8B models	91.65%

The Visual Quality Verification stage removes low-quality, artifact-heavy, or visually cluttered clips using VBench thresholds of Image quality (IQ) > 60.00 and Aesthetic quality (AQ) > 50.00. The Reference Clarity Verification stage uses Qwen3-VL-2B to reject clips whose first frame has low visibility, heavy occlusions, back-facing poses, or missing clear human facial cues. The Dance Video Verification stage again uses Qwen3-VL-2B to exclude talking, posing, and static-motion clips (Yang et al., 29 Jun 2026).

The Single-Dancer Filtering stage is notable because it inverts the detection problem: instead of directly proving singularity, it removes three negative cases—group dance, follow-along imitation videos with a small overlaid reference, and mirror-reflection artifacts. These three filters retain 80.35%, 97.02%, and 95.57% of inputs respectively, giving 74.45% overall retention for the stage. The final Scene Stability Filtering stage uses two Qwen3-VL-8B models to remove abrupt scene or subject transitions and aggressive camera motion, with 93.95% and 97.55% retention for the two subfilters and 91.65% overall retention (Yang et al., 29 Jun 2026).

After all stages, the pipeline yields approximately 300k high-quality 5s clips. This suggests that CIPE-Dance is designed not merely as a passive scraped corpus, but as a benchmark-quality dataset in which identity stability, scene continuity, and dance salience are treated as first-class curation constraints.

3. Dataset statistics, modalities, and choreography-informed annotations

The resulting dataset comprises approximately 300,000 clips, >400 hours of video, and clips of 5 seconds at 16 FPS, giving 77 frames per clip. It spans over 30 dance genres and exhibits a long-tailed genre distribution. The dataset also includes substantial diversity in environment, performer appearance, motion complexity, and camera setup, while retaining scene stability through filtering (Yang et al., 29 Jun 2026).

CIPE-Dance is multimodal. Each item contains a full-body dance video, the original audio track, and five separate textual descriptions generated by Qwen3-VL-8B, each aligned to a distinct choreographic aspect (Yang et al., 29 Jun 2026). These five aspects are:

Body Dynamics
Choreographic Content
Expressiveness
Camera Presentation
Overall Look

The annotation scheme is explicitly choreography-aware. Body Dynamics describes moving body parts and how they move; Choreographic Content captures genre, style, and named movement techniques; Expressiveness characterizes emotion and performance intensity; Camera Presentation records framing, angle, and camera movement; and Overall Look summarizes clothing, styling, and environment (Yang et al., 29 Jun 2026). Because the annotations are clip-level and multimodal, they support text-only generation, music-only generation, and combined text-plus-music control.

For evaluation, the authors randomly sample 100 clips as a held-out test split, excluding them from training. No detailed separate validation split is specified in the provided description (Yang et al., 29 Jun 2026).

4. Role in OmniDance and multimodal training

CIPE-Dance functions as the primary training and evaluation dataset for OmniDance, a framework for integrating music into a text-image-to-video foundation model while preserving controllability and visual fidelity (Yang et al., 29 Jun 2026). OmniDance treats text as carrying low-frequency semantics and music as carrying high-frequency temporal dynamics, and this division of labor is directly motivated by the annotation and synchronization structure of CIPE-Dance.

The training objective is a Flow Matching loss over latent video trajectories. For a CIPE-Dance clip encoded into latent $\mathbf{x}_0$ , Gaussian noise $\epsilon \sim \mathcal{N}(0,I)$ , and timestep $t \sim \mathcal{U}(0,1)$ , the model uses

$\mathbf{x}_t = (1 - t)\cdot \mathbf{x}_0 + t\cdot \epsilon, \qquad \mathbf{v}_t = \epsilon - \mathbf{x}_0,$

with objective

$\mathcal{L}_{\text{fm}} = \mathbb{E}_{\mathbf{x}_0,\epsilon,t} \left[ \left\| \mathbf{v}_\theta - \mathbf{v}_t \right\|_2^2 \right].$

This loss is applied under different conditioning configurations corresponding to TI2V, MI2V, and MTI2V (Yang et al., 29 Jun 2026).

OmniDance uses a three-stage curriculum built entirely on CIPE-Dance. Stage I is a Text-Adaptation Warm-Up using TI2V only, with the music branch frozen. Stage II is Anchored Music Integration, training on both TI2V and MTI2V while enabling the music branch with zero-initialized residual projection and a warmed-up learning rate. Stage III is Modality Decoupling and Specialization, training on TI2V, MI2V, and MTI2V jointly (Yang et al., 29 Jun 2026). A plausible implication is that the dataset’s paired text–music–video structure is not ancillary metadata but the enabling condition for this staged multimodal optimization.

Two architectural mechanisms are explicitly tied to the CIPE-Dance design. First, Music–Text Progressive Specialization (MTPS) scales the audio residual depth-wise:

$\mathbf{x}_\ell \leftarrow \mathbf{x}_\ell + \left( \frac{\ell}{L} \right)^{\gamma} \cdot \mathbf{r}_{\text{audio}}^\ell,$

with $\gamma=1.5$ and $L=30$ . Second, OmniDance uses a modality-specialized time-dependent CFG strategy, with text guidance decaying from 5→1 over denoising time and music guidance decaying more slowly from 4→2, increasing the relative influence of music in later stages of generation (Yang et al., 29 Jun 2026).

5. Benchmarking and evaluation on CIPE-Dance

CIPE-Dance is also used as a benchmark for dance video generation. All reported metrics in OmniDance’s main comparison table are computed on the 100-clip CIPE-Dance test set (Yang et al., 29 Jun 2026). The evaluation spans video quality, motion quality, and alignment.

For video quality, the benchmark uses VBench-style metrics: IQ (Image Quality), AQ (Aesthetic Quality), SC (Subject Consistency), BC (Background Consistency), MS (Motion Smoothness), and TF (Temporal Flickering) (Yang et al., 29 Jun 2026). For motion quality, the pipeline extracts 2D keypoints using ViTPose, computes kinetic and geometric features following AI Choreographer-style descriptors, and reports FID_k, FID_g, DIV_k, and DIV_g. The Fréchet distance is given by

$\text{FID}(P_r, P_g) = \|\mu_r - \mu_g\|_2^2 + \mathrm{Tr}\left(\Sigma_r + \Sigma_g - 2(\Sigma_r \Sigma_g)^{1/2}\right),$

where $(\mu_r,\Sigma_r)$ and $\epsilon \sim \mathcal{N}(0,I)$ 0 are statistics of real and generated feature distributions (Yang et al., 29 Jun 2026).

For alignment, the benchmark uses BAS (Beat Alignment Score) for music–motion synchronization and OC (Overall Consistency) as a text–video consistency score. In MI2V, a generic prompt—“a person is dancing”—is used for OC computation (Yang et al., 29 Jun 2026). The qualitative analysis attributes OmniDance’s gains on this benchmark to more diverse hand articulations, natural smiles and facial expressions, stable identity, and rhythm-aware choreography. The paper also reports that MTI2V outputs achieve BAS ≈ 0.287 compared with 0.288 for real CIPE-Dance videos, indicating near-ground-truth beat alignment on that metric (Yang et al., 29 Jun 2026).

This benchmark design differs from earlier dance-generation evaluations centered on pose reconstruction, beat hit rate, or style-embedding FID. For example, prior 2D music-to-dance work emphasized style embeddings, beat hit rate, and FID in style space (Zhang et al., 2021), while later 3D dual-learning work used Fréchet Dist, Diversity, and Beats Align in SMPL motion space (Wu et al., 2022). CIPE-Dance extends evaluation into full video generation, where motion fidelity must coexist with appearance consistency, background stability, and multimodal controllability.

6. Research context, misconceptions, and limitations

CIPE-Dance emerges in a lineage of dance-generation research that previously relied on substantially smaller or more specialized resources. Earlier work on style-aware music-to-dance synthesis built a dataset of anime, popping, and locking videos with approximately 1.48M frames after filtering and used 2D OpenPose keypoints with learned style embeddings (Zhang et al., 2021). Dual music–dance modeling later advanced to 3D SMPL sequences using Tang et al. and AIST++-derived data, but training still occurred on short fixed windows and without Internet-scale choreography-aware video captions (Wu et al., 2022). CIPE-Dance differs by targeting full-body dance video generation in the wild, with synchronized music and rich clip-level text annotations (Yang et al., 29 Jun 2026).

One misconception is to treat CIPE-Dance as a pose dataset analogous to AIST++ or FineDance. The provided description does not present stored 2D or 3D pose annotations as part of the dataset itself; instead, ViTPose keypoints are extracted on the fly for evaluation (Yang et al., 29 Jun 2026). Another misconception is to interpret the dataset name as implying a direct use of the graph CIPE positional encoding framework. The available description does not state such a connection; the shared acronym reflects different expansions and different research problems (Zhang et al., 24 Jun 2026).

The reported limitations are largely those of the source domain and collection strategy. Because CIPE-Dance is assembled from Douyin/TikTok dance creators with >50K followers, it is likely biased toward East Asian and global pop dance trends and toward styles common on short-video platforms. Recording conditions are also skewed toward vertical, phone-camera, influencer-style production. The dataset is clip-oriented at 5 seconds, so longer choreographic structure is not explicitly annotated. The paper also does not provide a detailed legal or ethical discussion beyond noting that the source videos are publicly available (Yang et al., 29 Jun 2026).

These constraints do not diminish the dataset’s importance; rather, they define its research position. CIPE-Dance supplies a large-scale, choreography-aware, multimodal benchmark for TI2V, MI2V, and MTI2V dance video generation, while leaving open problems in cultural coverage, longer-horizon choreography, richer structured labels, and explicit bias analysis (Yang et al., 29 Jun 2026).