SoulDance: Holistic 3D Dance Generation

• SoulDance is a comprehensive 12.5-hour music-paired 3D dance dataset capturing synchronized body, hand, and facial motions through professional motion capture techniques.
• It fills gaps in previous datasets by integrating a unified SMPL-X and FLAME representation with precise temporal synchronization to music.
• SoulNet employs a discrete, hierarchical motion modeling approach using residual vector quantization and a Music-Motion Retrieval Module to achieve detailed, semantically coherent dance generation.

SoulDance denotes both a professionally captured music–dance dataset and the generative framework built around it, as introduced in "Music-Aligned Holistic 3D Dance Generation via Hierarchical Motion Modeling" (Li et al., 20 Jul 2025). In that formulation, SoulDance is a 12.5-hour, music-paired 3D dance corpus with synchronized body, hand, and facial motion, while SoulNet is the corresponding music-conditioned model for generating holistic 3D dance. The central premise is that music-aligned dance generation requires simultaneous modeling of full-body locomotion, detailed finger articulation, and facial expression, together with a cross-modal alignment mechanism that enforces temporal synchronization and semantic coherence between motion and music (Li et al., 20 Jul 2025).

1. Definition and scope

In the source paper, SoulDance was created to address three gaps. Existing music–dance datasets either only have body motion, add hands but omit face, or are reconstructed from video with noisy poses. As a result, prior models cannot properly learn coordinated body–hand–face movement, and prior work is described as having weak, implicit music–dance alignment, which leads to dances that match rhythm and coarse style but not detailed temporal and emotional structure (Li et al., 20 Jul 2025).

The term "holistic 3D dance" has a specific meaning in this setting. It refers to simultaneous, coordinated modeling of full-body locomotion, posture, and style; detailed hand gestures and finger articulation; and dynamic facial expressions that reflect the music’s emotional tone. SoulDance supports this by combining synchronized mocap, a unified SMPL-X-plus-FLAME representation, and a model architecture whose quantization and generative components explicitly treat body, hands, and face jointly (Li et al., 20 Jul 2025).

This emphasis distinguishes SoulDance from systems that focus on body-only dance synthesis or on image-conditioned video generation. A plausible implication is that SoulDance occupies the intersection of motion generation, affective computing, and cross-modal retrieval, rather than being only a choreography generator.

2. Dataset design and capture pipeline

SoulDance is a 12.5-hour, professionally captured, music-paired 3D dance dataset providing synchronized body, hands, and face motion for the first time in a single holistic setting (Li et al., 20 Jul 2025). The paper reports five professional dancers, solo and group sequences with 1–5 dancers, 284 music segments across 15 genres, and an average sequence length of approximately 158.5 seconds. For experiments, 6.3 hours of SoulDance are selected for training and testing, and the split used for SoulDance, AIST++, and FineDance is 8:1:1 for train, validation, and test (Li et al., 20 Jul 2025).

The capture setup combines a professional marker-based optical mocap system from ChingMu for body and hands with an iPhone 12 mounted on a helmet for facial capture. The body-and-hand system uses 15 cameras in a 140 m² studio at 60 FPS, producing BVH files that are later retargeted to SMPL-X. Facial capture uses Apple ARKit to record 52 blendshape weights per frame. Body and hand data are converted to SMPL-X with Autodesk MotionBuilder and Unreal Engine 5 through T-pose normalization, bone-length calibration, and joint-name mapping, followed by manual refinement to correct non-physical poses. Facial parameters are mapped from ARKit to FLAME by learning a transformation matrix $\mathbf{W}$ from 52 ARKit blendshapes to 103 FLAME parameters, comprising 100 expression parameters and 3 jaw parameters (Li et al., 20 Jul 2025).

The choreography protocol is equally specific. A professional choreographer designed dances for specific music segments, the dancers rehearsed them in a studio, and a director monitored execution and music alignment during recording. The dataset therefore emphasizes choreographed, music-specific performances rather than free improvisation. Temporal synchronization is established by capture design: dancers perform while listening to playback, motion capture and audio are recorded concurrently, and an engineer corrects recording misalignment on the spot (Li et al., 20 Jul 2025).

This capture regime contrasts with LM2D’s dataset construction, which extracts 3D motion from Ubisoft Just Dance videos using EasyMocap and manual lyric synchronization, yielding 4.6 hours and 1,867 motion–music–lyrics triplets (Yin et al., 2024). It also differs from ChoreoMuse’s operating assumptions, which are oriented toward arbitrary-person video synthesis from a single reference image rather than high-precision holistic mocap (Wang et al., 26 Jul 2025).

3. Motion representation and music–motion alignment

SoulDance uses a unified holistic motion representation derived from SMPL-X and FLAME. Each frame $m_i$ is represented as

$$\mathbf{m}_i = \{\dot{r}^a, \dot{r}^x, \dot{r}^z, r^y, \mathbf{j}^p, \mathbf{j}^r, \mathbf{j}^v, \dot{c}, \mathbf{f}\},$$

where $\dot{r}^a$ is root angular velocity around $Y$, $\dot{r}^x,\dot{r}^z$ are ground-plane root linear velocities, $r^y$ is root height, $\mathbf{j}^p \in \mathbb{R}^{3N-3}$ are local joint positions excluding the root, $\mathbf{j}^r \in \mathbb{R}^{6N-6}$ are 6-DoF joint rotations, $\mathbf{j}^v \in \mathbb{R}^{3N}$ are joint velocities, $m_i$0 are binary-like foot-contact indicators, and $m_i$1 is the FLAME expression vector. With $m_i$2 joints, the total feature dimension is 723 per frame (Li et al., 20 Jul 2025).

The skeletal component contains 52 body-and-hand joints, explicitly decomposed as 22 SMPL-X body joints and 30 SMPL-X hand joints. Facial motion is represented by 100-dimensional FLAME expression parameters, plus jaw information obtained from ARKit. Raw formats include BVH skeleton files, SMPL-X parameters, and FLAME face parameters (Li et al., 20 Jul 2025).

Music is handled differently from motion. Raw audio is first encoded with Jukebox’s pretrained music encoder and then passed through a trainable encoder $m_i$3 to project it into a latent space aligned with motion (Li et al., 20 Jul 2025). Alignment is not based on hand-labeled beats or phrases. Instead, it is learned through a Music-Motion Retrieval Module, which places synchronized music and motion pairs close in a shared latent space. The explicit alignment representation is therefore a shared time index in the dataset and latent embeddings produced by the retrieval model for evaluation and supervision (Li et al., 20 Jul 2025).

This alignment philosophy is related to, but more explicit than, prior approaches. LM2D conditions motion on music and lyrics via cross-attention, but its music features are Librosa-based MFCC, chroma, and beat indicators, with lyrics embedded by BERT (Yin et al., 2024). ChoreoMuse learns a dance-specific audio embedding through MotionTune and fuses it with Jukebox features, emphasizing beat adherence and style control (Wang et al., 26 Jul 2025). SoulDance instead centers retrieval-based cross-modal alignment as an explicit prior for both evaluation and generation (Li et al., 20 Jul 2025).

4. SoulNet and hierarchical motion modeling

SoulNet consists of three principal components: Hierarchical Residual Vector Quantization, a Music-Aligned Generative Model, and the Music-Motion Retrieval Module used as an alignment prior (Li et al., 20 Jul 2025). The architecture is discrete-latent and transformer-based, rather than diffusion-based.

Hierarchical Residual Vector Quantization extends residual vector quantization with a body $m_i$4 hands $m_i$5 face dependency chain at every residual layer. The holistic motion sequence is decomposed into $m_i$6, $m_i$7, and $m_i$8, with dedicated encoders $m_i$9. Quantization at residual layer $$\mathbf{m}_i = \{\dot{r}^a, \dot{r}^x, \dot{r}^z, r^y, \mathbf{j}^p, \mathbf{j}^r, \mathbf{j}^v, \dot{c}, \mathbf{f}\},$$0 is defined by

$$\mathbf{m}_i = \{\dot{r}^a, \dot{r}^x, \dot{r}^z, r^y, \mathbf{j}^p, \mathbf{j}^r, \mathbf{j}^v, \dot{c}, \mathbf{f}\},$$1

with residual updates

$$\mathbf{m}_i = \{\dot{r}^a, \dot{r}^x, \dot{r}^z, r^y, \mathbf{j}^p, \mathbf{j}^r, \mathbf{j}^v, \dot{c}, \mathbf{f}\},$$2

where $$\mathbf{m}_i = \{\dot{r}^a, \dot{r}^x, \dot{r}^z, r^y, \mathbf{j}^p, \mathbf{j}^r, \mathbf{j}^v, \dot{c}, \mathbf{f}\},$$3 is a small neural transformation composed of an MLP and 1D convolution (Li et al., 20 Jul 2025). Reconstruction sums quantized contributions across layers and decodes them jointly. The training loss combines an $$\mathbf{m}_i = \{\dot{r}^a, \dot{r}^x, \dot{r}^z, r^y, \mathbf{j}^p, \mathbf{j}^r, \mathbf{j}^v, \dot{c}, \mathbf{f}\},$$4 reconstruction term with per-part embedding losses: $$\mathbf{m}_i = \{\dot{r}^a, \dot{r}^x, \dot{r}^z, r^y, \mathbf{j}^p, \mathbf{j}^r, \mathbf{j}^v, \dot{c}, \mathbf{f}\},$$5 The paper argues that lower residual layers capture coarse, low-frequency motion, while higher layers refine small details such as finger subtleties and micro-expression dynamics (Li et al., 20 Jul 2025).

The Music-Aligned Generative Model operates on discrete HRVQ tokens. For each layer $$\mathbf{m}_i = \{\dot{r}^a, \dot{r}^x, \dot{r}^z, r^y, \mathbf{j}^p, \mathbf{j}^r, \mathbf{j}^v, \dot{c}, \mathbf{f}\},$$6, tokens are concatenated as

$$\mathbf{m}_i = \{\dot{r}^a, \dot{r}^x, \dot{r}^z, r^y, \mathbf{j}^p, \mathbf{j}^r, \mathbf{j}^v, \dot{c}, \mathbf{f}\},$$7

The base layer $$\mathbf{m}_i = \{\dot{r}^a, \dot{r}^x, \dot{r}^z, r^y, \mathbf{j}^p, \mathbf{j}^r, \mathbf{j}^v, \dot{c}, \mathbf{f}\},$$8 is trained with MaskGIT-like masked modeling: $$\mathbf{m}_i = \{\dot{r}^a, \dot{r}^x, \dot{r}^z, r^y, \mathbf{j}^p, \mathbf{j}^r, \mathbf{j}^v, \dot{c}, \mathbf{f}\},$$9 while residual layers use

$\dot{r}^a$0

with $\dot{r}^a$1 (Li et al., 20 Jul 2025). At inference, music is encoded, base tokens are generated, residual tokens refine them, and the HRVQ decoder reconstructs continuous motion.

This design differs structurally from diffusion-based dance models. ChoreoMuse uses a two-stage diffusion pipeline from music to SMPL motion and then from SMPL to video (Wang et al., 26 Jul 2025), whereas LM2D uses continuous-time diffusion and consistency distillation for one-step music-and-lyrics-conditioned motion synthesis (Yin et al., 2024). A plausible implication is that SoulNet prioritizes discrete compositional structure and retrieval-guided alignment over iterative denoising.

5. Music-Motion Retrieval Module and alignment prior

The Music-Motion Retrieval Module is a pretrained cross-modal model that learns a shared latent space for music and dance motion and serves both as a retrieval mechanism and as an alignment prior for SoulNet (Li et al., 20 Jul 2025). Its motion encoder $\dot{r}^a$2 is a temporal encoder derived from TEMOS, producing latent vectors $\dot{r}^a$3, while the music pipeline applies Jukebox followed by $\dot{r}^a$4 to produce $\dot{r}^a$5, with $\dot{r}^a$6. A transformer-based motion decoder $\dot{r}^a$7 is used during MMR training but not during SoulNet inference (Li et al., 20 Jul 2025).

The module uses reconstruction together with symmetric InfoNCE alignment losses: $\dot{r}^a$8 and

$\dot{r}^a$9

combined as

$Y$0

where cosine similarity is used and $Y$1 is the temperature (Li et al., 20 Jul 2025).

Two retrieval models are used: $Y$2, trained on a 263-dimensional HumanML3D-style body representation, and $Y$3, trained on the 723-dimensional holistic representation. During SoulNet training, predicted tokens are decoded to continuous motion, motion and music are re-encoded by MMR, and alignment losses are computed. Gradient flow through discrete tokenization is enabled by Gumbel-Softmax relaxation (Li et al., 20 Jul 2025).

The evaluation metric derived from this module is the proposed MMR-Matching Score: $Y$4 where the first term measures static embedding distance and the second measures dynamic trajectory difference over 1-second segments, with $Y$5 and $Y$6 typically used; lower is better (Li et al., 20 Jul 2025). The same section introduces Emotion Alignment Score, which evaluates how often generated facial emotion matches ground-truth emotion labels for the same music using FER models over 7 Ekman categories (Li et al., 20 Jul 2025).

This retrieval-centered alignment is more explicit than the BAS-based beat evaluation emphasized in ChoreoMuse (Wang et al., 26 Jul 2025) and more facially grounded than LM2D’s semantic matching score, which aligns lyric embeddings with motion embeddings through MotionBERT and BERT (Yin et al., 2024).

6. Evaluation, performance, and reproducibility

The reported baselines are FACT, Bailando, EDGE, and FineNet. On the SoulDance dataset, SoulNet with MMR attains FID $Y$7, Div $Y$8, Hand FID $Y$9, Hand Div $\dot{r}^x,\dot{r}^z$0, MM $\dot{r}^x,\dot{r}^z$1, MMR-MS $\dot{r}^x,\dot{r}^z$2, BAS $\dot{r}^x,\dot{r}^z$3, and EAS $\dot{r}^x,\dot{r}^z$4. The corresponding baseline figures reported in the paper include FID values of $\dot{r}^x,\dot{r}^z$5 for FACT, $\dot{r}^x,\dot{r}^z$6 for Bailando, $\dot{r}^x,\dot{r}^z$7 for EDGE, and $\dot{r}^x,\dot{r}^z$8 for FineNet, with MMR-MS values of $\dot{r}^x,\dot{r}^z$9 for FACT and $r^y$0 for FineNet (Li et al., 20 Jul 2025). On AIST++, SoulNet attains FID $r^y$1, while the paper notes that diversity is slightly lower than some baselines on that smaller dataset (Li et al., 20 Jul 2025).

Ablation studies attribute much of the gain to HRVQ and MMR. For reconstruction, HRVQ-5 yields MPJPE $r^y$2 mm on “all,” compared with RVQ-5 at $r^y$3 and VQ-1024 at $r^y$4, and also obtains the best Face Vertex Error. For generation, HRVQ-512 without MMR gives FID $r^y$5 and MMR-MS $r^y$6, while HRVQ-512 with MMR improves these to $r^y$7 and $r^y$8. The paper further reports that removing $r^y$9 affects FID and BAS more strongly, whereas removing $\mathbf{j}^p \in \mathbb{R}^{3N-3}$0 affects MMR-MS more (Li et al., 20 Jul 2025).

User studies include 28 participants, including 7 professional dancers. In a dataset comparison, SoulDance receives the highest average ratings across Whole sequence, Body, Hands, Expression, and Alignment; approximate values reported include Whole sequence $\mathbf{j}^p \in \mathbb{R}^{3N-3}$1 for SoulDance versus $\mathbf{j}^p \in \mathbb{R}^{3N-3}$2 for AIST++ and $\mathbf{j}^p \in \mathbb{R}^{3N-3}$3 for FineDance. In a method comparison on SoulDance, SoulNet outperforms FACT, Bailando, EDGE, and FineNet on Whole sequence, Body, Hands, Expression, and Alignment, with approximate scores including Whole sequence $\mathbf{j}^p \in \mathbb{R}^{3N-3}$4, Body $\mathbf{j}^p \in \mathbb{R}^{3N-3}$5, Hands $\mathbf{j}^p \in \mathbb{R}^{3N-3}$6, Expression $\mathbf{j}^p \in \mathbb{R}^{3N-3}$7, and Alignment $\mathbf{j}^p \in \mathbb{R}^{3N-3}$8 (Li et al., 20 Jul 2025).

Reproducibility details are unusually explicit. HRVQ uses a temporal downscale factor of 4, $\mathbf{j}^p \in \mathbb{R}^{3N-3}$9 residual layers, vector quantizers with 6 layers, codebook size 512, code dimension 512, and quantization dropout $\mathbf{j}^r \in \mathbb{R}^{6N-6}$0. The generator uses 6 transformer layers plus 6 residual transformer layers, 8 attention heads, and latent dimension 512. MMR uses embedding dimension $\mathbf{j}^r \in \mathbb{R}^{6N-6}$1, AdamW with learning rate $\mathbf{j}^r \in \mathbb{R}^{6N-6}$2, batch size 128 for MMR, 64 for MAGM, and 256 for HRVQ, with InfoNCE temperature $\mathbf{j}^r \in \mathbb{R}^{6N-6}$3 and contrastive weight $\mathbf{j}^r \in \mathbb{R}^{6N-6}$4. Learning rate warm-up is to $\mathbf{j}^r \in \mathbb{R}^{6N-6}$5 over 2000 steps for HRVQ and MAGM, training uses 4 NVIDIA V100 GPUs, and the full pipeline completes in roughly 3 days. Runtime on AIST++ is reported as 0.086 s per sequence for a fixed clip length (Li et al., 20 Jul 2025).

7. Position in the literature, limitations, and applications

Within the literature summarized by the paper, prior music-conditioned dance generation includes FACT as an autoregressive method, Bailando as VQ-VAE plus GPT with reinforcement learning for rhythm alignment, EDGE as diffusion-based, and FineNet as a body-plus-hand method. AIST++ is described as a body-only dataset reconstructed from multi-view videos, while FineDance provides body and hands but not face (Li et al., 20 Jul 2025). SoulDance and SoulNet are presented as filling three gaps simultaneously: the dataset gap through a mocap-based holistic corpus; the representation gap through HRVQ’s hierarchical body–hand–face dependency modeling; and the alignment gap through MMR’s CLIP-like cross-modal retrieval and alignment supervision (Li et al., 20 Jul 2025).

The limitations reported are concrete. SoulDance does not include clothing dynamics, props, or environment interactions. Although 12.5 hours is substantial, the dataset remains limited relative to large video corpora and includes only five dancers, which may induce bias in style and body shape. Domain generalization is also bounded: while SoulNet is tested on AIST++, it is primarily tuned to SoulDance-like data, and cross-dataset results show trade-offs such as lower diversity on small AIST++ (Li et al., 20 Jul 2025).

The stated applications include virtual avatars and digital performances, games and VR/AR, content creation tools for choreographers and creators, research in affective computing, and interactive systems such as music-driven avatars in online platforms or live performances (Li et al., 20 Jul 2025). This suggests a broader interpretation of SoulDance not merely as a benchmark but as an infrastructure for holistic generative motion research.

Relative to adjacent work, SoulDance is complementary rather than redundant. ChoreoMuse addresses music-to-dance video generation with style control, arbitrary identity, and arbitrary resolution through a two-stage SMPL-based diffusion pipeline (Wang et al., 26 Jul 2025). LM2D addresses 3D dance synthesis conditioned jointly on music and lyrics, with consistency distillation for one-step generation (Yin et al., 2024). SoulDance differs in prioritizing high-precision holistic mocap and retrieval-guided body–hand–face coordination. A plausible implication is that these three directions—holistic motion capture, multimodal semantic conditioning, and motion-to-video rendering—form partially interoperable components of a broader music-conditioned dance generation stack.