EgoAIST++: Egocentric Dance Motion Synthesis

• EgoAIST++ is a multimodal dataset comprising 36 hours of dance motion data, enabling precise 3D pose reconstruction from egocentric video and music audio.
• The EgoMusic Motion Network, featuring the Skeleton Mamba module, fuses visual and musical cues to generate temporally coherent full-body motion sequences.
• Experiments demonstrate significant improvements in MPJPE, head errors, and alignment scores, paving the way for advanced VR/AR dance training and cross-modal applications.

EgoAIST++ is a large-scale multimodal dataset designed for the estimation and synthesis of human dance motion from egocentric video in conjunction with music audio. It is introduced alongside a new generative architecture, the EgoMusic Motion Network (EMM) featuring the Skeleton Mamba module, enabling high-fidelity prediction of full-body 3D pose sequences aligned to both egocentric video and musical guidance. The dataset, protocols, and model architecture collectively target the challenge of reconstructing or generating detailed, temporally coherent human dance motion sequences when direct observation of the full body is unavailable due to the constraints of the egocentric viewpoint and when intricate synchronization with musical structure is required (Nguyen et al., 14 Aug 2025).

1. Dataset Composition and Protocol

EgoAIST++ contains approximately 36 hours of dance motion data, totaling about 3.9 million frames sampled at 30 fps. These are partitioned into roughly 26,000 non-overlapping 5-second subsequences, drawn from the AIST++ motion dataset, which features 10 professional dancers across various dance genres. Each sequence is paired with synchronized:

• Egocentric video frames (RGB, 512×512, 30 fps, rendered with a virtual head-mounted pinhole camera positioned at the SMPL head joint with a 90° FOV in a synthetic indoor environment from Replica).
• Audio, retained from the matched music tracks of original AIST++ performances, processed into high-level Jukebox embeddings (22,000 Hz, 0.033 s per embedding).
• SMPL skeleton parameters (24 joint rotations as 3×3 matrices or 6D vectors, plus global root translation).

During dataset construction, the dancer mesh is randomly located and rotated in one of the virtual Replica scenes per clip, with a penetration constraint to preserve plausible human-object and ground contact, enforced by human annotators. No music track or scene appears in both training and test splits to ensure generalization. There are 980 training music-choreography pairs and 40 in testing, split across 13 and 5 distinct scenes, respectively. Temporal alignment is enforced: for each skeleton frame $i$, the associated video frame and audio embedding coincide and correspond to $[i \cdot 0.033, (i+1) \cdot 0.033)$ seconds of the track. Augmentation includes random spatial jittering (±5 cm), hue/saturation perturbation, but no temporal warping.

2. EgoMusic Motion Network Architecture

The generative model stack accepts egocentric video $\mathbf{v} = \{v^1, \ldots, v^T\}$, music audio $\mathbf{a} = \{a^1, \ldots, a^T\}$, and seeks to predict the target motion sequence $\mathbf{x}_0 = \{x_0^1, \ldots, x_0^T\}$, representing 3D skeleton poses. The pipeline operates as follows:

1. Visual Encoder: Employs a ResNet-50 (conv1-conv4) to extract $z_v \in \mathbb{R}^{T \times D_c}$ (with $D_c = 512$) from video frames.
2. Music Encoder: Utilizes a pre-trained Jukebox model to obtain frame-level (0.033 s) 512-d audio embeddings, further encoded by a 4-layer Transformer (hidden dim 512, 8 heads) to yield $z_a \in \mathbb{R}^{T \times D_c}$.
3. Fusion Module: Concatenates $z_v$ and $z_a$, feeding them into a linear projection and positional encoding via DenseFiLM for spatio-temporal feature fusion, producing $[i \cdot 0.033, (i+1) \cdot 0.033)$0.
4. Conditional Diffusion Denoiser: A U-Net-style 1D network with 6 denoising blocks, each integrating cross-attention to $[i \cdot 0.033, (i+1) \cdot 0.033)$1 and timestep via FiLM gating (256 channels per block), processes the noisy skeleton sequence $[i \cdot 0.033, (i+1) \cdot 0.033)$2 and produces the denoised output.

The diffusion process is governed by forward and backward equations:

• Forward noising: $[i \cdot 0.033, (i+1) \cdot 0.033)$3,
• Backward prediction: $[i \cdot 0.033, (i+1) \cdot 0.033)$4.

3. Skeleton Mamba: Sequence Modeling for Skeleton Structure

The Skeleton Mamba module is embedded within each denoiser block and is specifically tailored to model spatio-temporal dependencies within the human body:

• Human Tokenizer: Groups joints into overlapping body part sets (e.g., limbs, torso), creating $[i \cdot 0.033, (i+1) \cdot 0.033)$5 tokens per segment ($[i \cdot 0.033, (i+1) \cdot 0.033)$6 for $[i \cdot 0.033, (i+1) \cdot 0.033)$7 joints per group).
• Multi-directional SSD (Group Scan): Scans the tokenized data over $[i \cdot 0.033, (i+1) \cdot 0.033)$8 permutations of group ordering using an SSD (State-Space Duality) block, concatenates, applies the SSD, and averages across all permutations to enhance groupwise context.
• Joint Scan: For each group, projects tokens into per-joint embeddings, applies an SSD block, and merges to recover spatially detailed joint trajectories.
• Temporal Scan: Reshapes and scans the skeleton over time with both forward and backward SSDs for enhanced temporal context, outputting the sum of both passes.
• SSD Block Equation: Each block computes $[i \cdot 0.033, (i+1) \cdot 0.033)$9, $\mathbf{v} = \{v^1, \ldots, v^T\}$0, where $\mathbf{v} = \{v^1, \ldots, v^T\}$1, $\mathbf{v} = \{v^1, \ldots, v^T\}$2, $\mathbf{v} = \{v^1, \ldots, v^T\}$3, $\mathbf{v} = \{v^1, \ldots, v^T\}$4 are learnable SSM parameters (with diagonalizable $\mathbf{v} = \{v^1, \ldots, v^T\}$5 for memory efficiency).

These mechanisms enable Skeleton Mamba to capture multi-scale, multi-directional, and temporally bidirectional dependencies among skeleton joints, yielding robust motion representation in the noisy generative process.

4. Training, Loss Formulation, and Hyperparameters

The loss function combines multiple objectives:

• Kinematic Loss: $\mathbf{v} = \{v^1, \ldots, v^T\}$6, balancing absolute joint positions, velocities, and physical contacts.
• Ego-Music Contrasting: $\mathbf{v} = \{v^1, \ldots, v^T\}$7 enforces local correlation between music and vision embeddings.
• Diffusion Simple Loss: $\mathbf{v} = \{v^1, \ldots, v^T\}$8 penalizes reconstruction error in the denoising process.
• Total Loss: $\mathbf{v} = \{v^1, \ldots, v^T\}$9.

Training employs AdamW (learning rate $\mathbf{a} = \{a^1, \ldots, a^T\}$0, $\mathbf{a} = \{a^1, \ldots, a^T\}$1, $\mathbf{a} = \{a^1, \ldots, a^T\}$2), batch size 32, 1000 diffusion steps ($\mathbf{a} = \{a^1, \ldots, a^T\}$3-linear schedule), and empirically selected loss weights ($\mathbf{a} = \{a^1, \ldots, a^T\}$4, $\mathbf{a} = \{a^1, \ldots, a^T\}$5, $\mathbf{a} = \{a^1, \ldots, a^T\}$6, $\mathbf{a} = \{a^1, \ldots, a^T\}$7).

5. Experimental Evaluation and Analysis

EgoAIST++ and the EMM framework are evaluated on both in-domain and cross-domain (EgoExo4D) test sets using:

• O_{head}: Average head rotation error.
• T_{head}: Average head translation error.
• MPJPE: Mean Per Joint Position Error.
• Accel: Acceleration error of joints.
• FS: Foot skating measure (lower is better).
• MMV: Cross-modal motion-music-vision alignment score.

Main Results: Test Set (Best values bolded)

Method	O_head↓	T_head↓	MPJPE↓	Accel↓	FS↓	MMV↑
PoseReg	1.78	423.6	351.4	37.1	98.8	0.182
KinPoly	1.16	392.7	338.7	16.3	25.8	0.197
EgoEgo	0.74	373.7	152.0	14.2	22.1	0.218
FACT	1.54	407.9	173.7	14.6	15.1	0.202
Bailando	1.57	411.4	175.3	14.7	15.5	0.210
EDGE	1.52	404.6	167.4	14.4	14.8	0.224
EMM (ours)	0.53	342.4	137.5	11.8	12.8	0.262

Ablation studies demonstrate that combining egocentric video and music yields optimal results, and employing the Skeleton Mamba (multi-directional + joint + temporal scan) reduces MPJPE by 8% over bidirectional scanning alone. Removing the music-vision alignment loss increases MPJPE by 5 mm.

6. Data Access, Limitations, and Applications

• Access: EgoAIST++ dataset and code repository, including pre-trained models, are available at https://zquang2202.github.io/SkeletonMamba/data and https://zquang2202.github.io/SkeletonMamba/code.
• Limitations: Sequences exceeding 10 seconds remain challenging due to bidirectional but not fully global temporal scanning. Performance degrades when egocentric video and audio inputs are unsynchronized or mismatched.
• Applications: Enabling VR/AR dance training and gaming without external trackers, metaverse avatar control aligning gaze and motion to music, automatic choreography, rehabilitation and dance therapy robotics, and first-person action and gesture recognition for sports or performance capture.

EgoAIST++ establishes a new benchmark for joint vision-music-driven human motion estimation and synthesis, providing a resource and modeling approach expected to facilitate progress in cross-modal embodiment, affective computing, and action-driven virtual agent systems (Nguyen et al., 14 Aug 2025).