EgoMusic Motion Network

• EgoMusic Motion Network is a diffusion-based framework that combines egocentric video and audio to generate full-body dance motions with state-of-the-art accuracy.
• It leverages the Skeleton Mamba, a bidirectional state-space model that explicitly encodes skeletal topology, and fuses features through cross-attention mechanisms.
• Evaluations on the EgoAIST++ dataset demonstrate significant improvements in pose reconstruction, identity preservation, and music-vision alignment compared to prior methods.

The EgoMusic Motion Network (EMM) is a diffusion-based framework for human dance motion estimation conditioned on both egocentric video and music. Leveraging the Skeleton Mamba—a bidirectional state-space sequence model that explicitly encodes the human body's skeletal topology—EMM addresses the challenge of pose reconstruction from severely body-obscuring egocentric observations while ensuring generated motions are musically and visually coherent. Trained on the EgoAIST++ dataset, which provides extensive temporally aligned ego-video, audio, and motion-capture pose annotations, EMM achieves state-of-the-art accuracy in full-body pose generation from egocentric perspectives, outperforming prior single-modality and cross-modal approaches in motion quality, identity preservation, and music-vision alignment (Nguyen et al., 14 Aug 2025).

1. EgoAIST++ Dataset

EgoAIST++ constitutes the foundational resource for method development and evaluation. The dataset comprises 36 hours of dance, encompassing approximately 3.9 million frames at 30 Hz. Each clip provides:

• Forward-facing egocentric RGB video, obtained by virtually mounting a camera onto a head pose simulated in 18 3D environments rendered via Replica [Straub et al. '19] and AI Habitat [Szot et al. '21].
• Synchronized music tracks, encoded with a pre-trained Jukebox model [Dhariwal et al. '20].
• Frame-wise ground-truth SMPL pose tensors $x \in \mathbb{R}^{T \times J \times D}$, where $J=24$ joints and $D=6\!-\!9$ (dependencies: quaternion, translation, velocity; format aligned with AIST++ motion capture).

The annotation pipeline segments each performance into 5-second clips, randomizes camera positions within scenes, and partitions 980 music-scene pairs for training (13 scenes) and 40 for testing (5 scenes), guaranteeing no overlap and robust generalization.

2. Architectural Composition

EMM integrates conditional diffusion with explicit skeletal and multimodal alignment mechanisms. Its workflow consists of:

• Video Encoder: A ResNet-50 derives feature embedding $f_v \in \mathbb{R}^{T \times D_c}$ from downsampled RGB sequences.
• Music Encoder: The Jukebox encoder (audio-to-embedding) and transformer subnetwork output $f_a \in \mathbb{R}^{T \times D_c}$ from mel-spectrogram audio.
• Fusion: Features $f_v$ and $f_a$ are merged via DenseFiLM [Pérez et al. '18] and cross-attention into conditional embedding $z$ to inject context into the denoising process.
• Diffusion Denoiser: A U-Net-style model, whose backbone is the Skeleton Mamba core, receives the noisy pose $x_m$, conditioning $z$, and step-embedding. The denoiser predicts noise $J=24$0; the DDPM reverse process refines the pose estimate iteratively until $J=24$1, recovering the final motion $J=24$2.

3. Skeleton Mamba Core

Skeleton Mamba is a multi-stage, symmetry-aware bidirectional state-space model (SSM) that explicitly respects human skeletal topology:

• Human Tokenizer: The input tensor $J=24$3 is partitioned into $J=24$4 overlapping joint groups (e.g., limbs), then mapped to tokens $J=24$5 via linear projection.
• Group Scan (Multi-directional SSD): $J=24$6 permutations $J=24$7 are applied along group dimension, producing $J=24$8; an SSD (State-Space Duality [Dao & Gu '24]) block processes these, enforcing $J=24$9-equivariance (symmetry group) and aggregating via inverse-permutation averaging.
• Joint Scan: Activations are mapped from group to individual joints, then processed by a unidirectional SSD per group, concatenated, and inverse-tokenized.
• Temporal Scan: Axes are swapped, and two SSDs scan the sequence forward and backward in time (bidirectional), with their outputs summed; axes are restored.

This structured decomposition exploits spatial and temporal priors for human motion, yielding improved awareness of physical symmetries and superior inter-joint coordination.

4. Diffusion Model and Training Objective

The forward process incrementally adds noise:

$D=6\!-\!9$0

while the reverse network learns:

$D=6\!-\!9$1

The loss is:

$D=6\!-\!9$2

where:

• $D=6\!-\!9$3 is the DDPM “simple” loss (predicted vs. true noise).
• $D=6\!-\!9$4 is a weighted sum of joint position-, velocity-, and contact-losses (e.g., foot sliding; see EDGE [Tseng et al. '23]).
• $D=6\!-\!9$5 (Eq.5) maximizes similarity between concurrent audio-visual embeddings via contrastive log-probability, encouraging temporal multimodal synchronization.

Hyperparameters $D=6\!-\!9$6 are determined by cross-validation.

5. Inference and Head-Guidance

Inference commences with Gaussian noise $D=6\!-\!9$7, running the learned reverse diffusion process. Head-guidance, if enabled, adds a gradient term $D=6\!-\!9$8 to steer output head pose toward an external ego-head estimator’s prediction, improving orientation metrics. Final outputs undergo translation smoothing (1D Gaussian) and foot contact enforcement.

6. Experimental Evaluation

Evaluation on the EgoAIST++ test split employs multiple metrics:

Method	$D=6\!-\!9$9(rad) ↓	$f_v \in \mathbb{R}^{T \times D_c}$0(mm) ↓	MPJPE (mm) ↓	Accel ↓	FS ↓	MMV ↑
PoseReg [2019]	1.78	423.6	351.4	37.1	98.8	0.182
Kinpoly [2021]	1.16	392.7	338.7	16.3	25.8	0.197
EgoEgo [2023]	0.74	373.7	152.0	14.2	22.1	0.218
FACT [2021]	1.54	407.9	173.7	14.6	15.1	0.202
Bailando [2022]	1.57	411.4	175.3	14.7	15.5	0.210
EDGE [2023]	1.52	404.6	167.4	14.4	14.8	0.224
EMM (music only)	1.43	398.4	157.4	14.3	14.0	–
EMM (ego only)	0.61	355.2	186.5	16.0	13.5	–
EMM (ego+music)	0.53	342.4	137.5	11.8	12.8	0.262

EMM with both modalities outperforms all baselines on head orientation, translation, MPJPE, foot-skate, acceleration, and MMV (motion-music-vision alignment). Ablation studies confirm the importance of the multimodal alignment loss ($f_v \in \mathbb{R}^{T \times D_c}$1: MMV drops to 0.242 if removed), head-guidance (improves $f_v \in \mathbb{R}^{T \times D_c}$2 by 0.2 radians), and Skeleton Mamba (10 mm MPJPE gain over transformers/unidirectional SSMs).

Cross-dataset evaluation (EgoExo4D), text-to-motion tasks (HumanML3D), and action recognition benchmarks (NTU60-XS/Kinetics) all demonstrate consistent generalization and performance improvements.

7. Theoretical Guarantees and Future Work

Skeleton Mamba is shown to have a universal approximation property for $f_v \in \mathbb{R}^{T \times D_c}$3-equivariant functions over skeletal joints, where $f_v \in \mathbb{R}^{T \times D_c}$4 is the symmetry group preserved by the HumanTokenizer. The proof leverages the universality of multi-directional SSMs for equivariant mapping, as informed by recent results (Wang & Xue '24).

Identified limitations include vulnerability to input desynchronization, drift on very long temporal sequences (>10 seconds), and the added computational burden of head-guidance. The work proposes several extensions: hierarchical or sliding-window SSMs (e.g., InfiniMotion), improved audio-visual beat alignment, integration of egocentric IMU data, and exploration of perceptual/adversarial losses in diffusion to enhance realism.

EMM with Skeleton Mamba constitutes a principled multimodal approach for full-body motion estimation from egocentric video and music, underpinned by strong empirical results, structure-preserving architecture, and symmetry-theoretic guarantees (Nguyen et al., 14 Aug 2025).