OmniMotion-X: Unified Motion Generation

• OmniMotion-X is a unified autoregressive diffusion transformer framework that synthesizes whole-body human motion from text, audio, speech, and spatial cues.
• It employs a progressive weak-to-strong multimodal training curriculum to manage complex signal interactions and improve motion synthesis.
• Comprehensive evaluations on the OmniMoCap-X dataset demonstrate its state-of-the-art performance in tasks like text-to-motion, music-to-dance, and speech-to-gesture.

OmniMotion-X is a unified, autoregressive diffusion transformer framework for whole-body human motion generation, notable for its ability to handle versatile multimodal input signals—text, music, speech, spatial constraints, and reference motions—in a single sequence-to-sequence model. It uniquely provides seamless support for a broad range of generation and control tasks, including text-to-motion, music-to-dance, speech-to-gesture, global spatial-temporal control (prediction, in-betweening, completion, trajectory guidance), as well as interactive, mixed-modality scenarios. OmniMotion-X introduces both architectural innovations and novel datasets, and it establishes new state-of-the-art performance benchmarks in multimodal motion synthesis (Xu et al., 22 Oct 2025).

1. Problem Formulation and Motivation

Traditional human motion generation models have treated each modality separately: text-to-motion (T2M) systems map language descriptions to joint trajectories; music-to-dance (M2D) systems ground motion in rhythmic audio features; speech-to-gesture (S2G) aligns body expressions to spoken cues. Multi-modal and interactive control—such as combining text, spatial masks, or user-supplied clips—has typically required custom-designed model branches or the cumbersome integration of separate systems.

OmniMotion-X closes this gap by employing a unified diffusion-based, autoregressive transformer. The key functionalities required in real-world animation, embodied AI, and gaming include:

• Textual grounding of motion (T2M)
• Audio-driven choreography (M2D)
• Speech and gesture coherence (S2G)
• Global spatial-temporal control (GSTC): prediction, in-betweening, completion, joint/trajectory guidance
• Arbitrary mixing and interactive refinement

No prior work has unified all these modalities and tasks under a single generative framework with simultaneous state-of-the-art performance.

2. Core Architecture: Autoregressive Diffusion Transformer

Model Structure

At the core of OmniMotion-X is a DiT-based (Diffusion Transformer) architecture, treating a motion sequence as:

$x_0 = [p_0, p_1, \dots, p_N]$

where each pose $p_i$ encodes SMPL-X body parameters, including root, joint rotations, hand, and facial coefficients. The synthesis process employs discrete-time diffusion, proceeding from noise $x_T$ toward clean poses $x_0$ using the standard Forward Process:

$$q(x_t\,|\,x_{t-1}) = \mathcal{N}(x_t;\sqrt{\alpha_t}x_{t-1},(1-\alpha_t)I)$$

with the learned reverse update step parameterized by a transformer $G_\theta$:

$$p_\theta(x_{t-1}\,|\,x_t,c) \propto \mathcal{N}(x_{t-1};\mu_\theta(x_t,t,c),\sigma_t^2I)$$

where the conditioning vector $c$ aggregates all available modalities. The model directly reconstructs $x_0$, optimizing the mean-squared error $$\mathbb{E}_{x_0, t, c} [\| x_0 - G_\theta(x_t, t, c)\|^2_2]$$.

Diffusion Details

Typical training employs $p_i$0 steps with a cosine noise schedule, yet high fidelity is achieved in as few as 50 sampling steps due to the transformer's expressivity. SMPL-X parameterization ensures compatibility across whole-body, hand, and face synthesis.

3. Multimodal Conditioning and Training Strategy

A central challenge in multimodal motion generation is conflict between high-level semantic signals (e.g., text, music) and strong low-level constraints (e.g., joint paths, reference clips). OmniMotion-X introduces two primary innovations:

a) Reference Motion Conditioning

For superior stylistic and temporal consistency, OmniMotion-X can prepend a reference motion segment $p_i$1—encoded by a compact 1D-conv transformer—alongside other conditions. At inference, $p_i$2 may derive from user input or preceding generated segments, supporting interactive refinement and temporal smoothness.

b) Progressive Weak-to-Strong Mixed-Condition Curriculum

To stabilize multimodal training, a phased curriculum incrementally increases signal complexity. Denote probabilities for text, reference, spatial, and audio conditioning as $p_i$3, $p_i$4, $p_i$5, $p_i$6. The training phases are:

Phase	Steps	Active Conditions
1	0–460K	text
2	460–920K	text, reference
3	920–1150K	text, reference, spatial
4	1150–2070K	text, reference, spatial, audio

This staged schedule ensures initial semantic alignment before introducing strong local or temporal controls, improving both coverage and controllability.

4. Dataset: OmniMoCap-X

Supporting the model's universality, OmniMotion-X utilizes OmniMoCap-X, the largest standardized multimodal motion capture corpus to date. Key properties include:

• Aggregates 28 sources (21 mocap, 7 video-estimated) across ten task domains
• Retargeted all formats (BVH, FBX, keypoints, SMPL, SMPL-H) to SMPL-X, normalized and resampled at 30 fps
• Contains 64.3 million frames (286.2 hours) split into 5-second clips
• Annotation via video renderings and GPT-4o, yielding 321,000+ captions (average 277 words) capturing low-level verbs and high-level semantics

This heterogeneous, richly-annotated dataset provides the empirical foundation for effective generalization across tasks and modalities (Xu et al., 22 Oct 2025).

5. Experimental Evaluation and Performance

Extensive benchmarking across multiple tasks demonstrates the advantage of OmniMotion-X.

• Text-to-motion: R-Precision (Top-1) = 0.303, FID = 5.04, Multimodal Distance = 4.678, Diversity = 8.65; substantially outperforms prior MoMask* (R-Prec = 0.267, FID = 17.43). Incorporating reference motion yields further improvements (R-Prec = 0.346, FID = 3.20).
• Global spatial-temporal control: Outperforms OmniControl (FID = 4.22 vs. 63.7, Top-3 R-Prec = 0.682 vs. 0.392).
• Music-to-dance and speech-to-gesture: Reports lower FID (S2G: 2.64 vs. 3.42, M2D: 5.83 vs. 7.10) and higher diversity.
• Ablation: Removal of the weak-to-strong curriculum degrades both R-Precision and FID, validating its necessity for conflict resolution in multimodal settings.

These results are established on a uniform 280-sample test set spanning all supported tasks.

6. Comparative Context: Extensions and Related Architectures

The unified, transformer-based approach of OmniMotion-X aligns with trends in other domains toward architectural generalization. For instance, the UniMotion framework for autonomous driving employs a decoder-only transformer with specialized attention masks, enabling simultaneous simulation, prediction, and planning under joint next-token and future regression losses. UniMotion demonstrates that such unified backbones promote generalization, modularity, and scalability, with per-task fine-tuning producing SOTA performance on simulation, prediction, and planning tasks (Song et al., 31 Jan 2026). Adopting joint training and shared representations, as in UniMotion, is a recommended strategy for extending OmniMotion-X to new tasks or sensor modalities.

In multimodal perception, OmniEncoder shows that dense, frame-aligned tokenization of audio and vision (at 25 fps), augmented by innovations like Omni-RoPE and temporal window shifting, substantially improves fine-grained motion and audio-visual tasks (Bai et al., 2 May 2026). The integration of similar token templates and position embeddings into OmniMotion-X supports extension to real-time, cross-sensor motion understanding with linear computational scaling.

7. Significance and Future Directions

OmniMotion-X establishes a comprehensive, multimodal generation benchmark for whole-body human motion, addressing limitations of earlier single-branch or separately-trained systems. Its flexible conditioning, progressive training, and unified backbone enable new interactive and cross-modal generation paradigms applicable to animation, gaming, virtual environments, and embodied AI.

Future directions include extending the tokenization and transformer mechanisms to additional sensory modalities (e.g., IMU, LiDAR), scalability via larger unified backbones, cross-dataset pre-training, and real-time deployment. The modular training and representation-sharing strategies described for both OmniMotion-X and related unified frameworks provide a robust path for expansion (Xu et al., 22 Oct 2025, Bai et al., 2 May 2026, Song et al., 31 Jan 2026).