Do You Have Freestyle? Expressive Humanoid Locomotion via Audio Control

Abstract: Humans intuitively move to sound, but current humanoid robots lack expressive improvisational capabilities, confined to predefined motions or sparse commands. Generating motion from audio and then retargeting it to robots relies on explicit motion reconstruction, leading to cascaded errors, high latency, and disjointed acoustic-actuation mapping. We propose RoboPerform, the first unified audio-to-locomotion framework that can directly generate music-driven dance and speech-driven co-speech gestures from audio. Guided by the core principle of "motion = content + style", the framework treats audio as implicit style signals and eliminates the need for explicit motion reconstruction. RoboPerform integrates a ResMoE teacher policy for adapting to diverse motion patterns and a diffusion-based student policy for audio style injection. This retargeting-free design ensures low latency and high fidelity. Experimental validation shows that RoboPerform achieves promising results in physical plausibility and audio alignment, successfully transforming robots into responsive performers capable of reacting to audio.

• The paper introduces RoboPerform, a framework for direct audio-to-locomotion control that generates expressive dance and co-speech gestures.
• It employs a DeltaMixture-of-Experts policy and diffusion-based distillation with transformer-driven audio alignment to ensure accurate, retargeting-free motion.
• Results show >93% task success and low positional errors in both simulated and real-world settings, emphasizing low-latency, robust performance.

Expressive Humanoid Locomotion via Audio Control: Analysis of RoboPerform

Introduction

"Do You Have Freestyle? Expressive Humanoid Locomotion via Audio Control" (2512.23650) introduces RoboPerform, a framework for direct, unified audio-to-locomotion control for humanoid robots. This approach enables robots to generate music-driven dance and speech-driven co-speech gestures from audio in a retargeting-free and low-latency manner. The system hinges on the physical and stylistic alignment of locomotion with audio, leveraging advances in reinforcement learning, mixture-of-experts architectures, and diffusion-based policy distillation to achieve temporally and stylistically coherent motion.

Figure 1: RoboPerform enables a humanoid to act as both dancer and talker; audio serves as the signal to control locomotion, generating rhythm-aligned gestures and dance via speech or music.

Methodology

Motion Decomposition: Content and Style

The framework conceptualizes motion as the sum of content (task semantics, e.g., "dancing" or "giving a speech") and style (audio-driven properties like rhythm and prosody). Content is abstracted into a latent vector with a pretrained text-to-motion model, while audio forms the continuously varying style signal. By decoupling these factors, RoboPerform can generate consistent, meaningful actions that vary expressively across diverse audio contexts.

Audio-Kinematic Alignment via InfoNCE

A transformer-based adaptor aligns raw audio latents with motion latents using InfoNCE contrastive loss, ensuring that audio signals are injected with kinematic structure and can modulate action generation in a way that is physically consistent with robot actuation requirements. This eliminates the need for explicit intermediate motion reconstruction and reduces system latency.

Delta Mixture-of-Experts Teacher Policy

At the core of RoboPerform is the $\Delta$MoE teacher policy, a residual mixture-of-experts (MoE) that partitions the condition space and enforces expert specialization via residual learning. Each expert receives an incrementally augmented subset of input conditions. The gating network assigns weights to each expert, and final actions are synthesized as a weighted sum of residual expert outputs. This structure enables fine-grained disentanglement of content and style, and eliminates redundancy observed in standard MoE approaches.

Figure 2: Overview of DeltaMoE, showing subspace partitioning and expert residual fusion.

Figure 3: t-SNE visualizations demonstrate independent specialization in $\Delta$MoE, in contrast to heavily overlapped vanilla MoE experts.

Diffusion-Based Student Policy

A diffusion model is trained as a student via DAgger-style distillation—the content latent provides foundational action, and the audio latent is injected as a style-control signal at multiple diffusion layers (AdaLN conditioning). The denoising process reconstructs joint-level actuation in a manner that dynamically aligns style with contextual audio, accommodating varying beat, rhythm, and energy.

Experimental Results

Datasets and Metrics

Evaluations use FineDance (3D full-body dance with paired music) and BEAT2 (co-speech gesture with audio, 30 speakers). Metrics include retrieval precision for audio-motion alignment, and, for control, task success rate, mean per-joint position error ($E_\mathrm{MPJPE}$), and mean per-keypoint position error ($E_\mathrm{MPKPE}$).

Audio-Motion Alignment

RoboPerform's audio adaptor achieves precise multimodal alignment, with R@1 scores of 66.7 (music) and 64.6 (speech), indicating effective structure transfer from audio to motion latent space.

Motion Tracking

The framework demonstrates high success rates (>93%) and low $E_\mathrm{MPJPE}$ on both speech-to-gesture and music-to-dance across simulated (IsaacGym, MuJoCo) and real-world (Unitree G1) platforms, outperforming baseline pipelines that employ explicit motion reconstruction and retargeting.

Figure 4: Ablation study highlights tracking improvements over pose-driven baselines, demonstrating reduced error and latency in retargeting-free control.

Figure 5: Qualitative tracking in IsaacGym and MuJoCo on music-to-locomotion and speech-to-locomotion, showcasing temporal alignment and dynamical fidelity.

Real-World Deployment

Direct deployment on Unitree G1 demonstrates practical low-latency performance, with the policy robust to uncontrolled environmental conditions—significant for real-time robot interaction.

Figure 6: Real-world music-to-locomotion.

Figure 7: Real-world speech-to-locomotion.

Ablation Studies

Comprehensive ablations underline the necessity of all major components:

• $\Delta$MoE vs Vanilla MoE: $\Delta$MoE significantly outperforms vanilla MoE in tracking accuracy and expert specialization, confirmed by independent t-SNE clusters in the $\Delta$MoE decomposition.
• Content-Style Disentanglement: Removing the content latent degrades task success by 3-5% and increases position error, confirming the importance of semantic separation.
• Kinematic Audio Adaptor: Injecting kinematic priors via the audio adaptor improves joint and keypoint accuracy, with rhythm hit rates closely tracking music beats—critical for expressive dance/gesture alignment.

  Figure 8: Comparison of MLP-based and diffusion policy on seen/unseen music; the diffusion policy shows consistent tracking and freestyle generalization.

Theoretical and Practical Implications

RoboPerform reframes the humanoid motion control problem as direct generation conditioned on hybrid semantic (content) and implicit style (audio) signals, eschewing the bottlenecks and error-accumulation of cascade retargeting pipelines. The $\Delta$MoE construction generalizes classifier-free guidance to arbitrary conditional hierarchies, providing a flexible framework for integrating new control modalities in robotics, e.g., visual or gestural inputs, with minimal policy redesign.

Practically, the reduced latency and retargeting-free inference are pivotal for interactive robots, supporting real-time improvisational performance and closed-loop adaptation to dynamic or user-driven environments. The successful sim-to-real transfer and robust low-level actuation further endorse the architecture for deployment in physically embodied agents.

Future Directions

This content-style separation and direct audio-driven paradigm can be extended toward:

• Multimodal Conditional Control: Integration with vision or multimodal context signals for richer semantic grounding.
• Generalization to Arbitrary Styles: Transfer learning to new styles or gestures by retraining only the audio adaptor.
• Adaptive Online Freestyle: Online adaptation modules for continuous unsupervised alignment as new styles emerge during interaction.
• Scalability: Larger-scale training with reinforcement learning in the loop, or hierarchical diffusion policies for even more dexterous skills.

Conclusion

RoboPerform establishes a new approach for expressive, context-sensitive humanoid control by leveraging direct alignment between audio and motion within a disentangled content-style framework. The use of a $\Delta$MoE teacher policy with diffusion-based distillation achieves both high-fidelity task grounding and temporally aligned performative behaviors. Its retargeting-free, low-latency design enables real-world deployment for humanoids responding “in freestyle” to arbitrary audio—paving the way for new research in multimodal robotic control and embodied intelligence.