Stylized Text-to-Motion Generation via Hypernetwork-Driven Low-Rank Adaptation

Abstract: Text-driven motion diffusion models are capable of generating realistic human motions, but text alone often struggles to express fine-level nuances of motion, commonly referred to as style. Recent approaches have tackled this challenge by attaching a style injection mechanism to a pretrained text-driven diffusion model. Existing stylization methods, however, either require style-specific fine-tuning of existing models or rely on heavy ControlNet-based architectures, limiting efficiency and generalization to unseen styles. We propose a lightweight style conditioning framework that dynamically modulates a pretrained diffusion model through hypernetwork-generated LoRA parameters. A style reference motion is encoded into a global style embedding, which is mapped by a hypernetwork to low-rank updates applied at each denoising step of the diffusion model. By structuring the style latent space with a supervised contrastive loss, our framework reliably captures diverse stylistic attributes, improves generalization to unseen styles, and supports optimization-based guidance without requiring predefined style categories. Experiments on the HumanML3D and 100STYLE datasets show state-of-the-art stylization results, while achieving improved stylization for unseen styles.

• The paper introduces a hypernetwork-driven LoRA conditioning method that adapts a frozen diffusion model to generate stylistically controlled motion from text and reference inputs.
• It leverages a supervised contrastive style latent space and gradient-based style encoder guidance to ensure robust generalization to unseen motion styles.
• Quantitative evaluations on HumanML3D and 100STYLE benchmarks demonstrate improved style recognition accuracy and motion quality with minimal computational overhead.

Hypernetwork-Driven Low-Rank Adaptation for Stylized Text-to-Motion Generation

Introduction

The paper "Stylized Text-to-Motion Generation via Hypernetwork-Driven Low-Rank Adaptation" (2605.13333) addresses the significant gap in controlling fine-grained motion style within state-of-the-art text-driven human motion diffusion models. While recent diffusion-based models capture text-to-motion mapping effectively, transferring and generalizing expressive style—such as emotional nuance or personality—remains challenging, particularly for unseen styles or when scaling to many stylistic modes. To this end, the proposed framework introduces a hypernetwork-driven conditioning pipeline leveraging LoRA-based parameter-efficient adaptation, combined with a supervised contrastive style representation and gradient-based guidance mechanisms.

The approach efficiently modulates a pretrained diffusion backbone to generate stylistically controllable motion from text and reference motion, achieving recognizable stylistic cues, strong generalization to unseen styles, and scalability without the computational burdens of prior architectures.

Methodological Overview

The central technical contribution is a modular pipeline building upon the text-conditioned skeleton-aware latent diffusion model SALAD [hong2025salad]. The key innovations can be decomposed into three tightly integrated components:

1. Hypernetwork-Driven LoRA Conditioning: Unlike ControlNet architectures or per-style fine-tuned models, a global style embedding is extracted from a reference motion. This embedding is mapped by a hypernetwork to generate LoRA parameters, which are injected as a low-rank update to the FiLM layers of the frozen diffusion model at each denoising step, enabling expressive and efficient style modulation.
2. Supervised Contrastive Style Latent Space: The style adapter is trained to produce a structured, content-invariant latent style space using supervised contrastive loss, ensuring that motions sharing style collapse to a common region, facilitating both discriminability and generalizability.
3. Style Encoder Guidance: During inference, an additional gradient-based guidance derived from the style embedding directly steers the latent trajectory, enabling instance-level, classifier-free control over stylistic fidelity for both seen and unseen styles.

   Figure 1: Schematic of the proposed method showing text and reference-based style conditioning, hypernetwork-generated LoRA injection, and style guidance throughout the denoising process.

   

   Figure 2: Comparison between SALAD's base FiLM mechanism (left) and the proposed HyperLoRA design (right), where low-rank style-dependent modifications are applied to FiLM projections.

Quantitative and Qualitative Results

Extensive evaluations are conducted on HumanML3D and 100STYLE benchmarks. The primary metrics are Style Recognition Accuracy (SRA), content preservation (R-Precision, MM Distance), and motion quality (FID, FSR), alongside perceptual user studies.

Numerical Highlights:

• The proposed method achieves an SRA of 76.0%, outperforming SMooDi zhong2024smoodi, LoRA-MDM sawdayee2025dance, and the unsupervised baseline wu2025semantically.
• Strong content preservation ($\sim$0.72 R-Precision) and competitive FID/FSR indicate minimal tradeoff between realism and stylization, outperforming ControlNet-based or per-style LoRA baselines, particularly for unseen styles.

Salient Claims:

• The claim is made that this framework achieves state-of-the-art stylization with significantly improved generalization to unseen styles, not requiring explicit style classifier supervision during inference.
• The architecture is highly parameter-efficient (minimal overhead relative to backbone), scalable, and outperforms heavy ControlNet schemes on both quantitative, qualitative, and runtime axes.

  Figure 3: Qualitative comparison illustrating expressive stylization and content preservation; generated sequences faithfully reflect both style and text content.

  

  Figure 4: Ablation on supervised contrastive loss and style encoder guidance; style expression and content metrics across varying training style cardinalities.

  

  Figure 5: Effect of style encoder guidance on qualitative stylization for a model trained on 100 styles.

Generalization and Ablations

The authors systematically probe the contributions of each design component:

• Supervised Contrastive Loss: The structured style latent produced by this loss is essential for content-invariant, generalizable style extraction, especially when training data is limited (e.g., 25 styles). Removal severely degrades expressive capacity and generalization.
• Style Encoder Guidance: Ablating this inference-time mechanism reduces SRA by $\sim$20 points, underscoring its necessity for balancing style and content tradeoffs during sampling.
• Architectural Choices: The HyperLoRA adapter demonstrates both higher SRA and lower runtime versus an equivalent ControlNet-based adapter, supporting claims of efficiency and expressivity.

  Figure 6: Generalization to unseen styles—motions generated by models trained on 100 vs. 25 styles; style transfer is robust even when reference styles are not present in training.

Applications

The framework supports several compelling downstream use cases:

• Controlled Stylized Generation: Integration of external trajectory and keyframe constraints during sampling is achieved via gradient-based updates in latent space, allowing for precise, style-conditioned motion control.

  Figure 7: Controlled generation with trajectory (left) and keyframe (right) constraints imposed during sampling.

• Motion Style Transfer: Through DDIM inversion, existing motions are adapted to new styles, enabling example-based editing and transfer with high content and style fidelity.

  Figure 8: Motion style transfer from an input sequence and style reference; output sequence retains the source action executed with reference style.

Theoretical and Practical Implications

This work introduces a unified, scalable paradigm for text-driven stylized motion generation. The decoupled, continuous style representation and parameter-efficient adaptation address longstanding issues in both style transfer and generative diversity, especially the bottlenecks posed by discrete style supervision or heavyweight multi-branch architectures. The combination of contrastive style structure and instance-level guidance provides an extensible mechanism for applications beyond current text-to-motion generation, including style manipulation, personalized animation synthesis, and controllable simulation environments.

For future research, broadening the latent style space to encompass a richer array of movement modalities (beyond locomotion), exploring relational/anchor-based style definitions, and harnessing unsupervised or large-scale pretraining for style extraction are pivotal directions. The demonstrated ability to generalize to unseen styles without classifier retraining or explicit label dependency suggests potential for robust transfer and adaptation in interactive and real-time systems.

Conclusion

The paper delivers a rigorous, modular solution for stylized text-to-motion generation, combining the flexibility and efficiency of hypernetwork-driven LoRA adaptation with structured, contrastive style space learning and gradient-based style guidance. Empirical evidence confirms state-of-the-art performance across style, content, and realism metrics, with strong generalization to unseen styles and substantial efficiency advantages over prior art. This work establishes a technical foundation for future developments in generative motion stylization and multi-modal, controllable character animation.