MotionLab: Unified Human Motion Generation and Editing via the Motion-Condition-Motion Paradigm

Abstract: Human motion generation and editing are key components of computer graphics and vision. However, current approaches in this field tend to offer isolated solutions tailored to specific tasks, which can be inefficient and impractical for real-world applications. While some efforts have aimed to unify motion-related tasks, these methods simply use different modalities as conditions to guide motion generation. Consequently, they lack editing capabilities, fine-grained control, and fail to facilitate knowledge sharing across tasks. To address these limitations and provide a versatile, unified framework capable of handling both human motion generation and editing, we introduce a novel paradigm: Motion-Condition-Motion, which enables the unified formulation of diverse tasks with three concepts: source motion, condition, and target motion. Based on this paradigm, we propose a unified framework, MotionLab, which incorporates rectified flows to learn the mapping from source motion to target motion, guided by the specified conditions. In MotionLab, we introduce the 1) MotionFlow Transformer to enhance conditional generation and editing without task-specific modules; 2) Aligned Rotational Position Encoding} to guarantee the time synchronization between source motion and target motion; 3) Task Specified Instruction Modulation; and 4) Motion Curriculum Learning for effective multi-task learning and knowledge sharing across tasks. Notably, our MotionLab demonstrates promising generalization capabilities and inference efficiency across multiple benchmarks for human motion. Our code and additional video results are available at: https://diouo.github.io/motionlab.github.io/.

• The paper introduces MotionLab, a unified framework for human motion generation and editing using the Motion-Condition-Motion paradigm.
• MotionLab employs a MotionFlow Transformer architecture with Joint Attention and Aligned ROPE, showing promising results in text-based and trajectory motion tasks.
• The unified MotionLab architecture reduces overhead in industrial applications like games and movies, enabling easier handling of diverse motion tasks.

Overview of "MotionLab: Unified Human Motion Generation and Editing via the Motion-Condition-Motion Paradigm"

The paper introduces "MotionLab," a framework designed to unify various tasks related to human motion generation and editing. It addresses limitations inherent in existing approaches that often require separate models for different tasks. MotionLab is built around the novel "Motion-Condition-Motion" paradigm, which consolidates task formulations into three components: source motion, condition, and target motion. This approach facilitates the seamless integration of generation and editing tasks within a unified framework.

Key Contributions

1. Motion-Condition-Motion Paradigm: This paradigm provides an elegant solution to unify motion-related tasks. In generation tasks, the source motion is set to null, and the generated motion aligns with specified conditions (e.g., text descriptions, joint trajectories). In editing tasks, modifications are applied to the source motion to meet specified conditions.
2. MotionFlow Transformer (MFT): The architecture uses rectified flows to map source motion to target motion, emphasizing conditional interaction and temporal synchronization between motions. It introduces Joint Attention for modality interplay and incorporates Aligned Rotational Position Encoding (ROPE) to maintain temporal coherence between related motions.
3. Task Instruction Modulation: Leveraging text embeddings from models like CLIP, this component effectively distills task-specific information, allowing the framework to differentiate and handle diverse tasks.
4. Motion Curriculum Learning: This training strategy progressively introduces tasks based on their complexity, ensuring effective multitask learning. Initial training involves masked reconstruction, with subsequent fine-tuning on a hierarchy of tasks distinguished by difficulty and modality involvement.

Numerical Results and Claims

The proposed framework showcases significant versatility and efficiency, with results on tasks such as text-based motion generation, trajectory generation, and style transfer exhibiting promising outcomes. MotionLab offers improved performance in trajectory-based tasks by ensuring low Average Error rates, attributed to its solid pre-training strategies and positional encoding techniques. For text-based generation, while not the top performer, MotionLab achieves a competitive balance of high fidelity (FID) scores and inference speed, highlighting its efficiency.

Implications and Future Directions

Practically, MotionLab's unified architecture reduces the operational overhead of maintaining task-specific models, offering a streamlined solution advantageous in industrial applications like games and movies where diverse motion tasks coexist. Theoretically, the Motion-Condition-Motion paradigm opens avenues for investigating richer interactions between generative tasks, potentially guiding future extensions to handle even more nuanced motion dynamics such as those involving facial expressions or finger movements.

The paper suggests that further research could explore integrating additional human-centric tasks, particularly those that involve intricate body motions, thereby extending the framework's applicability and utility across broader domains of human motion analysis and synthesis.