3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation (2412.07759v2)

Abstract: This paper aims to manipulate multi-entity 3D motions in video generation. Previous methods on controllable video generation primarily leverage 2D control signals to manipulate object motions and have achieved remarkable synthesis results. However, 2D control signals are inherently limited in expressing the 3D nature of object motions. To overcome this problem, we introduce 3DTrajMaster, a robust controller that regulates multi-entity dynamics in 3D space, given user-desired 6DoF pose (location and rotation) sequences of entities. At the core of our approach is a plug-and-play 3D-motion grounded object injector that fuses multiple input entities with their respective 3D trajectories through a gated self-attention mechanism. In addition, we exploit an injector architecture to preserve the video diffusion prior, which is crucial for generalization ability. To mitigate video quality degradation, we introduce a domain adaptor during training and employ an annealed sampling strategy during inference. To address the lack of suitable training data, we construct a 360-Motion Dataset, which first correlates collected 3D human and animal assets with GPT-generated trajectory and then captures their motion with 12 evenly-surround cameras on diverse 3D UE platforms. Extensive experiments show that 3DTrajMaster sets a new state-of-the-art in both accuracy and generalization for controlling multi-entity 3D motions. Project page: http://fuxiao0719.github.io/projects/3dtrajmaster

• The paper introduces 3DTrajMaster, a novel method for controllable video generation using 3D trajectory control for multi-entity motion, moving beyond traditional 2D control.
• The technical approach utilizes a plug-and-play object injector within a video diffusion model and addresses data challenges by creating the 360-Motion Dataset for comprehensive training.
• Experimental results demonstrate 3DTrajMaster achieves state-of-the-art accuracy and generalization, opening new applications in virtual cinematography, gaming, and AI training.

An Expert Analysis of "3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation"

In this paper, the authors introduce a novel approach to controllable video generation by focusing on 3D trajectory control for multi-entity motion, a significant departure from the conventional 2D control paradigms. The proposed method, 3DTrajMaster, seeks to address the limitations of prior techniques by offering a robust framework to simulate dynamic object motions in three-dimensional space.

The researchers highlight the inherent inadequacies of 2D control signals, which restrict expressive capabilities concerning 3D motion dynamics such as object rotation and occlusion handling. By employing a unified 3D-motion grounded video diffusion model, 3DTrajMaster facilitates intricate control over object motions, leveraging entity-specific six degrees of freedom (6DoF) pose sequences as inputs. This paradigm shift extends the application potential of video generative models to domains such as virtual cinematography and embodied AI systems.

Technical Approach and Contributions

3DTrajMaster's core innovation lies in its plug-and-play object injector, designed to associate entity descriptions with respective 3D trajectories within a video diffusion framework. The model utilizes a gated self-attention mechanism to fuse 3D motion data with the video model's existing knowledge, maintaining the generalization capability while managing multi-entity scenarios effectively.

One of the challenges addressed by the paper is data scarcity and diversity. Existing datasets lack varied entity types and often fail in pose estimation for non-rigid objects such as animals. To overcome this, the authors construct the 360-Motion Dataset, which combines the power of advanced Unreal Engine rendering to simulate entity motion with GPT-generated trajectory templates, thereby achieving a balanced and comprehensive dataset for training purposes.

The paper's experimental results are noteworthy; 3DTrajMaster sets a new state-of-the-art in accuracy and generalization for 3D motion control. It accomplishes this by introducing techniques such as a video domain adaptor and an annealed sampling strategy, which collectively enhance video quality and maintain motion coherence. This framework allows for elaborate customizations, including nuanced human attributes such as hair and clothing, as well as diverse entity categories ranging from humans to natural forces.

Implications and Future Directions

3DTrajMaster represents a significant step forward in video generation technology. By enabling precise control over 3D object dynamics, it opens up avenues for more realistic and interactive virtual environments. The potential applications in film, gaming, and AI training are vast, allowing for the creation of more complex scenes and interactions.

Looking ahead, further developments could focus on enhancing the granularity of motion and interaction modeling, potentially incorporating more entities simultaneously without performance degradation. Additionally, integrating finer local motions such as those involved in dance or complex interactions between entities may provide further realism and applicability to real-world scenarios.

In conclusion, 3DTrajMaster's introduction of 3D trajectory control into video generation marks a pivotal advancement in the field. The work showcases how integrating 3D spatial understanding into video generative models significantly augments their realism and versatility, setting a new benchmark for the future of video generation research and application.