Tora: Trajectory-oriented Diffusion Transformer for Video Generation (2407.21705v3)

Abstract: Recent advancements in Diffusion Transformer (DiT) have demonstrated remarkable proficiency in producing high-quality video content. Nonetheless, the potential of transformer-based diffusion models for effectively generating videos with controllable motion remains an area of limited exploration. This paper introduces Tora, the first trajectory-oriented DiT framework that concurrently integrates textual, visual, and trajectory conditions, thereby enabling scalable video generation with effective motion guidance. Specifically, Tora consists of a Trajectory Extractor(TE), a Spatial-Temporal DiT, and a Motion-guidance Fuser(MGF). The TE encodes arbitrary trajectories into hierarchical spacetime motion patches with a 3D video compression network. The MGF integrates the motion patches into the DiT blocks to generate consistent videos that accurately follow designated trajectories. Our design aligns seamlessly with DiT's scalability, allowing precise control of video content's dynamics with diverse durations, aspect ratios, and resolutions. Extensive experiments demonstrate Tora's excellence in achieving high motion fidelity, while also meticulously simulating the intricate movement of the physical world. Code is available at: https://github.com/alibaba/Tora.

Tora: Trajectory-oriented Diffusion Transformer for Video Generation

The research paper "Tora: Trajectory-oriented Diffusion Transformer for Video Generation," authored by Zhenghao Zhang et al. from Alibaba Group, explores advanced video generation using a Trajectory-oriented Diffusion Transformer (DiT) framework. This paper addresses the current limitations in motion control posed by traditional video generation models and introduces innovative techniques for generating high-fidelity, motion-controllable videos.

Overview

The authors present Tora, a novel trajectory-oriented Diffusion Transformer framework designed for video generation. This methodology integrates text, image, and trajectory conditions concurrently, significantly enhancing the capacity for motion control and generation versatility. Tora leverages the scalable nature of DiT to precisely generate video content with diverse durations, aspect ratios, and resolutions. Unlike prior methods constrained by fixed resolutions and short durations, Tora is capable of producing long videos, up to 204 frames, at a 720p resolution.

Technical Components

Tora's architecture consists of three core components:

1. Trajectory Extractor (TE): Utilizes a 3D video compression network to encode arbitrary trajectories into hierarchical spacetime motion patches. This component ensures that motion trajectories are represented as visualized displacement maps, which are then converted into latent representations compatible with video patches.
2. Spatial-Temporal Diffusion Transformer (ST-DiT): The ST-DiT alternates between Spatial DiT Blocks (S-DiT-B) and Temporal DiT Blocks (T-DiT-B). After reducing video dimensions using an autoencoder, it processes input video sequences to handle variable durations and ensures spatial and temporal consistency.
3. Motion-guidance Fuser (MGF): Integrates adaptive normalization layers to embed multi-level motion conditions into corresponding DiT blocks. This module is critical for maintaining alignment with the trajectory conditions and enhances the model's ability to generate videos that follow specified trajectories.

Methodology and Results

The authors propose a two-stage training strategy where dense optical flow data is initially used to accelerate motion learning, followed by fine-tuning with specified sparse trajectories. This strategy ensures precise motion control over arbitrary trajectories.

Quantitative evaluations demonstrate Tora's superiority over existing video generation models like VideoComposer, DragNUWA, and MotionCtrl. Specifically, Tora maintains stable motion control performance across varying frame numbers and resolutions, showcasing a significant reduction in Trajectory Error and improved FVD and CLIPSIM metrics.

Implications and Future Directions

The practical implications of Tora are considerable. The ability to generate videos with precise motion control makes it applicable to diverse domains such as animated content creation, virtual reality, and autonomous driving simulations. Additionally, the research offers significant theoretical advancements by integrating transformer-based scaling properties into video synthesis, thus overcoming the capacity constraints of traditional U-Net architectures.

Future directions for this research could include the exploration of additional motion conditions, such as gestures or body poses, to further enhance the model's flexibility. Moreover, an investigation into reduced computational costs and efficiency improvements will be crucial for practical real-world applications.

Conclusion

This paper establishes a significant advancement in trajectory-oriented video generation. By integrating hierarchical spacetime motion patches and adaptive normalization layers, Tora sets a new benchmark for motion-controllable video generation models. Its ability to generate high-quality videos with precise trajectory alignment underscores its robustness and practical utility, making it a valuable contribution to the field of video synthesis and AI-driven content creation.

This summary provides a rigorous overview of the Tora framework and its considerable enhancements in video generation, aimed at experienced researchers and practitioners in the field of computer science and artificial intelligence.