Scalable Training of Large-Scale Text-to-Video Foundation Models

Develop scalable training methodologies for large-scale text-to-video foundation models that effectively handle the complexities introduced by modeling motion, in order to synthesize realistic, temporally coherent videos under stringent memory, compute, and data scale constraints.

Background

The paper emphasizes that moving from images to videos adds a temporal dimension that substantially increases modeling complexity and resource demands. Beyond capturing realistic motion, training text-to-video models must contend with higher memory and compute requirements and larger datasets to learn the more complex distribution of videos.

The authors motivate their architectural choice (a Space-Time U-Net generating full clips at once) as a response to these challenges, contrasting it with cascaded temporal super-resolution approaches that struggle with global temporal consistency.