On-device Sora: Enabling Training-Free Diffusion-based Text-to-Video Generation for Mobile Devices (2502.04363v2)

Abstract: We present On-device Sora, the first model training-free solution for diffusion-based on-device text-to-video generation that operates efficiently on smartphone-grade devices. To address the challenges of diffusion-based text-to-video generation on computation- and memory-limited mobile devices, the proposed On-device Sora applies three novel techniques to pre-trained video generative models. First, Linear Proportional Leap (LPL) reduces the excessive denoising steps required in video diffusion through an efficient leap-based approach. Second, Temporal Dimension Token Merging (TDTM) minimizes intensive token-processing computation in attention layers by merging consecutive tokens along the temporal dimension. Third, Concurrent Inference with Dynamic Loading (CI-DL) dynamically partitions large models into smaller blocks and loads them into memory for concurrent model inference, effectively addressing the challenges of limited device memory. We implement On-device Sora on the iPhone 15 Pro, and the experimental evaluations show that it is capable of generating high-quality videos on the device, comparable to those produced by high-end GPUs. These results show that On-device Sora enables efficient and high-quality video generation on resource-constrained mobile devices. We envision the proposed On-device Sora as a significant first step toward democratizing state-of-the-art generative technologies, enabling video generation on commodity mobile and embedded devices without resource-intensive re-training for model optimization (compression). The code implementation is available at a GitHub repository(https://github.com/eai-lab/On-device-Sora).

• The paper proposes On-device Sora, a framework to enable diffusion-based text-to-video generation directly on resource-constrained mobile devices.
• It introduces novel techniques like Linear Proportional Leap, Temporal Dimension Token Merging, and Concurrent Inference with Dynamic Loading to optimize computation and memory for mobile inference.
• Experiments show that On-device Sora on an iPhone 15 Pro achieves video generation quality comparable to high-end GPUs, enabling private and efficient on-device applications.

Enabling Diffusion-Based Text-to-Video Generation on Mobile Devices: An Overview of On-device Sora

The paper "On-device Sora: Enabling Diffusion-Based Text-to-Video Generation for Mobile Devices" addresses a significant challenge in contemporary machine learning: the execution of complex generative models on mobile and resource-constrained devices. This work proposes On-device Sora, an adaptation of the Open-Sora video generation framework, engineered to operate within the computation and memory limitations inherent to mobile devices, such as smartphones.

Key Contributions and Methodologies

On-device Sora introduces a triad of innovative techniques aimed at overcoming the computational bottlenecks typically associated with diffusion-based text-to-video models. The following are the system's primary components:

1. Linear Proportional Leap (LPL): To mitigate the computational load from the numerous denoising steps required by diffusion models, LPL leverages the Rectified Flow methodology to significantly reduce denoising times without new architectural changes or retraining of the model. The authors report nearly a 50% reduction in the number of denoising steps needed for video output from the original model while maintaining video quality integrity.
2. Temporal Dimension Token Merging (TDTM): Recognizing the efficiency hurdle posed by token processing in video diffusion models, this method implements a novel approach to token merging along the temporal dimension of video frames within attention layers. By merging consecutive frame tokens, this approach reduces the token count by half, thereby improving computational efficiency up to four times for self-attention modules.
3. Concurrent Inference with Dynamic Loading (CI-DL): CI-DL addresses the memory constraints present in mobile devices through a dynamic model execution strategy. By partitioning models into smaller blocks and utilizing a high degree of parallelism between model loading and execution, On-device Sora efficiently manages memory allocation during inference, effectively utilizing available computational resources without compromising latency or model performance.

Experimental Evaluation

The On-device Sora was implemented on the iPhone 15 Pro, which is substantially less powerful than traditional high-end GPUs used for such generative tasks. Experimentally, this system was evaluated using VBench, a comprehensive benchmark suite for video generative models. When compared to the original Open-Sora running on NVIDIA A6000 GPUs, On-device Sora achieved comparable video generation quality. For example, when producing 68-frame videos at 256×256 resolution, the generated content maintained high subject and background consistency. It exhibited a minimal framing aesthetic quality drop, balanced by a slight improvement in dynamic degree.

Implications and Future Directions

The successful deployment of On-device Sora has several promising implications for both theoretical advancement and practical applications. From a theoretical standpoint, it underscores the potential for complex diffusion models to be simplified and optimized for usage in low-power environments without extensive retraining. Practically, this work paves the way for enhanced privacy, resource efficiency, and affordability in generative video technology. By reducing dependence on cloud services, users can enjoy lower latency, increased privacy, and more personalized content generation.

As the authors discuss future directions, there is a prospect for expanding this methodology to multi-modal generation tasks, such as image-to-video and real-time video synthesis directly on devices. Further exploration into leveraging device-specific hardware accelerators such as NPUs could also yield significant speed improvements and efficiency in future iterations of mobile video generation systems.

In conclusion, On-device Sora makes a substantial contribution to expanding the capabilities of text-to-video diffusion models to mobile platforms, demonstrating innovative computational strategies that balance performance with hardware constraints. This work sets the groundwork for democratizing video generation technology, suggesting future research avenues that could bring even more sophisticated generative capacities directly into the hands of users via everyday devices.