AnimateZero: Video Diffusion Models are Zero-Shot Image Animators (2312.03793v1)

Abstract: Large-scale text-to-video (T2V) diffusion models have great progress in recent years in terms of visual quality, motion and temporal consistency. However, the generation process is still a black box, where all attributes (e.g., appearance, motion) are learned and generated jointly without precise control ability other than rough text descriptions. Inspired by image animation which decouples the video as one specific appearance with the corresponding motion, we propose AnimateZero to unveil the pre-trained text-to-video diffusion model, i.e., AnimateDiff, and provide more precise appearance and motion control abilities for it. For appearance control, we borrow intermediate latents and their features from the text-to-image (T2I) generation for ensuring the generated first frame is equal to the given generated image. For temporal control, we replace the global temporal attention of the original T2V model with our proposed positional-corrected window attention to ensure other frames align with the first frame well. Empowered by the proposed methods, AnimateZero can successfully control the generating progress without further training. As a zero-shot image animator for given images, AnimateZero also enables multiple new applications, including interactive video generation and real image animation. The detailed experiments demonstrate the effectiveness of the proposed method in both T2V and related applications.

AnimateZero: Zero-Shot Image Animation via Video Diffusion Models

In this paper, the authors introduce AnimateZero, a novel methodology leveraging video diffusion models (VDMs) to achieve zero-shot image animation. AnimateZero builds upon the existing AnimateDiff framework by introducing enhancements that allow for precise control over appearance and motion components during video generation. At the core of this approach is the decoupling of spatial and temporal controls within the pre-trained text-to-video diffusion model.

The innovation in AnimateZero emerges from the integration of text-to-image (T2I) models to provide a foundation for image generation and animation. By utilizing intermediate latents and corresponding features from T2I generation, AnimateZero ensures that the initial video frame directly corresponds to a given image, effectively aligning with the desired visual style. The proposed system introduces a spatial appearance control mechanism, ensuring the synchronization of image-generated spatial properties across the ensuing frames.

One of the standout aspects of AnimateZero is its temporal consistency control. It replaces traditional global attention mechanisms within the motion modules of AnimateDiff with a positional-corrected window attention strategy. This novel attention mechanism facilitates the precise alignment of subsequent frames with the initial key frame, offering a marked improvement in temporal coherence.

Experimentation reveals the robust performance of AnimateZero across various personalized image domains, particularly showcasing its ability to maintain domain consistency which is a significant limitation in AnimateDiff. AnimateZero achieves superior results in text similarity (Text-Sim) and domain similarity (Domain-Sim) metrics, further evidenced by improved warping error outcomes indicative of enhanced temporal precision.

The implications of AnimateZero for practical applications are substantial. Immediate applications include interactive video generation and real image animation, with potential to influence the development of foundational video models and training-based image-to-video approaches. The paper underscores the capacity for zero-shot control over video generation processes, which may lead to substantial shifts in generative AI methodologies.

Moving forward, the research suggests expansions in handling complex motions and diverse image domains. The approach paves the way for broader adoption in varying real-world scenarios where video consistency and fidelity with pre-generated imagery is paramount. The paper offers a methodological step forward in the field of video generation using diffusion models, marking a strategic intersection between pre-trained T2I domains and sophisticated motion application.