Follow-Your-Click: Open-domain Regional Image Animation via Short Prompts (2403.08268v1)

Abstract: Despite recent advances in image-to-video generation, better controllability and local animation are less explored. Most existing image-to-video methods are not locally aware and tend to move the entire scene. However, human artists may need to control the movement of different objects or regions. Additionally, current I2V methods require users not only to describe the target motion but also to provide redundant detailed descriptions of frame contents. These two issues hinder the practical utilization of current I2V tools. In this paper, we propose a practical framework, named Follow-Your-Click, to achieve image animation with a simple user click (for specifying what to move) and a short motion prompt (for specifying how to move). Technically, we propose the first-frame masking strategy, which significantly improves the video generation quality, and a motion-augmented module equipped with a short motion prompt dataset to improve the short prompt following abilities of our model. To further control the motion speed, we propose flow-based motion magnitude control to control the speed of target movement more precisely. Our framework has simpler yet precise user control and better generation performance than previous methods. Extensive experiments compared with 7 baselines, including both commercial tools and research methods on 8 metrics, suggest the superiority of our approach. Project Page: https://follow-your-click.github.io/

• The paper introduces a novel framework that animates image regions using a simple click and concise prompt to overcome controllability challenges.
• It employs innovations such as first-frame masking and a motion-augmented module to improve temporal consistency and detail in video generation.
• Flow-based motion magnitude control enables precise adjustment of animation speed and intensity, significantly outperforming prior methods.

Enhancing Regional Image Animation with Follow-Your-Click Framework

Introduction

Image-to-video generation (I2V) is a prominent task aimed at animating static images to produce realistic and coherent video sequences. Despite significant progress in the field, existing methods have limitations regarding controllability, especially for local animation, and often require detailed descriptions of the scene for prompt-based methods. The novel framework "Follow-Your-Click" addresses these challenges by providing a practical solution for region-specific animation in images, requiring only a user-specified point (click) and a concise motion prompt to guide the animation.

Key Contributions

The paper introduces several technical innovations to achieve this fine-grained control over the animation process:

• First-Frame Masking Strategy: This technique significantly enhances video generation quality by leveraging a masking mechanism that improves temporal consistency and detail retention in generated animations.
• Motion-Augmented Module: To effectively utilize short motion prompts, a specialized module is proposed, complemented by a custom dataset curated to emphasize motion-related phrases, thereby improving the model's sensitivity to concise instructions.
• Flow-Based Motion Magnitude Control: A novel approach to controlling the animation's speed and intensity more precisely by utilizing optical flow estimates, moving beyond traditional FPS-based adjustments and achieving a more nuanced manipulation of motion.

Technical Details and Implementation

"Follow-Your-Click" utilizes latent diffusion models (LDMs) as its backbone for generation, with novel interventions in the form of a motion-augmented module and first-frame masking for enhanced control and quality. The framework is trained on a purpose-built dataset (WebVid-Motion) focusing on short motion cues to closely follow user prompts. It supports segmentation-to-animation conversion, allowing a simple user click to define the region of interest for animation, significantly simplifying the user interface for specifying animation targets.

Evaluation and Results

Extensive experiments showcase the framework's superiority in generating high-quality animations with localized movements, significantly outperforming existing baselines across multiple metrics such as $I_1$-MSE, Temporal Consistency, Text-Alignment, and FVD. The framework demonstrates remarkable aptitude in adhering to user-specified regions for animation without unnecessary global scene movements, preserving the static aspects of scenes as intended by the user. This represents a significant advance over prior methods, which often lack this level of control or require detailed scene descriptions for animation.

Implications and Future Directions

"Follow-Your-Click" opens up new possibilities for user-controlled animation, providing tools that can significantly streamline workflows for artists, filmmakers, and content creators, offering precise control over the movement within their visual pieces. Future work could explore the integration of this framework with three-dimensional animation and real-time animation systems, further broadening its applicability and impact on multimedia, gaming, and virtual reality experiences.

Conclusion

The "Follow-Your-Click" framework represents a significant step forward in the domain of image-to-video generation, specifically addressing the need for better user control and efficiency in animating selected regions of images. By simplifying the input required from the user to a click and a short prompt, while also introducing advanced technical strategies to improve generation quality and motion control, this work paves the way for more intuitive, effective, and creative animation tools in various applications.