MotiF: Making Text Count in Image Animation with Motion Focal Loss

Abstract: Text-Image-to-Video (TI2V) generation aims to generate a video from an image following a text description, which is also referred to as text-guided image animation. Most existing methods struggle to generate videos that align well with the text prompts, particularly when motion is specified. To overcome this limitation, we introduce MotiF, a simple yet effective approach that directs the model's learning to the regions with more motion, thereby improving the text alignment and motion generation. We use optical flow to generate a motion heatmap and weight the loss according to the intensity of the motion. This modified objective leads to noticeable improvements and complements existing methods that utilize motion priors as model inputs. Additionally, due to the lack of a diverse benchmark for evaluating TI2V generation, we propose TI2V Bench, a dataset consists of 320 image-text pairs for robust evaluation. We present a human evaluation protocol that asks the annotators to select an overall preference between two videos followed by their justifications. Through a comprehensive evaluation on TI2V Bench, MotiF outperforms nine open-sourced models, achieving an average preference of 72%. The TI2V Bench and additional results are released in https://wang-sj16.github.io/motif/.

• The paper presents a novel Motion Focal Loss to enhance text-driven video generation by emphasizing high-motion regions.
• It integrates seamlessly with existing diffusion frameworks and introduces TI2V Bench for robust evaluation of text-to-video models.
• Human evaluations show a 72% preference, demonstrating improved alignment between dynamic video content and textual descriptions.

Overview of MotiF: Improving Text-Alignment in Image Animation Through Motion Focal Loss

The research paper presents "MotiF," a novel approach aimed at enhancing the capabilities of Text-Image-to-Video (TI2V) generation models. The primary challenge addressed is improving these models' ability to generate videos that are not just visually consistent with the initial image but also semantically aligned with the accompanying text description, particularly in animating motion as specified by the text prompts. MotiF is introduced as a mechanism to direct the model's learning towards regions with greater motion, employing a unique loss reweighting strategy termed Motion Focal Loss.

Key Contributions and Methods

• Motion Focal Loss (MotiF): The core contribution is the novel Motion Focal Loss, which aims to enhance the fidelity of text-aligned motion generation by imposing a focus on the dynamic regions of the video. By calculating a motion heatmap using optical flow and weighting the loss functions based on the intensity of motion across frames, MotiF effectively encourages models to pay more attention to regions of high motion.
• MotiF's Compatibility with Existing Frameworks: MotiF is agnostic to the underlying diffusion model architecture and complements existing strategies that enhance motion learning by manipulating input signals. It can be integrated into any existing TI2V pipeline, emphasizing its robustness and flexibility.
• Benchmarking and Evaluation Protocols: The authors also address a crucial gap in TI2V evaluation by introducing TI2V Bench, a dataset comprising diverse image-text pairs designed explicitly for benchmarking text-guided video generation. Alongside this dataset, a comprehensive evaluation protocol is detailed, focusing on human judgments of video quality concerning text adherence, image quality, object motion, and overall quality.

Results and Implications

MotiF demonstrates significant improvements over existing models, as evidenced by an extensive human evaluation that shows MotiF's results are preferred 72% of the time compared to nine open-sourced models. The results underscore the effectiveness of MotiF in facilitating better text alignment in video generation.

• Performance Analysis: The evaluation protocol revealed that MotiF is particularly advantageous in improving text alignment and object motion within the generated videos. This suggests that focusing on motion-dense areas during training can mitigate common issues like conditional image leakage, where models overly rely on static elements at the expense of dynamic behavior alignment to text prompts.
• Challenges and Future Research: Despite its overall success, MotiF still faces difficulties in scenarios involving complex object interactions or the introduction of novel objects. These challenges indicate directions for future research, such as refining motion prior generation techniques or integrating more sophisticated scene understanding capabilities into these models.
• Theoretical and Practical Implications: Theoretically, MotiF’s use of motion-centric training objectives might inspire similar methodologies across other domains where motion dynamics play a critical role. Practically, its applications could extend beyond entertainment to areas such as automated storytelling or simulation training where accurate and engaging visual narratives are valuable.

Conclusion

MotiF represents a significant step forward in the TI2V field by addressing the nuanced challenge of aligning motion generation closely with specified text while maintaining visual consistency. The introduction of the TI2V Bench sets a new standard for evaluating these models, paving the way for continued advancements in this domain. Future work could explore integrating advanced motion segmentation techniques and expanding the model's understanding of diverse textual descriptions for even more robust video content generation.