Ingredients: Blending Custom Photos with Video Diffusion Transformers (2501.01790v2)

Abstract: This paper presents a powerful framework to customize video creations by incorporating multiple specific identity (ID) photos, with video diffusion Transformers, referred to as Ingredients. Generally, our method consists of three primary modules: (i) a facial extractor that captures versatile and precise facial features for each human ID from both global and local perspectives; (ii) a multi-scale projector that maps face embeddings into the contextual space of image query in video diffusion transformers; (iii) an ID router that dynamically combines and allocates multiple ID embedding to the corresponding space-time regions. Leveraging a meticulously curated text-video dataset and a multi-stage training protocol, Ingredients demonstrates superior performance in turning custom photos into dynamic and personalized video content. Qualitative evaluations highlight the advantages of proposed method, positioning it as a significant advancement toward more effective generative video control tools in Transformer-based architecture, compared to existing methods. The data, code, and model weights are publicly available at: https://github.com/feizc/Ingredients.

• The paper introduces the Ingredients framework that merges custom photos with video diffusion transformers, dramatically enhancing identity preservation in video content.
• It employs a multi-stage training protocol with facial embedding alignment and router fine-tuning to ensure precise identity mapping across video frames.
• Experimental results show significant improvements over baseline methods, highlighting enhanced facial similarity and versatile applications in digital media production.

Overview of "Ingredients: Blending Custom Photos with Video Diffusion Transformers"

The paper "Ingredients: Blending Custom Photos with Video Diffusion Transformers" introduces a novel framework designed to enhance video creation through the integration of multiple specific identity (ID) photos using video diffusion transformers. This method is referred to as Ingredients, which consists of three primary modules to customize video content while maintaining identity consistency and high-quality synthesis.

Core Components of the Framework

Ingredients is composed of three key components:

1. Facial Extractor: This module is tasked with extracting detailed facial features from both global and local perspectives. By utilizing a combination of techniques, it captures versatile and precise facial attributes necessary for identity preservation across videos.
2. Multi-Scale Projector: This component maps the facial embeddings into the contextual space required by video diffusion transformers. It ensures the smooth integration of facial features with the video content, allowing for better blending and consistency.
3. ID Router: The ID router dynamically manages and allocates multiple ID embeddings to the appropriate space-time regions within the video. This dynamic routing is crucial for maintaining identity consistency, especially when multiple IDs are involved.

Methodology

The paper outlines a multi-stage training protocol that effectively integrates text-video datasets. The training is divided into two main phases:

• Facial Embedding Alignment: In this phase, the system optimizes the integration of facial embeddings, focusing on aligning extracted features with the video frames.
• Router Fine-Tuning: The ID router is fine-tuned to ensure precise allocation of identities across space-time positions in generated videos. This process includes implementing a supervisory signal for ID consistency using routing logits and a multi-label cross-entropy loss.

Experimental Validation

Extensive qualitative and quantitative evaluations demonstrate the superiority of the Ingredients framework over existing methods. Numerical results indicate a substantial improvement in identity preservation, with Identity-Preserving Video Generation (IPVG) achieving a higher facial similarity percentage compared to baseline methods. Furthermore, the framework supports several applications, including personal storytelling and promotional video creation, due to its ability to allow precise control over video content aligned with user-defined prompts.

Implications and Future Directions

The development of Ingredients underscores the potential for diffusion transformers in customizable video synthesis. The adaptability of the framework allows for diverse applications and suggests a path forward in generating more personalized and coherent multimedia content.

One significant implication of this research is its applicability in areas requiring high levels of personalization, such as digital avatars and virtual media production. The methodology could be extended to support further developments in AI-driven content creation, potentially incorporating real-time adjustments based on user interactions or external inputs.

Despite its advancements, the paper also acknowledges some limitations, such as initial frame setup issues and ID misclassification during routing. Addressing these will refine the system further.

In conclusion, the Ingredients framework is positioned as a significant step toward more expansive and effective generative video control, providing a reproducible and extendable benchmark for future research in video diffusion models. Through its integration of sophisticated modules and training strategies, it sets a foundation for more refined and controllable video generation technologies.