Identity-Preserving Text-to-Video Generation by Frequency Decomposition (2411.17440v3)

Abstract: Identity-preserving text-to-video (IPT2V) generation aims to create high-fidelity videos with consistent human identity. It is an important task in video generation but remains an open problem for generative models. This paper pushes the technical frontier of IPT2V in two directions that have not been resolved in literature: (1) A tuning-free pipeline without tedious case-by-case finetuning, and (2) A frequency-aware heuristic identity-preserving DiT-based control scheme. We propose ConsisID, a tuning-free DiT-based controllable IPT2V model to keep human identity consistent in the generated video. Inspired by prior findings in frequency analysis of diffusion transformers, it employs identity-control signals in the frequency domain, where facial features can be decomposed into low-frequency global features and high-frequency intrinsic features. First, from a low-frequency perspective, we introduce a global facial extractor, which encodes reference images and facial key points into a latent space, generating features enriched with low-frequency information. These features are then integrated into shallow layers of the network to alleviate training challenges associated with DiT. Second, from a high-frequency perspective, we design a local facial extractor to capture high-frequency details and inject them into transformer blocks, enhancing the model's ability to preserve fine-grained features. We propose a hierarchical training strategy to leverage frequency information for identity preservation, transforming a vanilla pre-trained video generation model into an IPT2V model. Extensive experiments demonstrate that our frequency-aware heuristic scheme provides an optimal control solution for DiT-based models. Thanks to this scheme, our ConsisID generates high-quality, identity-preserving videos, making strides towards more effective IPT2V. Code: https://github.com/PKU-YuanGroup/ConsisID.

• The paper introduces a novel frequency decomposition technique that decouples high- and low-frequency identity features to enhance text-to-video generation.
• It employs a hierarchical training strategy that transitions a pre-trained video model through coarse-to-fine refinement for improved identity consistency.
• Extensive experiments demonstrate superior identity preservation and text relevance performance over previous methods, highlighting scalable application potential.

Identity-Preserving Text-to-Video Generation by Frequency Decomposition: A Formal Analysis

The paper introduces ConsisID, a novel framework for Identity-Preserving Text-to-Video (IPT2V) generation, addressing significant challenges in maintaining human identity consistency across video frames. ConsisID leverages a Diffusion Transformer (DiT)-based model, focusing on frequency decomposition to ensure identity preservation. The framework strategically implements a tuning-free approach, minimizing the need for case-by-case finetuning, a common drawback in existing methodologies.

Core Contributions

1. Innovative Frequency Decomposition Technique: The paper delineates a method that decouples identity features into high-frequency and low-frequency components. High-frequency information captures intrinsic identity details, while low-frequency aspects encapsulate global facial features. This separation is accomplished through a global and a local facial extractor, which subsequently introduce these signals at strategic layers within the DiT architecture.
2. Hierarchical Training Strategy: A hierarchical training approach is formulated to transition a pre-trained video generation model into an IPT2V model. This strategy incorporates coarse-to-fine training, mitigating the training complexity by initially focusing on low-frequency global features before exploring high-frequency details.
3. Enhanced Model Convergence: To address the convergence issues inherent in DiT models compared to U-Net architectures, ConsisID integrates low-frequency information early in the process, facilitating pixel-level predictions crucial for diffusion models. This approach draws from previous findings on frequency analysis, highlighting the limitations of transformers in perceiving high-frequency information.
4. Extensive Experimental Validation: The authors present an extensive array of experiments demonstrating the efficacy of ConsisID. The results indicate superior performance in generating high-quality videos that maintain identity consistency, significantly surpassing prior methods like ID-Animator in identity preservation and text relevance.

Implications and Future Directions

The methodology proposed in ConsisID has profound implications for video generation, particularly in scenarios requiring high fidelity and identity consistency. Practically, it paves the way for applications in virtual reality and personalized content creation, where preserving human likeness is paramount. Theoretically, it sets a precedent for leveraging frequency decomposition in generative models, potentially inspiring further research into frequency domain representations in diffusion transformers.

The success of ConsisID underscores an emergent paradigm in video generation, where the strategic integration of frequency components can enhance the capabilities of existing architectures. Future research could explore extending these techniques to accommodate multi-identity scenarios within a single video or advancing the hierarchical strategies to further refine model generalization.

Moreover, the paper identifies limitations in current evaluation metrics, which do not fully encapsulate model capabilities, pointing towards a need for more comprehensive benchmarks aligned with human perception. Addressing these metrics could significantly improve model assessment, providing a clear pathway for subsequent advancements.

In conclusion, ConsisID represents a significant technical advancement in the IPT2V domain, introducing innovative strategies that could serve as foundational elements for future advancements in generative video models. The tuning-free frequency decomposition approach not only enhances identity preservation but also offers a scalable framework adaptable to broader applications in AI-driven content generation.