AnimeCeleb: Large-Scale Animation CelebHeads Dataset for Head Reenactment (2111.07640v2)

Abstract: We present a novel Animation CelebHeads dataset (AnimeCeleb) to address an animation head reenactment. Different from previous animation head datasets, we utilize 3D animation models as the controllable image samplers, which can provide a large amount of head images with their corresponding detailed pose annotations. To facilitate a data creation process, we build a semi-automatic pipeline leveraging an open 3D computer graphics software with a developed annotation system. After training with the AnimeCeleb, recent head reenactment models produce high-quality animation head reenactment results, which are not achievable with existing datasets. Furthermore, motivated by metaverse application, we propose a novel pose mapping method and architecture to tackle a cross-domain head reenactment task. During inference, a user can easily transfer one's motion to an arbitrary animation head. Experiments demonstrate the usefulness of the AnimeCeleb to train animation head reenactment models, and the superiority of our cross-domain head reenactment model compared to state-of-the-art methods. Our dataset and code are available at https://github.com/kangyeolk/AnimeCeleb.

• The paper introduces AnimeCeleb, a dataset of 2.4M synthetic head images with detailed pose annotations, enhancing training for head reenactment models.
• The paper employs a semi-automated 3D graphics pipeline using Blender, ensuring consistent style and high intra-identity consistency.
• The paper demonstrates improved cross-domain head reenactment performance, with superior FID and SSIM scores over state-of-the-art methods.

AnimeCeleb: A Comprehensive Dataset for Animation Head Reenactment

The paper "AnimeCeleb: Large-Scale Animation CelebHeads Dataset for Head Reenactment" addresses the challenges of creating a dataset tailored for high-quality animation head reenactment. The authors present AnimeCeleb, a large-scale dataset that leverages 3D animation models to generate a vast number of synthetic head images with detailed pose annotations, providing a rich resource for training head reenactment models. This contribution is distinct from existing animation datasets by offering pose annotations, ensuring uniformity in style and content, and facilitating advanced head reenactment methodologies.

Key Contributions

1. Creation of AnimeCeleb Dataset:
   • The authors propose a principled approach for generating animation datasets using 3D animation models as image samplers.
   • A semi-automated pipeline, utilizing 3D graphics software Blender, is developed to efficiently render images and annotate poses.
   • AnimeCeleb comprises 2.4 million images depicting detailed expressions and head rotations, ensuring high intra-identity consistency, which is particularly advantageous for training neural networks for tasks like head reenactment.
2. Pose Annotation and Mapping:
   • A novel pose mapping technique is introduced to bridge the domain gap between human and animation datasets by aligning pose representations through 3D Morphable Models (3DMM).
   • This alignment enables cross-domain transfer, facilitating applications such as animating virtual avatars based on human facial movements.
3. Head Reenactment Models:
   • The dataset facilitates the training of existing head reenactment models, such as PIRenderer and First Order Motion Model (FOMM). When trained on AnimeCeleb, these models demonstrate enhanced performance, retaining vivid character identity during transformation tasks.
   • Introduces the Animation Motion model (AniMo), designed to leverage both the 3DMM-aligned dataset and real-world human datasets, effectively performing cross-domain head reenactment and demonstrating superior performance over state-of-the-art techniques.

Experimental Results

The paper provides extensive experimental validation, showing significant improvements in head reenactment tasks:

• Quantitative Metrics: Compared to models trained on human datasets alone or other animation datasets, AnimeCeleb-trained models show substantial improvements in metrics such as FID and SSIM, indicating higher quality and realism of the generated frames.
• Cross-Domain Performance: Demonstrates that the AniMo architecture exceptionally handles the task of animating characters using human inputs, bridging the stylistic differences without noticeable artifacts or identity leakage.
• Expressive Diversity: Numerical results emphasize the dataset's capacity to train models that capture a diverse range of facial expressions and head movements well beyond what existing datasets provide.

Practical and Theoretical Implications

AnimeCeleb propels the field of animation and virtual reality by enhancing the development cycle of animation applications, reducing reliance on manually curated comic or film datasets. From a theoretical perspective, the dataset and model provide a framework for future research, particularly in areas requiring detailed pose annotations and high fidelity in animated character rendering.

Future Prospects

The authors discuss expanding the dataset's resolution capabilities and diversifying lighting conditions and viewpoint variations, which would enrich its application scope. The future development of models building on the AniMo architecture could further refine cross-domain reenactment tasks, enhancing performance in varying ambient and compositional contexts. These advancements would beneficially impact industries relying on animated content, such as gaming, film, and virtual reality.

In conclusion, AnimeCeleb presents a robust stepping stone for evolving digital animation research, providing a direction for future advancements that simplify complex creative processes in animation and virtual reality technologies.