Sakuga-42M Dataset: Scaling Up Cartoon Research (2405.07425v1)

Abstract: Hand-drawn cartoon animation employs sketches and flat-color segments to create the illusion of motion. While recent advancements like CLIP, SVD, and Sora show impressive results in understanding and generating natural video by scaling large models with extensive datasets, they are not as effective for cartoons. Through our empirical experiments, we argue that this ineffectiveness stems from a notable bias in hand-drawn cartoons that diverges from the distribution of natural videos. Can we harness the success of the scaling paradigm to benefit cartoon research? Unfortunately, until now, there has not been a sizable cartoon dataset available for exploration. In this research, we propose the Sakuga-42M Dataset, the first large-scale cartoon animation dataset. Sakuga-42M comprises 42 million keyframes covering various artistic styles, regions, and years, with comprehensive semantic annotations including video-text description pairs, anime tags, content taxonomies, etc. We pioneer the benefits of such a large-scale cartoon dataset on comprehension and generation tasks by finetuning contemporary foundation models like Video CLIP, Video Mamba, and SVD, achieving outstanding performance on cartoon-related tasks. Our motivation is to introduce large-scaling to cartoon research and foster generalization and robustness in future cartoon applications. Dataset, Code, and Pretrained Models will be publicly available.

• The paper introduces a 42M keyframe dataset that addresses the challenge of limited cartoon data for training advanced animation models.
• The methodology employs an automated pipeline for collection, keyframe detection, and detailed captioning to ensure data quality and adaptability.
• The paper demonstrates marked improvements in text-to-video retrieval and animation generation metrics when fine-tuning vision-language models on Sakuga-42M.

Unveiling Sakuga-42M: Bridging the Gap in Cartoon Research with Large-Scale Data

Opening the Doors to Cartoon Animation Research

Hand-drawn cartoons have been enchanting us for over a century, from early animations like "Humorous Phases of Funny Faces" to the latest anime styles. Yet, creating these beautiful animations involves a lot of manual and repetitive work. Imagine the tasks—storyboarding, sketching, inbetweening, and coloring—each demanding immense effort and time. Even with advancements in computer vision, automating these processes across various styles has proven difficult.

Recent large-scale models like CLIP, Stable Video Diffusion (SVD), and Sora have demonstrated impressive capabilities in understanding and generating natural videos. However, they fall short when applied to the cartoon domain due to the distinct nature of hand-drawn animations. This discrepancy stems from the lack of a sizable, quality cartoon dataset for training these models. Enter Sakuga-42M: a game-changing dataset with 42 million keyframes, designed specifically to cater to cartoon animation research.

The Sakuga-42M Dataset: A Deep Dive

A Rich, Diverse Resource

Sakuga-42M is a treasure trove for researchers, encompassing a wide range of artistic styles, regions, and time periods. The dataset includes videos with comprehensive annotations, such as video-text pairs, anime tags, and content taxonomies. This richness paves the way for more generalized and robust cartoon research.

The dataset preparation involves an automatic pipeline for collecting, splitting, keyframe detection, and captioning. This modular approach ensures adaptability and longevity, allowing the integration of new tools as they become available.

Numerical Scale and Quality

The numbers are staggering: 42 million keyframes from 1.4 million video clips. These keyframes come from various animation stages—rough drawings, tie-downs, finished products—and cover different resolutions and quality metrics. For example, the majority of clips rate high in aesthetic value and dynamic movement.

Captions and Annotations

One standout feature is the detailed captioning process. The pipeline uses specialized anime tagging models and LLMs to generate rich and coherent text descriptions. This additional layer of semantic depth boosts the effectiveness of vision-LLMs for cartoon data.

Applying Sakuga-42M

Sakuga-42M demonstrates substantial improvements in both understanding and generating cartoons.

Understanding Tasks

To gauge the dataset's utility, the researchers fine-tuned existing vision-LLMs (like Video CLIP and Video Mamba). These finetuned models showed remarkable improvements in zero-shot text-to-video and video-to-text retrieval tasks. For instance, the VideoMamba model showed a nearly tripled accuracy in cross-referencing text and video components after being trained on Sakuga-42M.

Generating Cartoons

When it comes to generating cartoons, the improvements are equally impressive. Fine-tuning the SVD model on Sakuga-42M resulted in better stability, dynamics, and overall quality in generated animations. Metrics such as Inception Score (IS), Fréchet Video Distance (FVD), and frame-wise CLIP similarity showed significant improvements, underscoring the practical benefits of this dataset.

Broad Implications and Future Potential

The impact of Sakuga-42M extends far beyond immediate applications in understanding and generating cartoons. Here are a few broader implications and future possibilities:

• Cartoon Generation: Seamless creation of new cartoon videos based on text prompts or user input.
• Automatic Colorization: Enhanced colorization algorithms using extensive supervised image pairs from Sakuga-42M.
• Automatic Inbetweening: Improved automation in generating inbetween frames, thanks to the rich dataset of rough sketches and keyframes.
• Video Retrieval Systems: More effective search systems enabling animators to find specific references effortlessly.
• Cartoon Understanding: Higher accuracy in captioning, scene interpretation, and dialog systems for animations, leveraging the domain-specific richness of Sakuga-42M.
• Automatic Editing: Refined tools for editing and adjusting cartoon animations with greater flexibility and accuracy.

Conclusion

The Sakuga-42M dataset serves as a significant leap forward in cartoon research. Its extensive collection of keyframes, rich annotations, and comprehensive captions provide an invaluable resource for numerous applications. As researchers continue to explore and expand upon this dataset, the future of automated cartoon animation looks brighter than ever. Whether you're involved in developing new AI models or creating stunning animations, Sakuga-42M is set to be an essential tool in your toolbox.