MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modeling (2409.16160v2)

Abstract: Character video synthesis aims to produce realistic videos of animatable characters within lifelike scenes. As a fundamental problem in the computer vision and graphics community, 3D works typically require multi-view captures for per-case training, which severely limits their applicability of modeling arbitrary characters in a short time. Recent 2D methods break this limitation via pre-trained diffusion models, but they struggle for pose generality and scene interaction. To this end, we propose MIMO, a novel framework which can not only synthesize character videos with controllable attributes (i.e., character, motion and scene) provided by simple user inputs, but also simultaneously achieve advanced scalability to arbitrary characters, generality to novel 3D motions, and applicability to interactive real-world scenes in a unified framework. The core idea is to encode the 2D video to compact spatial codes, considering the inherent 3D nature of video occurrence. Concretely, we lift the 2D frame pixels into 3D using monocular depth estimators, and decompose the video clip to three spatial components (i.e., main human, underlying scene, and floating occlusion) in hierarchical layers based on the 3D depth. These components are further encoded to canonical identity code, structured motion code and full scene code, which are utilized as control signals of synthesis process. The design of spatial decomposed modeling enables flexible user control, complex motion expression, as well as 3D-aware synthesis for scene interactions. Experimental results demonstrate effectiveness and robustness of the proposed method.

• The paper introduces spatial decomposed modeling that lifts 2D video frames into 3D via monocular depth estimation for controllable synthesis.
• It employs disentangled encoding for human identity, motion, scene, and occlusion, and reconstructs videos through a diffusion-based decoder.
• Experimental results demonstrate MIMO’s ability to animate diverse characters with complex 3D motions and enhanced scene interactivity.

Overview of "MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modeling"

The paper "MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modeling" introduces a novel methodology for synthesizing character videos with controllable attributes via spatial decomposed modeling. This paper presents a framework, referred to as MIMO, that adeptly integrates arbitrary characters, complex 3D motions, and interactive real-world scenes into a unified synthesis process. The primary contribution lies in its ability to encode 2D video input into compact spatial codes, leveraging the inherent 3D characteristics of video data.

Methodology

The core of MIMO's approach is spatial decomposed modeling. This involves lifting 2D frames into 3D space using monocular depth estimation and decomposing video clips into three spatial components: the main human character, the underlying scene, and floating occlusions. Each component is then encoded into distinct latent codes: a canonical identity code, a structured motion code, and a full scene code. These codes provide control signals for the synthesis process within a diffusion-based decoder framework.

1. Hierarchical Spatial Layer Decomposition: Videos are split into human, scene, and occlusion layers based on depth values, enabling precise separation for synthesis.
2. Disentangled Human Encoding: Human components are encoded by disentangling identity and motion using structured body codes, which surpass traditional 2D skeleton representations in handling 3D motion complexity.
3. Scene and Occlusion Encoding: A VAE encoder is employed to derive scene and occlusion codes, ensuring natural interaction between characters and their environments.
4. Composed Decoding: By combining these spatial codes, MIMO reconstructs video clips via a diffusion-based algorithm, ensuring coherent synthesis guided by user inputs.

Results and Implications

The experimental results reveal MIMO's versatility and robustness across various contexts. The framework demonstrates proficiency in animating diverse character types, from realistic to cartoonish, while preserving unique body shapes. It achieves high generality, smoothly handling novel and complex 3D motions both from posed sequences and real-world sources. Moreover, MIMO adeptly synthesizes scenes with interactive object dynamics, showcasing its applicability in character replacement tasks within real-world videos.

The ability to control and synthesize video attributes with simple inputs marks a significant advancement over existing methodologies that struggle with pose generality and scene complexity. By adopting 3D-aware synthesis, MIMO enhances the naturalness of character insertion and interaction within scenes.

Future Directions

MIMO's architectural considerations, especially its focus on spatial decomposition and 3D encoding, could catalyze future research in 3D-aware video synthesis. Future work might explore expansion to broader video synthesis domains, including those involving non-human characters or dynamic environments beyond character-centric narratives. Integrating higher fidelity texture synthesis and real-time processing are possible directions for enhancing the practical applications of this framework.

In conclusion, the MIMO framework represents an intricate yet efficient approach to character video synthesis, addressing limitations of previous models and establishing a foundation for further advancements in controllability and realism in video generation.