Skip-and-Play: Depth-Driven Pose-Preserved Image Generation for Any Objects (2409.02653v1)

Abstract: The emergence of diffusion models has enabled the generation of diverse high-quality images solely from text, prompting subsequent efforts to enhance the controllability of these models. Despite the improvement in controllability, pose control remains limited to specific objects (e.g., humans) or poses (e.g., frontal view) due to the fact that pose is generally controlled via camera parameters (e.g., rotation angle) or keypoints (e.g., eyes, nose). Specifically, camera parameters-conditional pose control models generate unrealistic images depending on the object, owing to the small size of 3D datasets for training. Also, keypoint-based approaches encounter challenges in acquiring reliable keypoints for various objects (e.g., church) or poses (e.g., back view). To address these limitations, we propose depth-based pose control, as depth maps are easily obtainable from a single depth estimation model regardless of objects and poses, unlike camera parameters and keypoints. However, depth-based pose control confronts issues of shape dependency, as depth maps influence not only the pose but also the shape of the generated images. To tackle this issue, we propose Skip-and-Play (SnP), designed via analysis of the impact of three components of depth-conditional ControlNet on the pose and the shape of the generated images. To be specific, based on the analysis, we selectively skip parts of the components to mitigate shape dependency on the depth map while preserving the pose. Through various experiments, we demonstrate the superiority of SnP over baselines and showcase the ability of SnP to generate images of diverse objects and poses. Remarkably, SnP exhibits the ability to generate images even when the objects in the condition (e.g., a horse) and the prompt (e.g., a hedgehog) differ from each other.

• The paper introduces a novel method leveraging depth information to generate images of objects in new poses while preserving original appearance.
• It ensures realistic object geometry and consistent pose generation, setting it apart from traditional RGB-based approaches.
• The approach offers promising implications for improved pose transfer and object representation in advanced computer vision applications.

The paper "Skip-and-Play: Depth-Driven Pose-Preserved Image Generation for Any Object" (2409.02653) proposes a novel method for generating images of objects in new poses while preserving their original appearances. This approach leverages depth information to ensure that the generated images maintain realistic and consistent object geometry.

To provide a richer context around this research, several related papers are particularly relevant:

1. Pose Estimation and Refinement: The paper "BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depth" introduces a method for 3D pose estimation that focuses on RGB images without the need for depth information, which contrasts with the depth-driven approach of "Skip-and-Play" (Rad et al., 2017). Another related work, "RePOSE: Fast 6D Object Pose Refinement via Deep Texture Rendering," focuses on refining object poses through a faster process involving deep texture rendering without requiring zoomed inputs (Iwase et al., 2021).
2. Pose Transfer and Person Image Generation: Several works on person image generation and pose transfer share underlying principles with the discussed paper. "Pose Guided Person Image Generation" and "Deformable GANs for Pose-based Human Image Generation" both address generating new images of people in altered poses by mapping input images to new skeletal structures. These differ by focusing on human figures rather than any object and primarily use adversarial networks (Ma et al., 2017, Siarohin et al., 2017).
3. Depth and Pose Learning: "Towards Better Generalization: Joint Depth-Pose Learning without PoseNet" addresses the learning of depth and pose jointly through disentangling scale from network estimation, which helps in maintaining consistency across varied environments (Zhao et al., 2020). This methodological consideration aligns with the depth-driven aspects of "Skip-and-Play."

The contributions of "Skip-and-Play" provide a significant advancement by combining depth data integration to pose-preserved image generation, potentially offering improvements in generating more consistent and realistic object images across different poses.