Dense Pose Transfer (1809.01995v1)

Abstract: In this work we integrate ideas from surface-based modeling with neural synthesis: we propose a combination of surface-based pose estimation and deep generative models that allows us to perform accurate pose transfer, i.e. synthesize a new image of a person based on a single image of that person and the image of a pose donor. We use a dense pose estimation system that maps pixels from both images to a common surface-based coordinate system, allowing the two images to be brought in correspondence with each other. We inpaint and refine the source image intensities in the surface coordinate system, prior to warping them onto the target pose. These predictions are fused with those of a convolutional predictive module through a neural synthesis module allowing for training the whole pipeline jointly end-to-end, optimizing a combination of adversarial and perceptual losses. We show that dense pose estimation is a substantially more powerful conditioning input than landmark-, or mask-based alternatives, and report systematic improvements over state of the art generators on DeepFashion and MVC datasets.

• The paper introduces a dual-stream framework combining predictive and warping modules to synthesize high-quality human images with detailed pose accuracy.
• It employs a blending module with reconstruction, adversarial, and perceptual losses to achieve superior structural consistency and visual realism.
• Experimental results on DeepFashion and MVC datasets demonstrate enhanced handling of occlusions and a significant improvement over landmark-based methods.

Dense Pose Transfer: A Novel Approach to Neural Pose Synthesis

The research paper "Dense Pose Transfer" by Neverova et al. introduces a novel methodology for the synthesis of human images in new poses using a combination of surface-based pose estimation and deep generative models. This approach aims to accurately perform pose transfer, synthesizing a new image of a person based on a single image and the pose from another image, termed the 'pose donor.' The method leverages the DensePose system to map pixels from both images to a unified surface-based coordinate system, facilitating a rich correspondence between the source and target poses.

Key Contributions and Methodology

The Dense Pose Transfer framework consists of multiple components designed to optimize the pose synthesis process. Central to the approach is the use of a dense surface representation of the human body, which enables more nuanced control over the synthesized image generation when compared to previous methods that relied on sparse landmarks or coarse segmentations.

1. Predictive and Warping Streams: The model comprises two parallel streams—a predictive stream and a warping stream. The predictive module functions as a black-box generative network conditioned on the DensePose outputs, which inform both the input and target. This allows for plausible image generation for familiar poses. Concurrently, the warping stream undertakes a spatial transformation using a UV mapping of the body to warp image observations into a surface-based parameterization. These observations are then inpainted to infer unseen body regions, helping to maintain texture fidelity across varying poses.
2. Blending Module: Outputs from the predictive and warping streams are combined via a blending module, which refines the synthesized images using a combination of reconstruction, adversarial, and perceptual losses. This module merges the robustness of the predictive stream with the detail preservation from the warping stream, resulting in more photorealistic and structurally accurate images.
3. Inpainting and Surface-Based Approach: One of the key innovations is the integration of a surface-based framework. By using DensePose, the approach provides a richer conditioning signal which captures both global and point-level information of the body surface. The inpainting component within the warp stream further allows for better handling of occlusions and missing body parts, benefiting the generation of coherent images even when substantial parts of the source or target body are occluded.

Experimental Results and Implications

The model was evaluated on the DeepFashion and MVC datasets, showing systematic improvements over existing state-of-the-art methods in terms of structural consistency and visual realism. The paper provides extensive quantitative analysis, demonstrating the superiority of dense pose-based conditioning over existing methods such as landmark-based ones. Structural metrics like SSIM and perceptual-based metrics like Inception Score were used to underscore these improvements.

The implications of this research are multifaceted. Practically, it has potential applications in fields such as virtual reality, gaming, and cinematography, where realistic human rendering is essential. Theoretically, it provides a framework for further exploration into surface-based neural representations and their efficacy in generative models. Additionally, this methodology might contribute to data augmentation and synthetic dataset generation for training image recognition systems, alongside the potential use in forgery detection systems by simulating realistic human images.

Future Prospects

Going forward, the paper suggests exploration into enhancing the quality of neural synthesis with more advanced loss functions or network architectures. There is also scope for extending surface-based neural synthesis to other object classes beyond human modeling, thereby widening the applicability of this dense correspondence and surface-based approach.

In summary, Dense Pose Transfer represents a significant step in neural synthesis, offering a method that leverages detailed surface representations to enhance the quality and control of generated human images. By integrating these advanced pose representations, the research provides a robust foundation for more realistic and flexible human image synthesis frameworks.