Single Image Portrait Relighting (1905.00824v1)

Abstract: Lighting plays a central role in conveying the essence and depth of the subject in a portrait photograph. Professional photographers will carefully control the lighting in their studio to manipulate the appearance of their subject, while consumer photographers are usually constrained to the illumination of their environment. Though prior works have explored techniques for relighting an image, their utility is usually limited due to requirements of specialized hardware, multiple images of the subject under controlled or known illuminations, or accurate models of geometry and reflectance. To this end, we present a system for portrait relighting: a neural network that takes as input a single RGB image of a portrait taken with a standard cellphone camera in an unconstrained environment, and from that image produces a relit image of that subject as though it were illuminated according to any provided environment map. Our method is trained on a small database of 18 individuals captured under different directional light sources in a controlled light stage setup consisting of a densely sampled sphere of lights. Our proposed technique produces quantitatively superior results on our dataset's validation set compared to prior works, and produces convincing qualitative relighting results on a dataset of hundreds of real-world cellphone portraits. Because our technique can produce a 640 $\times$ 640 image in only 160 milliseconds, it may enable interactive user-facing photographic applications in the future.

• The paper presents a neural network that relights portraits from a single image, achieving a 75% error reduction and processing 640×640 images in 160ms.
• It employs an encoder-decoder architecture with skip connections and a confidence-weighted average mechanism to model complex facial shading and reflectance.
• The approach leverages a real-world light stage dataset with 304 directional lights across 22 subjects to enhance realism and generalize to cellphone portraits.

Overview of Single Image Portrait Relighting

The paper at hand presents a novel approach to portrait relighting, a computational technique designed to manipulate the lighting in photographs of human subjects. The proposed method addresses the constraints faced by consumer photography, which often lacks access to specialized lighting equipment. The method utilizes a neural network to process a single RGB image of a portrait, captured with a standard cellphone camera in natural, unconstrained lighting, and modifies it to simulate relighting according to any given environmental map.

Methodology and Implementation

The system is founded on a deep neural network architecture, adapted to learn a function capable of relighting images while implicitly modeling complex phenomena in facial appearance, such as shading and reflectance, without explicit geometrical reconstructions. The network employs an encoder-decoder architecture with skip connections, augmented with a confidence-weighted average mechanism to predict the original illumination of the source image.

A standout feature of the approach is the use of a real-world dataset, captured in a light stage environment, which significantly enhances the realism of the relighting results when compared with synthetic data typically used in similar studies. The dataset comprises portraits of 18 training subjects and 4 validation subjects, captured from several viewpoints and illuminated with 304 directional lights. This setup provides a comprehensive directional lighting representation essential for training the neural network.

Results and Performance

Quantitatively, the proposed system delivers superior results compared to previous methodologies, showing a marked reduction in error rates (up to 75%) on the relighting task. Notably, the system manages to produce a $640 \times 640$ relit image in merely 160 milliseconds, highlighting its potential applicability in real-time or interactive environments.

Furthermore, the method successfully generalizes to hundreds of casually captured cellphone portraits, retaining realistic renderings under diverse manipulated lighting conditions. This generalization is largely attributed to the self-supervision loss incorporated during training, which improves both tasks of estimating the illumination of the input image and rendering the relit portrait.

Implications and Future Directions

The approach demonstrates significant potential not only for consumer-facing photographic applications but also for enhancing datasets used in other computer vision tasks such as facial recognition and 3D reconstruction. By training on genuinely captured light stage data, the model better preserves non-Lambertian reflectance subtleties, a common limitation when using synthetic data.

Challenges remain when extending generalizability, such as input images with hard shadows or oversaturated highlights, which the current system handles less effectively. Future improvements could be oriented towards enhancing the model's robustness to these edge cases, potentially through expanded or more varied training datasets.

In conclusion, the paper provides a comprehensive, efficient solution to portrait relighting with promising implications for various applications in computational photography and computer vision. The approach exemplifies how neural networks can be effectively harnessed to achieve complex image manipulation tasks while systematically reducing dependency on labor-intensive pre-existing models or datasets.