Estimating Reflectance Layer from A Single Image: Integrating Reflectance Guidance and Shadow/Specular Aware Learning

Abstract: Estimating the reflectance layer from a single image is a challenging task. It becomes more challenging when the input image contains shadows or specular highlights, which often render an inaccurate estimate of the reflectance layer. Therefore, we propose a two-stage learning method, including reflectance guidance and a Shadow/Specular-Aware (S-Aware) network to tackle the problem. In the first stage, an initial reflectance layer free from shadows and specularities is obtained with the constraint of novel losses that are guided by prior-based shadow-free and specular-free images. To further enforce the reflectance layer to be independent of shadows and specularities in the second-stage refinement, we introduce an S-Aware network that distinguishes the reflectance image from the input image. Our network employs a classifier to categorize shadow/shadow-free, specular/specular-free classes, enabling the activation features to function as attention maps that focus on shadow/specular regions. Our quantitative and qualitative evaluations show that our method outperforms the state-of-the-art methods in the reflectance layer estimation that is free from shadows and specularities. Code is at: \url{https://github.com/jinyeying/S-Aware-network}.

• The paper introduces a novel two-stage framework that integrates reflectance guidance with a shadow/specular-aware network to isolate true reflectance layers.
• It employs innovative shadow-free and specular-free loss functions that significantly improve accuracy over state-of-the-art methods on datasets like IIW, ShapeNet, and MPI-Sintel.
• Results demonstrate robust performance in real-world images, suggesting broad applicability in advanced computer vision and image reconstruction tasks.

Estimating Reflectance Layer from A Single Image: Integrating Reflectance Guidance and Shadow/Specular Aware Learning

Introduction

The estimation of a reflectance layer from a single image represents a fundamental challenge in computer vision, particularly when the image contains shadows or specular highlights. Traditional methods, including those employing deep learning, frequently encounter difficulties in accurately distinguishing between these features and inherent reflectance properties. This paper introduces an innovative two-stage learning framework to address these challenges, comprising reflectance guidance and a Shadow/Specular-Aware (S-Aware) network.

Methodology

The proposed framework consists of two distinct stages. The first stage utilizes reflectance guidance to derive an initial reflectance layer that is free from shadows and specularities. This is accomplished through innovative loss functions, specifically designed to leverage shadow-free and specular-free priors.

Figure 1: The framework's stages: an initial reflectance layer is derived, followed by refinement in the S-Aware network.

The second stage introduces the S-Aware network, which employs a classifier to focus on shadow and specular regions, effectively refining the reflectance layer. The network's ability to differentiate shadow/specular-free classes enhances its capability to disregard these elements from the reflectance estimation.

Results

The results demonstrate significant improvements over state-of-the-art methods in both quantitative and qualitative evaluations. The introduction of novel shadow-free and specular-free losses has shown to be particularly effective in enhancing the accuracy of the reflectance layer.

Figure 2: Comparison with state-of-the-art methods, illustrating the advantage of the proposed method in removing shadows and specularities.

Shadow-Free and Specular-Free Losses

Integral to the first stage of the framework are the shadow-free loss $\mathcal{L}_\mathit{R}^\text{sf}$ and specular-free loss $\mathcal{L}_\mathit{R}^\text{hf}$.

Figure 3: Shadow-free loss applied to the reflectance layer.

Figure 4: Specular-free loss depicted in its application.

These losses are formulated to guide the network in learning to omit shadow and specular highlights, ensuring they do not influence the resulting reflectance estimate. The shadow-free loss employs chromaticity constraints, while the specular-free loss targets saturation uniformity to suppress specular highlights effectively.

Performance and Comparisons

Quantitative analysis was conducted on several datasets, demonstrating the superiority of the proposed method in achieving lower Weighted Human Disagreement Rate (WHDR) on the IIW dataset and enhanced performance metrics on synthetic and real image datasets, including ShapeNet and MPI-Sintel.

Figure 5: Evaluation on the MIT dataset showing robust shadow removal ability.

Figure 6: Performance on real-world highlight datasets affirming specular handling efficiency.

The method's robustness across diverse datasets indicates its applicability to real-world scenarios, where lighting and reflective variabilities pose significant challenges.

Conclusions

This research proposes a novel dual-stage network that significantly advances the state of reflectance layer estimation by accurately differentiating reflectance from shadows and highlights through specialized learning mechanisms. The approach integrates innovative loss functions and a shadow/specular-attuned network, achieving superior performance in suppressing non-reflectance elements from the final estimate.

Future work may extend this framework to accommodate dynamic scenes or integrate with broader image reconstruction tasks, further enhancing real-time application feasibilities in advanced computer vision systems.