---
title: Shadow-Independent Structure Extraction Network
url: https://www.emergentmind.com/topics/shadow-independent-structure-extraction-network
type: topic
---

# Shadow-Independent Structure Extraction Network

Searching arXiv for the cited papers to ground the article in current records.
A **Shadow-Independent Structure Extraction Network** denotes a neural module that predicts a structure representation that preserves intrinsic scene or facial geometry while suppressing boundaries caused by illumination, especially cast shadows. In the most explicit usage of the term, the 2025 portrait shadow removal framework introduces **SE-Net** as a network that generates a **shadow-independent structure map** for diffusion-based inpainting, so that the downstream generator is guided by facial structure rather than by shadow edges [2507.04692]. Closely related formulations appear in earlier shadow analysis and shadow removal work: **FCSD-Net** with **Restore to Detect (R2D)** emphasizes fine local structure and restoration-guided features for shadow mask prediction [2109.09609], while **StructNet** and **MStructNet** reconstruct a **shadow-free structure** and use it as a prior for image restoration [2301.03182]. Taken together, these works define a technical lineage in which shadow handling is shifted from purely appearance-based mapping toward structure-centered representation learning.

## 1. Conceptual definition and problem setting

The central problem is that shadows introduce strong illumination discontinuities that often resemble real edges. For shadow removal, if those discontinuities are interpreted as true structure, a model may preserve them, producing **shadow residual**, **color distortion**, **structure blurring**, or **loss of details** in the reconstructed output [2507.04692]. For shadow detection, the corresponding failure mode is that small, unclear, or blurry shadow regions are missed, or that background texture and appearance cues are mistaken for shadows [2109.09609].

Within this context, a shadow-independent structure extractor aims to produce a representation that satisfies two conditions. First, it must **preserve structure**: in portrait settings this includes facial geometry and identity cues such as eyebrows, nose contours, lips, eye shape, and hairline; in general shadow analysis it includes edges, boundaries, object shapes, smooth color regions, and other low-level structures [2507.04692; 2301.03182]. Second, it must **exclude shadow boundaries** or otherwise reduce dependence on shadow appearance, so that downstream processing is not encouraged to reproduce illumination artifacts as content [2507.04692].

A common misconception is that “shadow-independent” means the network ignores shadows entirely. The cited work does not support that interpretation. In the 2021 detector, the goal is not to ignore shadows but to learn **structure that is robust to shadow appearance**, using fine-context architecture and restoration-guided features [2109.09609]. In the 2023 removal framework, extraction is explicitly **shadow-aware** because it uses masks and directional non-shadow-to-shadow transfer to construct a shadow-free structure prior [2301.03182]. This suggests that independence is best understood as **independence from illumination-induced boundaries as structure**, not independence from the shadow-removal or shadow-detection objective itself.

## 2. Antecedents in shadow detection: fine-context structure extraction

An important precursor to shadow-independent structure extraction is the 2021 **Fine Context-aware Shadow Detection Network (FCSD-Net)**, trained with the **Restore to Detect (R2D)** strategy [2109.09609]. The paper’s core idea is to make shadow detection more **structure-extracting** and less dependent on global appearance cues by combining two complementary mechanisms: explicit fine-context detection and restoration-guided auxiliary learning.

FCSD-Net is an end-to-end shadow segmentation network with a **ResNeXt** backbone. Features are taken from `conv1`, `res2c`, `res3b3`, `res4b22`, and `res5c`, denoted \(L1, L2, L3, L4, L5\). These are divided into two branches. The **Coarse Context Detector (CCD)** receives all backbone features \(L1\)–\(L5\), converts them into \(F1, F2, F3, F4, F5\), and applies a **Distraction-Aware Shadow (DS) block** at each scale to produce \(DS1, DS2, DS3, DS4, DS5\). The **Fine Context Detector (FCD)** receives only the low-level feature \(L1\), with the explicit purpose of preserving fine spatial detail by controlling receptive-field growth [2109.09609].

The receptive-field argument is central to the model’s structure sensitivity. In a typical undercomplete encoder with pooling, the receptive field grows as
\[
RF(w.r.t\ I) = 2^{2(i-1)} \times k \times k,
\]
whereas in the proposed overcomplete or fine-context design, bilinear upsampling is used instead of pooling, and receptive-field growth is constrained as
\[
RF(w.r.t\ I) = \left(\frac{1}{2}\right)^{2(i-1)} \times k \times k.
\]
The stated consequence is improved retention of **edges, boundaries, and low-level structures**, which is crucial for small, blurry, or unclear shadow regions [2109.09609].

FCD consists of **4 conv blocks**, each followed by **bilinear upsampling** and ReLU. The paper reports that the feature resolution saturates at about \(400 \times 400\), and increasing beyond 4 blocks does not improve BER meaningfully. FCD outputs \(F6\), which is further processed by a DS block to produce \(DS6\). The final detector interpolates \(F1,\dots,F6\) and \(DS1,\dots,DS6\) to image size, concatenates them, applies a \(1 \times 1\) convolution, and then a sigmoid to obtain the final binary shadow mask [2109.09609].

The associated **R2D** strategy provides the complementary mechanism. If the input image is \(I\), the detector output is \(d = D(I)\) and the remover output is \(r = R(I)\), with the removal objective conceptualized as
\[
d = D(I), \qquad r = R(I),
\]
and described as
\[
r = I - d.
\]
The paper explicitly notes that this is **not** used as a literal residual-based detection method, because subtracting restored output from the input yields a noisy residual that does not delineate the shadow mask well. Instead, the key observation is that a restoration network still learns internal representations that encode where shadows are, even when trained for image-to-image restoration. These internal features become complementary shadow-aware cues for the detector [2109.09609].

This architecture does not employ the exact phrase “shadow-independent structure extraction network,” but it establishes a closely related principle: a detector can be made less reliant on global darkness or broad context by emphasizing local structure and by importing restoration-derived features that implicitly encode shadow extent, boundaries, and transitions.

## 3. Structure-level shadow removal: StructNet and MStructNet

The 2023 **Structure-Informed Shadow Removal Networks** move the idea from detection into explicit structure reconstruction [2301.03182]. The paper argues that shadows mainly degrade images at the **image-structure level**, defined as the low-frequency component that captures object shapes, smooth color regions, and region homogeneity. Shadow remnants are said to be especially problematic because they often occur in homogeneous regions with low-intensity values and are therefore difficult to eliminate in a conventional image-to-image mapping framework.

On this basis, **StructNet** is formulated as a **two-stage** system. In the first stage, the network performs **structure-level shadow removal**: it takes shadow image \(\mathbf{I}\), extracted structure \(\mathbf{S}_l\), and shadow mask \(\mathbf{M}\), and outputs restored shadow-free structure \(\hat{\mathbf{S}_l}\). In the second stage, it performs **image-level shadow removal**: it takes \(\mathbf{I}\), \(\hat{\mathbf{S}_l}\), and \(\mathbf{M}\), and predicts the final shadow-free image \(\hat{\mathbf{I}_l}\) [2301.03182]. The core claim is that if the model first reconstructs a **shadow-free structure**, the subsequent image restoration becomes easier and yields fewer remnants.

The first stage contains three branches: a whole-scene feature extraction or structure prediction branch, a shift estimation branch, and a mask propagation branch. The distinctive module is **Mask-Guided Shadow-Free Extraction (MSFE)**, which replaces standard convolution with a mask-guided directional bridge. The generalized update is
\[
\mathbf{X}_\text{out}^j = \text{Fusion}(\mathbf{X}_\text{in}^j*\mathbf{W}^j, \mathbf{B}^j),
\]
and the shift tensor is recursively computed by
\[
\mathbf{B}^j = \text{Bridge}(\mathbf{X}_\text{in}^j, \mathbf{B}^{j-1},\mathbf{M}_\text{in}^j).
\]
The bridge itself is
\[
\mathbf{B}^j[\mathbf{p}] = \alpha_\mathbf{p}\sum_{\mathbf{q}\in\mathcal{N}_\mathbf{p}}\mathbf{B}^{j-1}[\mathbf{q}](1-\mathbf{M}_\text{in}^j[\mathbf{q}])\mathbf{W}^j_\text{B}[\mathbf{q-p}],
\]
with normalization
\[
\alpha_\mathbf{p} = \frac{1}{\sum_{\mathbf{q}\in\mathcal{N}_\mathbf{p}}\mathbf{M}_\text{in}^j[\mathbf{q}] }.
\]
Because the factor \((1-\mathbf{M}_\text{in}^j[\mathbf{q}])\) selects non-shadow locations, the transfer is explicitly **directional**, from non-shadow to shadow. The bridge weights are dynamically generated via
\[
\mathbf{W}^j_\text{B}= \eta(\mathbf{X}_\text{in}^j),
\]
which distinguishes the method from partial convolution [2301.03182].

The companion module is **Multi-Scale Feature & Residual Aggregation (MFRA)**, which adaptively fuses global convolutional perception with the shift tensor. MFRA computes
\[
\mathbf{X}_{s}^{j} = \sigma([\mathbf{X}_\text{in}^j*\mathbf{W}^j,\mathbf{B}^j]*\mathbf{D}_{s}^{j}),
\]
for scales \(s \in \mathbb{S}=\{1,24,12,6\}\), and then dynamically combines them as
\[
\mathbf{X}_\text{out}^{j} =  \sum_s^S \mathbf{w}_s^j\odot\mathbf{X}_s^j, \qquad \{\mathbf{w}_s^j|s\in\mathds{S}\} = \Phi([\mathbf{X}_\text{in}^j*\mathbf{W}^j,\mathbf{B}^j]).
\]
This is presented as a consistency-regularizing fusion rule that adapts to layer depth and local structure scale [2301.03182].

The paper also proposes **MStructNet**, a compact alternative that avoids explicit prediction of a final shadow-free structure image. Instead, it uses multi-level structures directly at the feature level through a structure-aware encoder, a fusion-oriented encoder, and a decoder. The structure levels are
\[
l \in \mathcal{L}=\{0.005, 0.015, 0.045, 0.1\}.
\]
Each level is assigned its own block, following the same MSFE + MFRA design [2301.03182].

Conceptually, StructNet and MStructNet are highly aligned with the notion of a shadow-independent structure extractor. The paper itself notes that such a label would be a reasonable conceptual paraphrase, while also emphasizing a more precise framing: the structure is not merely extracted but **reconstructed in a shadow-free form and then leveraged to improve image-level removal** [2301.03182].

## 4. Explicit formulation of SE-Net in portrait shadow removal

The most direct realization of the topic is the 2025 **Structure-Guided Diffusion Models for High-Fidelity Portrait Shadow Removal**, which introduces a dedicated **shadow-independent structure extraction network (SE-Net)** as the first stage of the pipeline [2507.04692]. The motivation is tied to portrait editing, where shadow boundaries often coincide with strong illumination discontinuities and ordinary edge or structure predictors “will inevitably include shadow boundaries.” For portrait shadow removal, those boundaries are regarded as artifacts of lighting rather than semantic structure.

SE-Net is therefore trained to output a **shadow-independent structure map** that preserves facial geometry and identity cues while excluding unwanted shadow boundaries. The architecture is described as an **encoder-decoder network** with a **flat convolution** at the input, **two downsampling convolution blocks**, **nine residual blocks** in the bottleneck, **two upsampling convolution blocks**, and a **final convolution layer** [2507.04692]. The output is a structure image in the style of portrait drawing or structure extraction, aligned to the domain of the pre-trained **PDG** model used for supervision.

The training strategy relies on synthetic illumination discontinuities constructed from the **CelebA** dataset. Starting from a well-lit portrait \(I\), the method applies a physics-based relighting method to obtain \(I_{\text{relit}}\), and then combines the original and relit images with a random facial mask \(M\):
\[
I_{syn}=M\odot I_{relit}+(1-M)\odot I,
\]
where \(\odot\) denotes element-wise multiplication. This synthesis is intended to teach SE-Net to ignore lighting changes and recover a structure map that is invariant to illumination discontinuities [2507.04692]. The paper explicitly contrasts this with the portrait shadow synthesis strategy from PSM, stating that the earlier strategy relies on simpler brightness, saturation, and subsurface-scattering adjustments and is not sufficient to model complex facial lighting.

SE-Net is supervised by a pseudo ground-truth structure map from the pre-trained **PDG** structure extractor. Letting \(G_s\) denote SE-Net and \(G_p\) denote PDG, the losses are
\[
\left\{ \begin{aligned}
&\mathcal{L}_{rec} = \Vert{\mathit{G}_s(I_{syn}) - G_p(I)}\Vert_1, \\
&\mathcal{L}_{perceptual} = \mathcal{L}_{LPIPS}(\mathit{G}_s(I_{syn}), G_p(I)),
\end{aligned} \right.
\]
along with an adversarial loss \(\mathcal{L}_{GAN}\). The full objective is
\[
\mathcal{L}_{total} = \mathcal{L}_{rec} + \lambda_1\mathcal{L}_{perceptual} + \lambda_2\mathcal{L}_{GAN}.
\]
The text states \(\lambda_1 = 0.5\) and \(\lambda_1 = 0.25\), which it also describes as a clear typo and likely intended to mean \(\lambda_1=0.5\), \(\lambda_2=0.25\) [2507.04692].

The training protocol uses **20,000 synthesized portrait images**, **3 epochs**, **batch size 8**, and learning rate **\(1.5 \times 10^{-5}\)** [2507.04692]. The resulting map \(S\) is then used as conditioning input for the downstream diffusion model, not merely as auxiliary supervision but as a direct structural prior for generation.

The paper’s stated rationale makes the role of SE-Net precise. By being trained on relighted images with synthetic illumination discontinuities and supervised by structure outputs from the original well-lit image, SE-Net learns a representation that is **insensitive to lighting changes**. This prevents the diffusion model from tracing shadow edges as if they were facial structure, while still preserving the main geometric features required to maintain identity [2507.04692].

## 5. Conditioning role in downstream restoration and generation

Across the cited works, the extracted or reconstructed structure is not an endpoint; it is a conditioning signal that guides a second-stage restoration model.

In the 2025 portrait framework, SE-Net outputs a structure map \(S\) that conditions a **structure-guided portrait inpainting diffusion model** [2507.04692]. The overall pipeline is: input portrait plus shadow mask; SE-Net predicts \(S\); then a diffusion-based inpainting model reconstructs the shadowed region conditioned on the masked input image \(I_M\), the structure map \(S\), the mask \(M\), and the timestep \(t\). The DDPM forward process is
\[
x_t = \sqrt{\bar{\alpha}_t}x_0 + \sqrt{1-\bar{\alpha}_t}\epsilon,
\]
with \(\epsilon \sim \mathcal{N}(0,I)\), \(\bar{\alpha}_t = \prod_{i=0}^t \alpha_i\), and \(\alpha_i = 1-\beta_i\). The reverse process is conditioned as
\[
\mathbf{e}_{t} = \epsilon_\theta(x_t, I_M, S, M, t),
\]
and the training objective is
\[
\mathcal{L}_{diff}(\epsilon) = \Vert \epsilon - \epsilon_\theta(x_t, I_{M}, S, M, t) \Vert^2_2.
\]
Under this formulation, the structure map directly shapes the denoising trajectory by providing a facial structure prior [2507.04692].

The full portrait system adds a final **gradient-guided detail restoration** stage because SE-Net is designed for structure rather than the finest texture. The paper notes that details such as eyelashes, moles, and spots may not be fully captured by the structure map, so gradients inside shadow regions are used as guidance for a detail restoration diffusion model [2507.04692]. This makes the division of labor explicit: SE-Net supplies the global face scaffold; the later stage restores micro-appearance details.

In the 2023 structure-informed shadow removal framework, the same broad principle appears in a non-diffusion setting. After obtaining a corrected structure image \(\hat{\mathbf{S}_l}\), the second stage performs image restoration via
\[
\hat{\mathbf{I}_{l}=\psi_{l}(\mathbf{I}, \hat{\mathbf{S}_l}, \mathbf{M}).
\]
The restored structure acts as a **shadow-free guide** for image-level shadow removal, with the paper reporting fewer color artifacts and better restoration of object colors in shadowed regions when such guidance is used [2301.03182].

The 2021 shadow detector is not a shadow remover, but it also uses an auxiliary restoration branch as a source of structural information. The restoration network \(R()\) is a lightweight **U-Net** with **5 encoder blocks**, **5 decoder blocks**, and skip connections. Rather than forwarding all restoration features, the method selects only \(R1\) from the second encoder layer and \(R2\) from the fifth encoder layer, providing a mixture of local and global semantic information [2109.09609]. Through the **Complementary Feature Learning (CFL)** block, these become \(C1\) and \(C2\), which are injected into FCSD-Net as
\[
C1 \rightarrow L2,\qquad C2 \rightarrow L5.
\]
This suggests a consistent pattern across tasks: structure-like or shadow-aware features are most useful when aligned to appropriate semantic levels rather than naively concatenated.

## 6. Optimization, empirical evidence, and interpretive boundaries

The optimization objectives differ across the three formulations but share a common aim: to bias representation learning toward shadow-invariant or shadow-free structure.

For FCSD-Net and R2D, the detector uses a weighted BCE
\[
\mathcal{L}_W = -\left( \sum_{i} \left(a_n y_i \log(x_i) + b_n (1-y_i)\log(1-x_i)\right)\right),
\]
with
\[
a_n = \frac{N_n}{N_n + N_p}, \qquad b_n = \frac{N_p}{N_n + N_p},
\]
and a distraction-aware term
\[
\mathcal{L}_{DS} = -\left( \sum_i \left(a_n y_i^{fnd} y_i \log(x_i) + b_n y_i^{fpd}(1-y_i)\log(1-x_i)\right) \right).
\]
The shadow loss is
\[
\mathcal{L}_{shadow} = \mathcal{L}_W + \mathcal{L}_W \mathcal{L}_{DS},
\]
the detection loss is
\[
\mathcal{L}_{det} = \sum_k \alpha \mathcal{L}_{shadow}^k + \beta \mathcal{L}_{FP}^k + \gamma \mathcal{L}_{FN}^k,
\]
with \(\alpha = 1,\ \beta = 2,\ \gamma = 2\), the restoration pretraining uses
\[
\mathcal{L}_{res} = \sum_i (y_i - p_i)^2,
\]
and the final joint objective is
\[
\mathcal{L}_{R2D} = \mathcal{L}_{det} + \mathcal{L}_{res}.
\]
The method is pretrained on **ISTD** for **500 epochs** with images resized to **\(320 \times 320\)**, batch size **16**, learning rate **0.001**, **SGD**, momentum **0.9**, weight decay **0.0001**, and an **ImageNet pretrained** ResNeXt-101 backbone. Fine-tuning is performed for **6000 iterations** with random flipping, and **stochastic weight averaging** is used over checkpoints at 4000, 5000, and 6000 iterations [2109.09609]. On **UCF**, **SBU**, and **ISTD**, the method reports best average BERs of **6.96**, **3.15**, and **1.69**, respectively, with mean **3.93**. Compared with DSDNet, the reported improvements are **8.30%** on UCF, **8.69%** on SBU, and **22.11%** on ISTD [2109.09609].

For StructNet and MStructNet, the training loss is
\[
L(\hat{\mathbf{I}},\mathbf{I}^*) = \lambda_{1}L_1(\hat{\mathbf{I}},\mathbf{I}^*) + \lambda_{2}L_\text{perc}(\hat{\mathbf{I}},\mathbf{I}^*),
\]
with \(\lambda_1=1\) and \(\lambda_2=0.1\), and
\[
L_{\text {perc}}(\hat{\mathbf{I}},\mathbf{I}^*)=\sum_{i=1}^3\|\text{VGG16}_{i}(\hat{\mathbf{I}})-\text{VGG16}_{i}(\mathbf{I}^{*})\|_{1}.
\]
At the structure level, the empirical study also uses
\[
L_1(\hat{\mathbf{S}_l},\mathbf{S}_l^*)=\|\hat{\mathbf{S}_l}-\mathbf{S}_l^*\|_1.
\]
The paper reports that as \(l\) increases, shadow-region RMSE decreases continuously, though too large \(l\) can cause shadow spreading into nearby non-shadow areas, and \(l=0.015\) is chosen as the best overall tradeoff [2301.03182]. On ISTD+, a two-stage pipeline with \(l=0.015\) improves shadow RMSE from **6.33** to **5.89** and all-region RMSE from **3.36** to **3.05** relative to direct single-stage image-level shadow removal. On benchmark datasets, **StructNet** and **MStructNet** outperform several baselines, while **MStructNet** achieves lower computation: **20.62 MB**, **28.77 G FLOPs**, **2.8 ms**, versus **67.06 MB**, **45.95 G FLOPs**, **3.3 ms** for StructNet [2301.03182].

For SE-Net, the most direct empirical evidence comes from ablation within the portrait diffusion system. In Table 1, **“Ours w/o structure guidance”** gives **SSIM 0.757**, **LPIPS 0.101**, **RMSE 23.598**, whereas the full method gives **SSIM 0.830**, **LPIPS 0.056**, **RMSE 17.162** [2507.04692]. Using **PDG predicted structure** yields **SSIM 0.779**, **LPIPS 0.092**, **RMSE 19.963**, and training with **data synthesis in PSM** gives **SSIM 0.785**, **LPIPS 0.086**, **RMSE 19.797**, both inferior to the full SE-Net configuration [2507.04692]. These numbers support two separate claims: structure guidance itself improves fidelity, and the **quality of the structure map** materially affects performance.

A further interpretive boundary is worth noting. In the portrait diffusion work, SE-Net is trained to generate a structure map “in the style of portrait drawing / structure extraction” under PDG supervision [2507.04692]. In StructNet, structure is defined in terms of the image-structure layer and multi-level structure extraction [2301.03182]. In FCSD-Net, the emphasis is on low-level boundaries and receptive-field control rather than on explicit structure images [2109.09609]. This indicates that “structure” is not represented identically across papers. A plausible implication is that shadow-independent structure extraction is better viewed as a **family of invariance-inducing designs** rather than a single canonical representation.

## 7. Scope, significance, and recurring design principles

Across the three works, several recurring principles define the technical meaning of shadow-independent structure extraction.

First, **structure is treated as a more stable carrier of content than raw appearance**. In StructNet, the structure layer captures object shapes and continuous colors and is less affected by the remnant problem than direct image-to-image translation [2301.03182]. In SE-Net, the structure map acts as a facial scaffold for diffusion inpainting and is explicitly intended to preserve identity while excluding shadow edges [2507.04692]. In FCSD-Net, constrained receptive fields and low-level feature emphasis are used to retain boundary structure that broad contextual encoders may overlook [2109.09609].

Second, **illumination-induced edges must be suppressed, not reinforced**. The 2025 portrait paper frames this most directly: if shadow boundaries are fed into the downstream diffusion model, the model is encouraged to reconstruct them rather than remove them [2507.04692]. The 2023 paper operationalizes suppression through mask-guided directional transfer from non-shadow to shadow regions [2301.03182]. The 2021 detector reaches a related goal by learning restoration-guided features that encode shadow extent and ambiguity without relying on direct residual differencing [2109.09609].

Third, **shadow-independent does not preclude shadow-aware mechanics**. The extraction modules are frequently driven by masks, auxiliary restoration streams, or synthesized relighting discontinuities. This is especially clear in MSFE, which is mask-guided and directional [2301.03182], and in R2D, where feature maps still flow from the pretrained remover \(R()\) to the detector \(D()\) even when clean images are unavailable during fine-tuning on SBU and UCF [2109.09609]. The resulting independence is therefore representational rather than procedural.

Fourth, **structure priors are most effective when integrated into a broader restoration or generation system**. SE-Net is foundational but paired with structure-guided diffusion inpainting and gradient-guided detail restoration [2507.04692]. StructNet reconstructs a shadow-free structure prior and then hands it to a second-stage image restorer [2301.03182]. FCSD-Net uses restoration features through the CFL block rather than treating removal as a standalone endpoint [2109.09609]. This suggests that the principal role of shadow-independent structure extraction is to provide an intermediate representation that regularizes downstream reasoning.

In that sense, the term designates both a specific 2025 network architecture and a broader methodological direction. It refers to networks that learn a representation of scene or facial structure that is **robust to shadow appearance**, **selectively excludes illumination boundaries**, and **serves as a prior for subsequent detection, restoration, or generative inpainting** [2507.04692; 2301.03182; 2109.09609].

Source: https://www.emergentmind.com/topics/shadow-independent-structure-extraction-network