Shadow-Independent Structure Extraction Network

• The paper introduces SE-Net, a neural module that predicts a shadow-independent structure map to guide downstream restoration.
• It employs an encoder-decoder architecture with residual blocks and mask-guided training to decouple facial geometry from shadow artifacts.
• Empirical evaluations demonstrate significant improvements in restoration fidelity, as measured by SSIM, LPIPS, and RMSE metrics.

Searching arXiv for the cited papers to ground the article in current records. A Shadow-Independent Structure Extraction Network denotes a neural module that predicts a structure representation that preserves intrinsic scene or facial geometry while suppressing boundaries caused by illumination, especially cast shadows. In the most explicit usage of the term, the 2025 portrait shadow removal framework introduces SE-Net as a network that generates a shadow-independent structure map for diffusion-based inpainting, so that the downstream generator is guided by facial structure rather than by shadow edges (Yu et al., 7 Jul 2025). Closely related formulations appear in earlier shadow analysis and shadow removal work: FCSD-Net with Restore to Detect (R2D) emphasizes fine local structure and restoration-guided features for shadow mask prediction (Valanarasu et al., 2021), while StructNet and MStructNet reconstruct a shadow-free structure and use it as a prior for image restoration (Liu et al., 2023). Taken together, these works define a technical lineage in which shadow handling is shifted from purely appearance-based mapping toward structure-centered representation learning.

1. Conceptual definition and problem setting

The central problem is that shadows introduce strong illumination discontinuities that often resemble real edges. For shadow removal, if those discontinuities are interpreted as true structure, a model may preserve them, producing shadow residual, color distortion, structure blurring, or loss of details in the reconstructed output (Yu et al., 7 Jul 2025). For shadow detection, the corresponding failure mode is that small, unclear, or blurry shadow regions are missed, or that background texture and appearance cues are mistaken for shadows (Valanarasu et al., 2021).

Within this context, a shadow-independent structure extractor aims to produce a representation that satisfies two conditions. First, it must preserve structure: in portrait settings this includes facial geometry and identity cues such as eyebrows, nose contours, lips, eye shape, and hairline; in general shadow analysis it includes edges, boundaries, object shapes, smooth color regions, and other low-level structures (Yu et al., 7 Jul 2025, Liu et al., 2023). Second, it must exclude shadow boundaries or otherwise reduce dependence on shadow appearance, so that downstream processing is not encouraged to reproduce illumination artifacts as content (Yu et al., 7 Jul 2025).

A common misconception is that “shadow-independent” means the network ignores shadows entirely. The cited work does not support that interpretation. In the 2021 detector, the goal is not to ignore shadows but to learn structure that is robust to shadow appearance, using fine-context architecture and restoration-guided features (Valanarasu et al., 2021). In the 2023 removal framework, extraction is explicitly shadow-aware because it uses masks and directional non-shadow-to-shadow transfer to construct a shadow-free structure prior (Liu et al., 2023). This suggests that independence is best understood as independence from illumination-induced boundaries as structure, not independence from the shadow-removal or shadow-detection objective itself.

2. Antecedents in shadow detection: fine-context structure extraction

An important precursor to shadow-independent structure extraction is the 2021 Fine Context-aware Shadow Detection Network (FCSD-Net), trained with the Restore to Detect (R2D) strategy (Valanarasu et al., 2021). The paper’s core idea is to make shadow detection more structure-extracting and less dependent on global appearance cues by combining two complementary mechanisms: explicit fine-context detection and restoration-guided auxiliary learning.

FCSD-Net is an end-to-end shadow segmentation network with a ResNeXt backbone. Features are taken from conv1, res2c, res3b3, res4b22, and res5c, denoted $L1, L2, L3, L4, L5$. These are divided into two branches. The Coarse Context Detector (CCD) receives all backbone features $L1$–$L5$, converts them into $F1, F2, F3, F4, F5$, and applies a Distraction-Aware Shadow (DS) block at each scale to produce $DS1, DS2, DS3, DS4, DS5$. The Fine Context Detector (FCD) receives only the low-level feature $L1$, with the explicit purpose of preserving fine spatial detail by controlling receptive-field growth (Valanarasu et al., 2021).

The receptive-field argument is central to the model’s structure sensitivity. In a typical undercomplete encoder with pooling, the receptive field grows as

$RF(w.r.t\ I) = 2^{2(i-1)} \times k \times k,$

whereas in the proposed overcomplete or fine-context design, bilinear upsampling is used instead of pooling, and receptive-field growth is constrained as

$$RF(w.r.t\ I) = \left(\frac{1}{2}\right)^{2(i-1)} \times k \times k.$$

The stated consequence is improved retention of edges, boundaries, and low-level structures, which is crucial for small, blurry, or unclear shadow regions (Valanarasu et al., 2021).

FCD consists of 4 conv blocks, each followed by bilinear upsampling and ReLU. The paper reports that the feature resolution saturates at about $400 \times 400$, and increasing beyond 4 blocks does not improve BER meaningfully. FCD outputs $F6$, which is further processed by a DS block to produce $L1$0. The final detector interpolates $L1$1 and $L1$2 to image size, concatenates them, applies a $L1$3 convolution, and then a sigmoid to obtain the final binary shadow mask (Valanarasu et al., 2021).

The associated R2D strategy provides the complementary mechanism. If the input image is $L1$4, the detector output is $L1$5 and the remover output is $L1$6, with the removal objective conceptualized as

$L1$7

and described as

$L1$8

The paper explicitly notes that this is not used as a literal residual-based detection method, because subtracting restored output from the input yields a noisy residual that does not delineate the shadow mask well. Instead, the key observation is that a restoration network still learns internal representations that encode where shadows are, even when trained for image-to-image restoration. These internal features become complementary shadow-aware cues for the detector (Valanarasu et al., 2021).

This architecture does not employ the exact phrase “shadow-independent structure extraction network,” but it establishes a closely related principle: a detector can be made less reliant on global darkness or broad context by emphasizing local structure and by importing restoration-derived features that implicitly encode shadow extent, boundaries, and transitions.

3. Structure-level shadow removal: StructNet and MStructNet

The 2023 Structure-Informed Shadow Removal Networks move the idea from detection into explicit structure reconstruction (Liu et al., 2023). The paper argues that shadows mainly degrade images at the image-structure level, defined as the low-frequency component that captures object shapes, smooth color regions, and region homogeneity. Shadow remnants are said to be especially problematic because they often occur in homogeneous regions with low-intensity values and are therefore difficult to eliminate in a conventional image-to-image mapping framework.

On this basis, StructNet is formulated as a two-stage system. In the first stage, the network performs structure-level shadow removal: it takes shadow image $L1$9, extracted structure $L5$0, and shadow mask $L5$1, and outputs restored shadow-free structure $L5$2. In the second stage, it performs image-level shadow removal: it takes $L5$3, $L5$4, and $L5$5, and predicts the final shadow-free image $L5$6 (Liu et al., 2023). The core claim is that if the model first reconstructs a shadow-free structure, the subsequent image restoration becomes easier and yields fewer remnants.

The first stage contains three branches: a whole-scene feature extraction or structure prediction branch, a shift estimation branch, and a mask propagation branch. The distinctive module is Mask-Guided Shadow-Free Extraction (MSFE), which replaces standard convolution with a mask-guided directional bridge. The generalized update is

$L5$7

and the shift tensor is recursively computed by

$L5$8

The bridge itself is

$L5$9

with normalization

$F1, F2, F3, F4, F5$0

Because the factor $F1, F2, F3, F4, F5$1 selects non-shadow locations, the transfer is explicitly directional, from non-shadow to shadow. The bridge weights are dynamically generated via

$F1, F2, F3, F4, F5$2

which distinguishes the method from partial convolution (Liu et al., 2023).

The companion module is Multi-Scale Feature & Residual Aggregation (MFRA), which adaptively fuses global convolutional perception with the shift tensor. MFRA computes

$F1, F2, F3, F4, F5$3

for scales $F1, F2, F3, F4, F5$4, and then dynamically combines them as

$F1, F2, F3, F4, F5$5

This is presented as a consistency-regularizing fusion rule that adapts to layer depth and local structure scale (Liu et al., 2023).

The paper also proposes MStructNet, a compact alternative that avoids explicit prediction of a final shadow-free structure image. Instead, it uses multi-level structures directly at the feature level through a structure-aware encoder, a fusion-oriented encoder, and a decoder. The structure levels are

$F1, F2, F3, F4, F5$6

Each level is assigned its own block, following the same MSFE + MFRA design (Liu et al., 2023).

Conceptually, StructNet and MStructNet are highly aligned with the notion of a shadow-independent structure extractor. The paper itself notes that such a label would be a reasonable conceptual paraphrase, while also emphasizing a more precise framing: the structure is not merely extracted but reconstructed in a shadow-free form and then leveraged to improve image-level removal (Liu et al., 2023).

4. Explicit formulation of SE-Net in portrait shadow removal

The most direct realization of the topic is the 2025 Structure-Guided Diffusion Models for High-Fidelity Portrait Shadow Removal, which introduces a dedicated shadow-independent structure extraction network (SE-Net) as the first stage of the pipeline (Yu et al., 7 Jul 2025). The motivation is tied to portrait editing, where shadow boundaries often coincide with strong illumination discontinuities and ordinary edge or structure predictors “will inevitably include shadow boundaries.” For portrait shadow removal, those boundaries are regarded as artifacts of lighting rather than semantic structure.

SE-Net is therefore trained to output a shadow-independent structure map that preserves facial geometry and identity cues while excluding unwanted shadow boundaries. The architecture is described as an encoder-decoder network with a flat convolution at the input, two downsampling convolution blocks, nine residual blocks in the bottleneck, two upsampling convolution blocks, and a final convolution layer (Yu et al., 7 Jul 2025). The output is a structure image in the style of portrait drawing or structure extraction, aligned to the domain of the pre-trained PDG model used for supervision.

The training strategy relies on synthetic illumination discontinuities constructed from the CelebA dataset. Starting from a well-lit portrait $F1, F2, F3, F4, F5$7, the method applies a physics-based relighting method to obtain $F1, F2, F3, F4, F5$8, and then combines the original and relit images with a random facial mask $F1, F2, F3, F4, F5$9: $DS1, DS2, DS3, DS4, DS5$0 where $DS1, DS2, DS3, DS4, DS5$1 denotes element-wise multiplication. This synthesis is intended to teach SE-Net to ignore lighting changes and recover a structure map that is invariant to illumination discontinuities (Yu et al., 7 Jul 2025). The paper explicitly contrasts this with the portrait shadow synthesis strategy from PSM, stating that the earlier strategy relies on simpler brightness, saturation, and subsurface-scattering adjustments and is not sufficient to model complex facial lighting.

SE-Net is supervised by a pseudo ground-truth structure map from the pre-trained PDG structure extractor. Letting $DS1, DS2, DS3, DS4, DS5$2 denote SE-Net and $DS1, DS2, DS3, DS4, DS5$3 denote PDG, the losses are

$DS1, DS2, DS3, DS4, DS5$4

along with an adversarial loss $DS1, DS2, DS3, DS4, DS5$5. The full objective is

$DS1, DS2, DS3, DS4, DS5$6

The text states $DS1, DS2, DS3, DS4, DS5$7 and $DS1, DS2, DS3, DS4, DS5$8, which it also describes as a clear typo and likely intended to mean $DS1, DS2, DS3, DS4, DS5$9, $L1$0 (Yu et al., 7 Jul 2025).

The training protocol uses 20,000 synthesized portrait images, 3 epochs, batch size 8, and learning rate $L1$1 (Yu et al., 7 Jul 2025). The resulting map $L1$2 is then used as conditioning input for the downstream diffusion model, not merely as auxiliary supervision but as a direct structural prior for generation.

The paper’s stated rationale makes the role of SE-Net precise. By being trained on relighted images with synthetic illumination discontinuities and supervised by structure outputs from the original well-lit image, SE-Net learns a representation that is insensitive to lighting changes. This prevents the diffusion model from tracing shadow edges as if they were facial structure, while still preserving the main geometric features required to maintain identity (Yu et al., 7 Jul 2025).

5. Conditioning role in downstream restoration and generation

Across the cited works, the extracted or reconstructed structure is not an endpoint; it is a conditioning signal that guides a second-stage restoration model.

In the 2025 portrait framework, SE-Net outputs a structure map $L1$3 that conditions a structure-guided portrait inpainting diffusion model (Yu et al., 7 Jul 2025). The overall pipeline is: input portrait plus shadow mask; SE-Net predicts $L1$4; then a diffusion-based inpainting model reconstructs the shadowed region conditioned on the masked input image $L1$5, the structure map $L1$6, the mask $L1$7, and the timestep $L1$8. The DDPM forward process is

$L1$9

with $RF(w.r.t\ I) = 2^{2(i-1)} \times k \times k,$0, $RF(w.r.t\ I) = 2^{2(i-1)} \times k \times k,$1, and $RF(w.r.t\ I) = 2^{2(i-1)} \times k \times k,$2. The reverse process is conditioned as

$RF(w.r.t\ I) = 2^{2(i-1)} \times k \times k,$3

and the training objective is

$RF(w.r.t\ I) = 2^{2(i-1)} \times k \times k,$4

Under this formulation, the structure map directly shapes the denoising trajectory by providing a facial structure prior (Yu et al., 7 Jul 2025).

The full portrait system adds a final gradient-guided detail restoration stage because SE-Net is designed for structure rather than the finest texture. The paper notes that details such as eyelashes, moles, and spots may not be fully captured by the structure map, so gradients inside shadow regions are used as guidance for a detail restoration diffusion model (Yu et al., 7 Jul 2025). This makes the division of labor explicit: SE-Net supplies the global face scaffold; the later stage restores micro-appearance details.

In the 2023 structure-informed shadow removal framework, the same broad principle appears in a non-diffusion setting. After obtaining a corrected structure image $RF(w.r.t\ I) = 2^{2(i-1)} \times k \times k,$5, the second stage performs image restoration via

$RF(w.r.t\ I) = 2^{2(i-1)} \times k \times k,$6

The restored structure acts as a shadow-free guide for image-level shadow removal, with the paper reporting fewer color artifacts and better restoration of object colors in shadowed regions when such guidance is used (Liu et al., 2023).

The 2021 shadow detector is not a shadow remover, but it also uses an auxiliary restoration branch as a source of structural information. The restoration network $RF(w.r.t\ I) = 2^{2(i-1)} \times k \times k,$7 is a lightweight U-Net with 5 encoder blocks, 5 decoder blocks, and skip connections. Rather than forwarding all restoration features, the method selects only $RF(w.r.t\ I) = 2^{2(i-1)} \times k \times k,$8 from the second encoder layer and $RF(w.r.t\ I) = 2^{2(i-1)} \times k \times k,$9 from the fifth encoder layer, providing a mixture of local and global semantic information (Valanarasu et al., 2021). Through the Complementary Feature Learning (CFL) block, these become $$RF(w.r.t\ I) = \left(\frac{1}{2}\right)^{2(i-1)} \times k \times k.$$0 and $$RF(w.r.t\ I) = \left(\frac{1}{2}\right)^{2(i-1)} \times k \times k.$$1, which are injected into FCSD-Net as

$$RF(w.r.t\ I) = \left(\frac{1}{2}\right)^{2(i-1)} \times k \times k.$$2

This suggests a consistent pattern across tasks: structure-like or shadow-aware features are most useful when aligned to appropriate semantic levels rather than naively concatenated.

6. Optimization, empirical evidence, and interpretive boundaries

The optimization objectives differ across the three formulations but share a common aim: to bias representation learning toward shadow-invariant or shadow-free structure.

For FCSD-Net and R2D, the detector uses a weighted BCE

$$RF(w.r.t\ I) = \left(\frac{1}{2}\right)^{2(i-1)} \times k \times k.$$3

with

$$RF(w.r.t\ I) = \left(\frac{1}{2}\right)^{2(i-1)} \times k \times k.$$4

and a distraction-aware term

$$RF(w.r.t\ I) = \left(\frac{1}{2}\right)^{2(i-1)} \times k \times k.$$5

The shadow loss is

$$RF(w.r.t\ I) = \left(\frac{1}{2}\right)^{2(i-1)} \times k \times k.$$6

the detection loss is

$$RF(w.r.t\ I) = \left(\frac{1}{2}\right)^{2(i-1)} \times k \times k.$$7

with $$RF(w.r.t\ I) = \left(\frac{1}{2}\right)^{2(i-1)} \times k \times k.$$8, the restoration pretraining uses

$$RF(w.r.t\ I) = \left(\frac{1}{2}\right)^{2(i-1)} \times k \times k.$$9

and the final joint objective is

$400 \times 400$0

The method is pretrained on ISTD for 500 epochs with images resized to $400 \times 400$1, batch size 16, learning rate 0.001, SGD, momentum 0.9, weight decay 0.0001, and an ImageNet pretrained ResNeXt-101 backbone. Fine-tuning is performed for 6000 iterations with random flipping, and stochastic weight averaging is used over checkpoints at 4000, 5000, and 6000 iterations (Valanarasu et al., 2021). On UCF, SBU, and ISTD, the method reports best average BERs of 6.96, 3.15, and 1.69, respectively, with mean 3.93. Compared with DSDNet, the reported improvements are 8.30% on UCF, 8.69% on SBU, and 22.11% on ISTD (Valanarasu et al., 2021).

For StructNet and MStructNet, the training loss is

$400 \times 400$2

with $400 \times 400$3 and $400 \times 400$4, and

$400 \times 400$5

At the structure level, the empirical study also uses

$400 \times 400$6

The paper reports that as $400 \times 400$7 increases, shadow-region RMSE decreases continuously, though too large $400 \times 400$8 can cause shadow spreading into nearby non-shadow areas, and $400 \times 400$9 is chosen as the best overall tradeoff (Liu et al., 2023). On ISTD+, a two-stage pipeline with $F6$0 improves shadow RMSE from 6.33 to 5.89 and all-region RMSE from 3.36 to 3.05 relative to direct single-stage image-level shadow removal. On benchmark datasets, StructNet and MStructNet outperform several baselines, while MStructNet achieves lower computation: 20.62 MB, 28.77 G FLOPs, 2.8 ms, versus 67.06 MB, 45.95 G FLOPs, 3.3 ms for StructNet (Liu et al., 2023).

For SE-Net, the most direct empirical evidence comes from ablation within the portrait diffusion system. In Table 1, “Ours w/o structure guidance” gives SSIM 0.757, LPIPS 0.101, RMSE 23.598, whereas the full method gives SSIM 0.830, LPIPS 0.056, RMSE 17.162 (Yu et al., 7 Jul 2025). Using PDG predicted structure yields SSIM 0.779, LPIPS 0.092, RMSE 19.963, and training with data synthesis in PSM gives SSIM 0.785, LPIPS 0.086, RMSE 19.797, both inferior to the full SE-Net configuration (Yu et al., 7 Jul 2025). These numbers support two separate claims: structure guidance itself improves fidelity, and the quality of the structure map materially affects performance.

A further interpretive boundary is worth noting. In the portrait diffusion work, SE-Net is trained to generate a structure map “in the style of portrait drawing / structure extraction” under PDG supervision (Yu et al., 7 Jul 2025). In StructNet, structure is defined in terms of the image-structure layer and multi-level structure extraction (Liu et al., 2023). In FCSD-Net, the emphasis is on low-level boundaries and receptive-field control rather than on explicit structure images (Valanarasu et al., 2021). This indicates that “structure” is not represented identically across papers. A plausible implication is that shadow-independent structure extraction is better viewed as a family of invariance-inducing designs rather than a single canonical representation.

7. Scope, significance, and recurring design principles

Across the three works, several recurring principles define the technical meaning of shadow-independent structure extraction.

First, structure is treated as a more stable carrier of content than raw appearance. In StructNet, the structure layer captures object shapes and continuous colors and is less affected by the remnant problem than direct image-to-image translation (Liu et al., 2023). In SE-Net, the structure map acts as a facial scaffold for diffusion inpainting and is explicitly intended to preserve identity while excluding shadow edges (Yu et al., 7 Jul 2025). In FCSD-Net, constrained receptive fields and low-level feature emphasis are used to retain boundary structure that broad contextual encoders may overlook (Valanarasu et al., 2021).

Second, illumination-induced edges must be suppressed, not reinforced. The 2025 portrait paper frames this most directly: if shadow boundaries are fed into the downstream diffusion model, the model is encouraged to reconstruct them rather than remove them (Yu et al., 7 Jul 2025). The 2023 paper operationalizes suppression through mask-guided directional transfer from non-shadow to shadow regions (Liu et al., 2023). The 2021 detector reaches a related goal by learning restoration-guided features that encode shadow extent and ambiguity without relying on direct residual differencing (Valanarasu et al., 2021).

Third, shadow-independent does not preclude shadow-aware mechanics. The extraction modules are frequently driven by masks, auxiliary restoration streams, or synthesized relighting discontinuities. This is especially clear in MSFE, which is mask-guided and directional (Liu et al., 2023), and in R2D, where feature maps still flow from the pretrained remover $F6$1 to the detector $F6$2 even when clean images are unavailable during fine-tuning on SBU and UCF (Valanarasu et al., 2021). The resulting independence is therefore representational rather than procedural.

Fourth, structure priors are most effective when integrated into a broader restoration or generation system. SE-Net is foundational but paired with structure-guided diffusion inpainting and gradient-guided detail restoration (Yu et al., 7 Jul 2025). StructNet reconstructs a shadow-free structure prior and then hands it to a second-stage image restorer (Liu et al., 2023). FCSD-Net uses restoration features through the CFL block rather than treating removal as a standalone endpoint (Valanarasu et al., 2021). This suggests that the principal role of shadow-independent structure extraction is to provide an intermediate representation that regularizes downstream reasoning.

In that sense, the term designates both a specific 2025 network architecture and a broader methodological direction. It refers to networks that learn a representation of scene or facial structure that is robust to shadow appearance, selectively excludes illumination boundaries, and serves as a prior for subsequent detection, restoration, or generative inpainting (Yu et al., 7 Jul 2025, Liu et al., 2023, Valanarasu et al., 2021).