ShadowHack: Adversarial Shadows & Removal

• ShadowHack is a dual-purpose computer vision paradigm that encompasses a natural shadow-based adversarial attack and a single-image shadow removal framework using luminance-color decomposition.
• The adversarial attack manipulates natural shadows with simple occluders and Particle Swarm Optimization to significantly lower classifier confidence, achieving up to 98% success in digital tests and 95–100% in real-world scenarios.
• The shadow removal approach employs a U-shaped transformer with Rectified Outreach Attention and cross-attention color injection, yielding state-of-the-art restoration results on benchmarks like ISTD+ and SRD.

ShadowHack denotes two distinct but influential paradigms in computer vision research: (1) a physical-world adversarial attack using natural shadows to mislead vision classifiers (Zhong et al., 2022) and (2) a state-of-the-art shadow removal framework grounded in a luminance-color divide-and-conquer design (Hu et al., 2024). Each leverages the physical and perceptual complexity of shadows—either as a vulnerability in recognition pipelines or as a restoration challenge in image enhancement.

1. ShadowHack as a Physical-World Adversarial Attack

ShadowHack, in the context of security and adversarial machine learning, is defined as a black-box optical attack that manipulates natural illumination to cast intentionally designed shadows on target objects without physically altering the object or its immediate environment (Zhong et al., 2022). The attacker employs simple occluders and controls the geometry, position, and darkness of the shadow by exploiting a single dominant light source (e.g., sunlight or flashlight). The adversarial perturbation is implemented by selectively darkening pixel regions (in LAB space, lightness channel) inside polygonal masks, parameterized by vertices $V$ and a darkness factor $k<1$.

Mathematically, given a clean RGB image $x\in\mathbb{R}^d$ and a classifier $f:\mathbb{R}^d\to\mathbb{R}^k$, the attack defines a transformation $S(x;\theta)$, where $\theta=(V,k)$, yielding an adversarial example $x_{\text{adv}} = S(x;\theta)$. Within the polygonal mask, the lightness channel is scaled by $k$ while chromaticity channels are retained. The perturbation $\Delta(x;\theta) = S(x;\theta)-x$ is visually natural and localized.

The objective is to minimize the model’s confidence in the true class, typically via the negative logit of the true label, while satisfying constraints on $k$ and geometric naturalness. Formally:

$$\min_{\theta} L(f(S(x;\theta)),y_{\text{true}}) + \lambda R(\theta)$$

subject to $\hat{y}_{\text{adv}}\ne y_{\text{true}}$, $0 \leq k \leq 1$, and $V_i$ inside the bounding box.

Optimization is conducted via Particle Swarm Optimization (PSO) with multiple restarts, making use of confidence-only queries to the black-box model. Robustness across transformations (scale, rotation, illumination, motion blur) is achieved by optimizing the expected attack loss over random augmentations (Expectation Over Transformation), and prediction stabilization techniques maximize misclassification persistence over video frames.

Empirical evaluation on digital (LISA and GTSRB sign datasets) and physical (moving camera, sunlight/flashlight) platforms reports near-maximum success: up to 98.23% and 90.47% in digital and 95–100% in real-world settings, with misclassifications dominated by a single incorrect label across varied conditions.

2. Limitations and Countermeasures in Adversarial Shadow Attacks

While ShadowHack typifies a stealthy, physically plausible adversarial attack, several practical limitations constrain its applicability (Zhong et al., 2022):

• A dominant, single light source is required; the method fails under multiple or highly diffuse illuminants.
• Directly achieving targeted misclassification (to a specified wrong label) is challenging.
• The precise occluder placement and timing can limit attack windows in dynamic environments.

Defenses against such attacks focus on both model-level and pipeline-level strategies:

1. Adversarial Training with Random Shadows: Augmenting training data with synthetic random shadows (varying mask and darkness) substantially boosts robustness metrics (from 1–9% up to 25–40%), with a modest clean-accuracy penalty and a 4–8× increase in attack query complexity.
2. Shadow Removal Preprocessing: Physics-based LAB-ratio methods or other shadow-detection and removal techniques can filter out adversarial shadows before classification.
3. Multi-modal Fusion: Fusing RGB input with depth, LiDAR, or infrared data (insensitive to projected shadows) mitigates the attack vector.
4. Temporal Consistency Checks: Monitoring abrupt prediction changes correlated with rapid shadow movement can enable detection and suppression of transient adversarial effects.

A plausible implication is that the widespread use of monocular RGB vision in safety-critical domains (autonomous driving, surveillance) remains susceptible to subtle natural-phenomenon attacks, underscoring the importance of multimodal robustness and physically-grounded threat modeling.

3. ShadowHack in Single-Image Shadow Removal

A separate and independently developed ShadowHack framework addresses the inverse problem: removal of shadows in single RGB images with maximal fidelity (Hu et al., 2024). This approach reframes shadow removal as a composite of two coupled image degradations—luminance loss (brightness and texture) and chroma shift (color distortion)—and proposes an explicit YCbCr-space decomposition.

The input $I$ is converted to luminance $I_t=Y$ and chroma $I_c=(Cb,Cr)$:

• Luminance restoration $\mathcal{R}$ is applied to $I_t$ by a dedicated U-shaped transformer (LRNet).
• Color regeneration $\mathcal{C}$ is conditioned on the restored luminance and operates on $I_c$.

The final reconstruction is synthesized by:

$$\hat I = \mathcal{D}^{-1}\left(\mathcal{C}\left(\mathcal{R}(I_t), I_c\right)\right),$$

where $\mathcal{D}^{-1}$ denotes inverse YCbCr$\rightarrow$RGB.

4. Luminance Restoration and the Rectified Outreach Attention Network

LRNet is designed as a four-level, U-shaped light-weight transformer. Each level incorporates:

• Local Range Blocks (depthwise convs) for fine-scale texture extraction,
• Multi-head Transposed Attention for channel mixing,
• Feed-forward networks for nonlinearity.

At the deepest levels, the Rectified Outreach Attention (ROA) module extends standard attention by aggregating cues from distant, potentially non-shadowed regions through dilated, overlapping windows and rectifying spurious correlations by subtracting a color-guided map. For features $F_t$ (luminance) and $F_c$ (chroma), attention maps are computed as:

$$\mathrm{ROA}(F_t, F_c) = \left(\mathrm{Att}_1 - \lambda\,\mathrm{Att}_2\right)V,$$

where $\lambda$ is a learnable scalar, and $\mathrm{Att}_1$ and $\mathrm{Att}_2$ are attention maps generated from $[F_t;F_c]$ and $F_c$ respectively.

Supervision is based on both $L_1$ loss in luminance and a perceptual (VGG) loss in RGB.

5. Color Regeneration with Cross-Attention and Checkpoint Ensemble

The subsequent CRNet operates an all-convolutional U-shaped backbone for luminance, guided by a ConvNeXt-v2 atto network pretrained solely on chroma channels. At each skip connection, CRNet performs cross-attention to inject multi-scale color features:

$$A = \mathrm{Softmax}\!\left(\frac{Q K^\top}{\sqrt{d} + B'}\right)V,$$

where $Q,K$ are projected luminance features and $V$ is the corresponding color feature.

Training uses a "checkpoint ensemble" strategy, presenting CRNet with random non-final LRNet checkpoints, thereby increasing robustness to luminance restoration errors.

The overall loss for joint optimization is:

$$\mathcal{L} = \left\|\hat I - I^*\right\|_1 + \alpha\sum_l \left\|\phi_l(\hat I)-\phi_l(I^*)\right\|_2^2.$$

6. Benchmarking, Ablations, and Failure Modes

ShadowHack is evaluated on ISTD+, SRD, UIUC, and UCF datasets. Metrics include PSNR, SSIM, and RMSE in LAB. Table summaries:

ISTD+ and SRD Performance

Method	Params (M)	PSNR↑ (SRD)	RMSE↓ (SRD)
RASM (MM’24)	5.2	34.46	3.37
HomoFormer (CVPR’24)	17.8	35.37	3.33
ShadowHack (Ours)	23.3	35.94	2.90

Ablation studies confirm the efficacy of YCbCr decoupling, the specific ROA variant, and cross-attention color injection. Performance on both single-image and patch-based shadow removal surpasses prior SOTA in both quantitative and qualitative analysis, achieving high-fidelity restoration of brightness, texture, and local chrominance.

Failure cases occur over extremely soft/penumbra-only shadows or under domain shifts in the chroma encoder (e.g., infrared datasets).

7. Technical Summary and Broader Impact

ShadowHack in adversarial settings reveals latent vulnerabilities in standard RGB vision systems, especially for safety-critical perception pipelines, motivating multi-modal defense architectures and physically-realistic adversarial robustness efforts (Zhong et al., 2022). As an image restoration architecture, ShadowHack’s divide-and-conquer methodology with ROA and dual-path color injection yields a new benchmark in single-image shadow removal (Hu et al., 2024).

The dual usage exemplifies the intersection of physical scene understanding, adversarial robustness, and modular architectural design in modern vision research, highlighting both the risks and opportunities posed by the complex interaction of light, material, and perception in machine learning pipelines.