shadowAttn: Vision & Mobile Sparse Attention

• shadowAttn is a dual-purpose mechanism that leverages shadow structures—either through explicit shadow masks or low‐precision pilot computations—to direct attention in both vision tasks and mobile LLM inference.
• In vision applications, it guides restoration, detection, and adversarial robustness by focusing on localized shadow and non-shadow regions using techniques like mask-guided and regional attention.
• In mobile systems, shadowAttn employs a dynamic sparse attention module that preselects key tokens, reducing latency and energy consumption while maintaining high inference accuracy.

“shadowAttn” is used in two distinct senses in the literature represented here. In vision research, it functions as a convenient umbrella for shadow-aware mechanisms that either exploit shadows as adversarial perturbations or use shadow/non-shadow structure to guide restoration, detection, and robustness analysis in natural and document images (Zhong et al., 2022, Guo et al., 2023, Liu et al., 2024, Chen et al., 2023, Wang et al., 2024). In mobile systems research, “shadowAttn” is also the explicit name of a dynamic sparse attention module for on-device LLM inference, where a low-precision pilot computation identifies important tokens before high-precision sparse attention is computed on a small subset (Yin et al., 22 Aug 2025). Across these uses, the common theme is selective emphasis: either spatial emphasis over shadow-relevant image regions or computational emphasis over a small set of informative tokens.

1. Conceptual scope and terminological variants

Within shadow-related vision work, “shadowAttn” is not a single standardized operator. Rather, it denotes several task-specific mechanisms that exploit the fact that shadows are localized, structured, and physically meaningful degradations. In image shadow removal, this includes explicit mask-guided interaction between shadow and non-shadow regions, neighborhood-based regional attention, soft and hard spatial attention, threshold-guided document masking, and recurrent shadow attention maps (Guo et al., 2023, Liu et al., 2024, Cholopoulou et al., 2024, Chen et al., 2023, Ding et al., 2019). In adversarial settings, the same term has been used conceptually to describe a model’s implicit sensitivity to illumination patterns, particularly shadowed regions, even when no explicit attention layer is present (Zhong et al., 2022).

A useful way to organize the term is by function. One family uses shadows as an attack channel: ordinary shadows, or shadow-adaptive perturbation budgets, steer model behavior while remaining visually plausible (Zhong et al., 2022, Wang et al., 2024). A second family uses shadow-aware feature routing for restoration: non-shadow regions serve as reference signals for reconstructing shadowed regions, often through mask-conditioned or locality-constrained interactions (Guo et al., 2023, Liu et al., 2024, Cholopoulou et al., 2024). A third family, specific to documents, combines thresholding, mask estimation, and attention-based refinement to separate shadow from text and background (Chen et al., 2023, Liu et al., 2 Jul 2025). A fourth family, outside vision, uses “shadowAttn” as the proper name of an NPU-centric sparse attention design for LLM inference on mobile SoCs (Yin et al., 22 Aug 2025).

This suggests that the term is best understood as a pattern rather than a single architecture: an attention or attention-like mechanism whose selectivity is driven by shadow structure, shadow masks, illumination cues, or a computational proxy for importance.

2. Shadows as adversarial attention modulators

The physical-world attack of Zhong et al. uses ordinary shadows as adversarial perturbations without modifying the traffic sign itself (Zhong et al., 2022). The attacker casts a shadow by placing an occluding object such as cardboard between a light source and the sign, using sunlight outdoors or a flashlight indoors, and controls the approximate shape and position of the shadow under a black-box threat model (Zhong et al., 2022). The perturbation is modeled in LAB space by attenuating only the lightness channel inside a polygonal shadow region, with a coefficient $k \in (0,1)$ restricted to realistic ranges such as $0.2$–$0.7$, and with empirical analysis on the SBU Shadow dataset indicating a mean ratio $$L_{\text{shadow}} / L_{\text{non-shadow}} \approx 0.43$$ while $A$ and $B$ remain approximately unchanged (Zhong et al., 2022).

The attack optimizes shadow geometry rather than additive pixel noise. For a sign image $x$, true label $y_{\text{true}}$, and classifier $f$, the adversarial goal is

$$\arg\max_i f_i(x) \neq \arg\max_i f_i(x_{\text{adv}}),$$

with the shadowed image produced by a shadow-generation pipeline $0.2$0 (Zhong et al., 2022). Because gradients are unavailable and polygon membership is discrete, the paper uses Particle Swarm Optimization to minimize the true-class score over polygon vertices, with random restarts when needed (Zhong et al., 2022). For physical robustness, it adds Expectation Over Transformation over downsampling, brightness adjustment, perspective transformations, motion blur, and variation in $0.2$1, followed by a stabilization phase that maximizes a dominant wrong class under the same transformation distribution (Zhong et al., 2022).

On traffic sign recognition, this shadow attack reached $0.2$2 and $0.2$3 success rates on the LISA and GTSRB test sets, respectively, for $0.2$4, with average queries until success of approximately $0.2$5 on LISA and $0.2$6 on GTSRB (Zhong et al., 2022). In an outdoor real-world test on a US speed limit 25 sign, $0.2$7 of video frames were misclassified and $0.2$8 were misclassified as the same wrong class, “speed limit 35” (Zhong et al., 2022). The paper also simulated a scheduled attack driven by solar geometry, showing successful windows around a planned time, with durations ranging from tens of seconds to several minutes (Zhong et al., 2022).

Although this work does not instantiate a neural attention layer, it explicitly motivates a shadowAttn interpretation: localized intensity changes in key sign regions can shift the classifier’s effective focus and redirect its decision toward another class (Zhong et al., 2022). That interpretation becomes more concrete in follow-up defense work. A defense for road sign recognition augments RGB inputs with a fourth channel containing either a binary adaptive threshold map or a Canny edge map, thereby steering the classifier toward structural features such as contours, text strokes, and foreground/background separation (Wang et al., 2022). This defense achieved approximately $0.2$9 robustness at $0.7$0 for adaptive thresholding and approximately $0.7$1 robustness at $0.7$2 for edge maps on GTSRB, while maintaining approximately $0.7$3 benign test accuracy; the abstract reports $0.7$4 robustness and $0.7$5 benign test accuracy for the edge defense (Wang et al., 2022). The same paper reformulates the shadow attack as an $0.7$6-bounded perturbation after LAB-to-RGB conversion, with

$0.7$7

which links shadow-based attacks to more conventional perturbation-based robustness analysis (Wang et al., 2022).

3. Shadow-aware attention for natural-image restoration

In shadow removal, the central architectural idea is that only part of the image is corrupted, and the most informative cues for restoration lie in nearby or globally corresponding non-shadow regions. ShadowFormer formulates this explicitly through a Retinex-based shadow model in which a shadowed image is

$0.7$8

with shared reflectance $0.7$9 across shadow and non-shadow regions (Guo et al., 2023). This motivates a transformer-based encoder-decoder with a multi-scale channel attention framework and a Shadow-Interaction Module containing Shadow-Interaction Attention at the bottleneck (Guo et al., 2023).

Shadow-Interaction Attention is mask-guided self-attention biased toward shadow–non-shadow interactions. After max-pooling the shadow mask to the bottleneck scale, ShadowFormer constructs a patch-wise correlation map

$$L_{\text{shadow}} / L_{\text{non-shadow}} \approx 0.43$$0

where XOR marks whether two patches belong to different illumination regimes (Guo et al., 2023). Attention then becomes

$$L_{\text{shadow}} / L_{\text{non-shadow}} \approx 0.43$$1

so shadow–non-shadow pairs are retained while same-region interactions are attenuated (Guo et al., 2023). On ISTD at $$L_{\text{shadow}} / L_{\text{non-shadow}} \approx 0.43$$2, the large variant achieved all-image PSNR $$L_{\text{shadow}} / L_{\text{non-shadow}} \approx 0.43$$3, SSIM $$L_{\text{shadow}} / L_{\text{non-shadow}} \approx 0.43$$4, and RMSE $$L_{\text{shadow}} / L_{\text{non-shadow}} \approx 0.43$$5 with $$L_{\text{shadow}} / L_{\text{non-shadow}} \approx 0.43$$6M parameters; on SRD it achieved all-image PSNR $$L_{\text{shadow}} / L_{\text{non-shadow}} \approx 0.43$$7, SSIM $$L_{\text{shadow}} / L_{\text{non-shadow}} \approx 0.43$$8, and RMSE $$L_{\text{shadow}} / L_{\text{non-shadow}} \approx 0.43$$9; on ISTD+ it achieved all-image PSNR $A$0 and RMSE $A$1 (Guo et al., 2023). The paper characterizes this as state-of-the-art performance with up to $A$2 fewer model parameters (Guo et al., 2023).

RASM, introduced later, replaces global attention with a regional attention mechanism tailored to the local structure of shadows (Liu et al., 2024). Operating at the bottleneck on a downsampled feature map $A$3, it defines a local dilated neighborhood for each token and computes neighborhood attention

$A$4

with relative positional bias and regional keys and values (Liu et al., 2024). The intended effect is that shadow tokens attend primarily to nearby non-shadow tokens that carry reliable local illumination, color, and texture cues (Liu et al., 2024). RASM uses a U-shaped encoder-decoder with channel attention modules, a Regional Attention Module only at the bottleneck, and a loss consisting of Charbonnier content loss plus VGG-19 perceptual loss with $A$5 and $A$6 (Liu et al., 2024).

The computational argument is explicit. RASM uses embedding dimension $A$7, regional attention only at the bottleneck, and local neighborhoods such as $A$8 with dilation $A$9, so complexity is $B$0 rather than $B$1 (Liu et al., 2024). On $B$2 SRD inputs, RASM has $B$3M parameters and $B$4 GFLOPs, compared with ShadowFormer at $B$5M parameters and $B$6 GFLOPs, and ShadowDiffusion at $B$7M parameters and $B$8 GFLOPs (Liu et al., 2024). On ISTD+, RASM achieved best reported values across all-image PSNR $B$9 dB, SSIM $x$0, and RMSE $x$1, together with shadow-region PSNR $x$2 dB, SSIM $x$3, and RMSE $x$4 (Liu et al., 2024). On SRD, it achieved best non-shadow PSNR $x$5, SSIM $x$6, and RMSE $x$7, and best all-image SSIM $x$8 and RMSE $x$9 (Liu et al., 2024).

A different design point appears in SHAU, the Soft-Hard Attention U-Net for multiscale shadow removal (Cholopoulou et al., 2024). SHAU uses a U-Net backbone with multiscale feature extraction blocks employing parallel $y_{\text{true}}$0, $y_{\text{true}}$1, and $y_{\text{true}}$2 convolutions, hard attention in the encoder via binary shadow-mask gating, and soft attention in the decoder for fine-grained feature fusion (Cholopoulou et al., 2024). The hard attention uses downsampled masks produced by max-pooling, while soft attention follows each decoder multiscale block and a final soft-attention layer combines shallow encoder features with the original image (Cholopoulou et al., 2024). On the synthetic MSRD benchmark, SHAU achieved $y_{\text{true}}$3, $y_{\text{true}}$4, $y_{\text{true}}$5, with $y_{\text{true}}$6, $y_{\text{true}}$7, and $y_{\text{true}}$8 (Cholopoulou et al., 2024). The paper states that SHAU improves the PSNR and RMSE performance metrics for the shadow image area by $y_{\text{true}}$9 and $f$0, respectively, over the state of the art across all datasets (Cholopoulou et al., 2024).

4. Document shadow removal and explicit shadow-centric masking

Document shadow removal sharpens the role of shadowAttn because the dominant ambiguity is not only shadow versus non-shadow, but also shadow versus text. ShaDocFormer addresses this with a two-module transformer-based pipeline comprising a Shadow-attentive Threshold Detector and a Cascaded Fusion Refiner (Chen et al., 2023). The detector begins with a classical Otsu threshold on grayscale intensities, selecting

$f$1

and binarizing pixels into dark versus light regions (Chen et al., 2023). Convolutional layers then encode local patterns, and transformer blocks with multi-head self-attention refine the representation so that dark regions can be disambiguated using global document layout, repeated text lines, and background consistency (Chen et al., 2023). The output is a refined shadow probability mask $f$2 (Chen et al., 2023).

The Cascaded Fusion Refiner uses the original shadowed image together with the detected mask in a U-shaped encoder-decoder with Convolutional Depth Grouped Fusion Net blocks, multi-head attention, a dual-gated feed-forward network, and spatial pyramid pooling (Chen et al., 2023). The loss is

$f$3

combining pixel-wise reconstruction, SSIM, and perceptual loss (Chen et al., 2023). On the RDD dataset, ShaDocFormer achieved PSNR $f$4, SSIM $f$5, and RMSE $f$6; on the Kligler dataset it achieved PSNR $f$7, SSIM $f$8, and RMSE $f$9 (Chen et al., 2023). In the ablation reported in the text, removing the detector degrades PSNR from $$\arg\max_i f_i(x) \neq \arg\max_i f_i(x_{\text{adv}}),$$0 to $$\arg\max_i f_i(x) \neq \arg\max_i f_i(x_{\text{adv}}),$$1 and increases RMSE, which the paper uses to attribute a substantial role to the shadow-attentive masking stage (Chen et al., 2023).

ARGAN provides an earlier explicit learnable shadow attention mechanism in a recurrent GAN (Ding et al., 2019). At each progressive step $$\arg\max_i f_i(x) \neq \arg\max_i f_i(x_{\text{adv}}),$$2, a shadow attention detector produces a soft attention map $$\arg\max_i f_i(x) \neq \arg\max_i f_i(x_{\text{adv}}),$$3, supervised against a shadow matte $$\arg\max_i f_i(x) \neq \arg\max_i f_i(x_{\text{adv}}),$$4 using

$$\arg\max_i f_i(x) \neq \arg\max_i f_i(x_{\text{adv}}),$$5

with $$\arg\max_i f_i(x) \neq \arg\max_i f_i(x_{\text{adv}}),$$6 and $$\arg\max_i f_i(x) \neq \arg\max_i f_i(x_{\text{adv}}),$$7 in practice (Ding et al., 2019). A shadow removal encoder then produces a residual map that is multiplied by $$\arg\max_i f_i(x) \neq \arg\max_i f_i(x_{\text{adv}}),$$8 so that the correction is concentrated on shadow regions (Ding et al., 2019). On shadow detection, ARGAN achieved BER $$\arg\max_i f_i(x) \neq \arg\max_i f_i(x_{\text{adv}}),$$9 on SBU, $0.2$00 on UCF, and $0.2$01 on ISTD, while ARGAN+SS improved these to $0.2$02, $0.2$03, and $0.2$04, respectively (Ding et al., 2019). On shadow removal, ARGAN achieved RMSE $0.2$05 on SRD and $0.2$06 on ISTD over the entire image, with ARGAN+SS improving to $0.2$07 and $0.2$08 (Ding et al., 2019).

DocShaDiffusion moves document shadow removal into latent-space diffusion and implements shadow awareness without explicit self-attention layers (Liu et al., 2 Jul 2025). Its Shadow Soft-mask Generation Module produces a continuous mask $0.2$09 from a single shadowed image using grayscale conversion, dilation, median filtering, darkest-pixel statistics, and a final median filtering step (Liu et al., 2 Jul 2025). The mask then modulates both the initial noisy state and the diffusion process through a Shadow Mask-aware Guided Diffusion Module. The forward and reverse SDEs are

$0.2$10

and

$0.2$11

so stochasticity is spatially modulated by the shadow mask (Liu et al., 2 Jul 2025). Training uses

$0.2$12

with $0.2$13, where $0.2$14 is a shadow-robust perceptual feature loss built from weighted VGG-19 slices (Liu et al., 2 Jul 2025). A plausible implication is that this realizes shadowAttn at the level of generative dynamics rather than feature-level dot-product attention.

5. Robustness analysis, synthetic data, and region-adaptive perturbation

A robustness-centered branch of the literature studies shadow-aware attacks and defenses on restoration networks themselves. The shadow-adaptive adversarial attack for shadow removal argues that a uniform perturbation budget is mismatched to the spatially varying illumination of shadow images (Wang et al., 2024). Instead of constraining $0.2$15, it uses an intensity-normalized constraint

$0.2$16

equivalently $0.2$17 per pixel (Wang et al., 2024). The attack is optimized with PGD for $0.2$18 iterations and step size $0.2$19, with adaptive budgets $0.2$20 in the direct comparison and $0.2$21 in the robustness benchmark (Wang et al., 2024).

The paper also proves an $0.2$22-energy equivalence result: choosing the uniform budget as $0.2$23 makes the maximum achievable average $0.2$24 perturbation equal under uniform and adaptive attacks (Wang et al., 2024). On ShadowFormer, for ISTD with $0.2$25, uniform and adaptive attacks yield PSNR $0.2$26 versus $0.2$27 and SSIM $0.2$28 versus $0.2$29; for ISTD+ at the same budget they yield PSNR $0.2$30 versus $0.2$31 and SSIM $0.2$32 versus $0.2$33 (Wang et al., 2024). The attack strength is therefore comparable, but the adaptive attack is visually less conspicuous in shadow regions because $0.2$34 is more uniform (Wang et al., 2024). The broader benchmark shows that as $0.2$35 increases to $0.2$36, all evaluated models on ISTD drop by approximately $0.2$37 dB in PSNR, and robustness differs systematically between shadow and non-shadow regions (Wang et al., 2024).

Synthetic data have played a major role in making shadow-aware models trainable. SynShadow provides a large-scale synthetic dataset of shadow image, shadow-free image, and matte triplets, built from $0.2$38 shadow matte images synthesized from 3D models and approximately $0.2$39 shadow-free background images from USR, with on-the-fly composition governed by a physically grounded affine illumination model (Inoue et al., 2021). The underlying relation is

$0.2$40

and the final shadow image is formed by alpha composition,

$0.2$41

with non-binary mattes capturing penumbra (Inoue et al., 2021). Pre-training on SynShadow improves several existing models. For example, on ISTD+, SP+M improves all-image RMSE from $0.2$42 to $0.2$43 and DHAN improves from $0.2$44 to $0.2$45 after SynShadow pre-training and fine-tuning; on SRD+, SP+M improves from $0.2$46 to $0.2$47 and DHAN from $0.2$48 to $0.2$49 (Inoue et al., 2021). For shadow detection on ISTD, BDRAR-ft and DSDNet++-ft achieve all-image BER $0.2$50 and $0.2$51, compared with prior DSDNet at $0.2$52 (Inoue et al., 2021).

The defense side of the original physical shadow attack also includes shadow-aware adversarial training. Augmenting training images with random shadow polygons and random shadow intensity $0.2$53 yields robust models $0.2$54 and $0.2$55 (Zhong et al., 2022). At $0.2$56, on LISA, clean accuracy changes from $0.2$57 to $0.2$58, robustness increases from $0.2$59 to $0.2$60, and average queries until success increase from approximately $0.2$61 to approximately $0.2$62 (Zhong et al., 2022). On GTSRB, clean accuracy changes from $0.2$63 to $0.2$64, robustness increases from $0.2$65 to $0.2$66, and queries increase from approximately $0.2$67 to approximately $0.2$68 (Zhong et al., 2022). This suggests that even simple random-shadow augmentation can substantially harden a classifier against naturalistic shadow perturbations.

6. shadowAttn as a named sparse attention module for mobile SoCs

A separate usage appears in systems research, where shadowAttn is the name of a dynamic sparse attention layer for LLM inference on mobile SoCs (Yin et al., 22 Aug 2025). The problem setting is not shadow removal or illumination, but the mismatch between quantization-sensitive attention and NPU hardware: in existing on-device frameworks, attention often falls back from the NPU to CPU/GPU because activations $0.2$69, $0.2$70, and $0.2$71 are much more quantization-sensitive than weights, and mobile NPUs typically support only per-tensor static quantization (Yin et al., 22 Aug 2025). Empirically, putting full attention on the NPU with INT8 produces an average accuracy drop of approximately $0.2$72 percentage points on mobile tasks (Yin et al., 22 Aug 2025).

shadowAttn addresses this by splitting attention into a pilot estimation phase and an exact sparse phase. Standard attention is

$0.2$73

shadowAttn first computes an approximate dense score matrix $0.2$74 on the NPU, uses top-$0.2$75 selection per head on CPU/GPU to identify important positions, and then computes exact high-precision attention only on those retained tokens (Yin et al., 22 Aug 2025). The kept-token ratio is controlled globally and per head. Head-specific ratios are derived from offline head and layer importance measurements on a calibration set of $0.2$76 WikiText-2 samples, with a formula that allocates larger kept-token ratios to more important heads (Yin et al., 22 Aug 2025).

Several system techniques make the design practical. NPU compute graph bucketing precompiles multiple static Q·K graphs for different scale-factor buckets; the paper uses $0.2$77 buckets and reports that each Q·K graph is only $0.2$78–$0.2$79 KB (Yin et al., 22 Aug 2025). A head-wise NPU–CPU/GPU pipeline overlaps three stages per head: NPU Q·K estimation, CPU/GPU top-$0.2$80, and CPU/GPU sparse QKV (Yin et al., 22 Aug 2025). With fused NPU launches, a single-head Q·K takes $0.2$81 ms on MI14, two fused heads take $0.2$82 ms, and four fused heads take $0.2$83 ms (Yin et al., 22 Aug 2025). A global sparsity ratio of $0.2$84 is identified as a knee point giving minimal accuracy loss with substantial latency reduction (Yin et al., 22 Aug 2025).

The reported outcomes are primarily systems metrics. Compared with full float attention on CPU/GPU, shadowAttn yields an average accuracy of $0.2$85 versus $0.2$86, a drop of only $0.2$87 percentage points across four models and three datasets (Yin et al., 22 Aug 2025). It delivers up to $0.2$88 and on average $0.2$89 speedup for attention kernel latency, and up to $0.2$90 and on average $0.2$91 end-to-end speedup on prefill-heavy workloads (Yin et al., 22 Aug 2025). On Redmi K60, single attention kernel energy drops from $0.2$92 J to $0.2$93 J for PhoneLM-0.5B and from $0.2$94 J to $0.2$95 J for PhoneLM-1.5B when comparing full CPU/GPU attention with shadowAttn (Yin et al., 22 Aug 2025). In this usage, “shadow” refers not to illumination but to a low-precision pilot computation that “shadows” the exact attention to identify important tokens.

This second lineage is terminologically separate from shadow-aware vision models, yet structurally related: both senses of shadowAttn use a lightweight mechanism to decide where expensive computation or representational capacity should be concentrated. In vision, that selectivity is driven by shadow masks, boundaries, non-shadow references, or luminance-adaptive perturbation structure (Guo et al., 2023, Liu et al., 2024, Wang et al., 2024). In mobile LLM systems, it is driven by pilot attention scores used to retain only a tiny portion of tokens (Yin et al., 22 Aug 2025).