---
title: 'shadowAttn: Vision & Mobile Sparse Attention'
url: https://www.emergentmind.com/topics/shadowattn
type: topic
---

# shadowAttn: Vision & Mobile Sparse Attention

“shadowAttn” is used in two distinct senses in the literature represented here. In vision research, it functions as a convenient umbrella for shadow-aware mechanisms that either exploit shadows as adversarial perturbations or use shadow/non-shadow structure to guide restoration, detection, and robustness analysis in natural and document images [2203.03818] [2302.01650] [2411.14201] [2309.06670] [2403.10076]. In mobile systems research, “shadowAttn” is also the explicit name of a dynamic sparse attention module for on-device large language model inference, where a low-precision pilot computation identifies important tokens before high-precision sparse attention is computed on a small subset [2508.16703]. Across these uses, the common theme is selective emphasis: either spatial emphasis over shadow-relevant image regions or computational emphasis over a small set of informative tokens.

## 1. Conceptual scope and terminological variants

Within shadow-related vision work, “shadowAttn” is not a single standardized operator. Rather, it denotes several task-specific mechanisms that exploit the fact that shadows are localized, structured, and physically meaningful degradations. In image shadow removal, this includes explicit mask-guided interaction between shadow and non-shadow regions, neighborhood-based regional attention, soft and hard spatial attention, threshold-guided document masking, and recurrent shadow attention maps [2302.01650] [2411.14201] [2408.03734] [2309.06670] [1908.01323]. In adversarial settings, the same term has been used conceptually to describe a model’s implicit sensitivity to illumination patterns, particularly shadowed regions, even when no explicit attention layer is present [2203.03818].

A useful way to organize the term is by function. One family uses shadows as an attack channel: ordinary shadows, or shadow-adaptive perturbation budgets, steer model behavior while remaining visually plausible [2203.03818] [2403.10076]. A second family uses shadow-aware feature routing for restoration: non-shadow regions serve as reference signals for reconstructing shadowed regions, often through mask-conditioned or locality-constrained interactions [2302.01650] [2411.14201] [2408.03734]. A third family, specific to documents, combines thresholding, mask estimation, and attention-based refinement to separate shadow from text and background [2309.06670] [2507.01422]. A fourth family, outside vision, uses “shadowAttn” as the proper name of an NPU-centric sparse attention design for LLM inference on mobile SoCs [2508.16703].

This suggests that the term is best understood as a pattern rather than a single architecture: an attention or attention-like mechanism whose selectivity is driven by shadow structure, shadow masks, illumination cues, or a computational proxy for importance.

## 2. Shadows as adversarial attention modulators

The physical-world attack of Zhong et al. uses ordinary shadows as adversarial perturbations without modifying the traffic sign itself [2203.03818]. The attacker casts a shadow by placing an occluding object such as cardboard between a light source and the sign, using sunlight outdoors or a flashlight indoors, and controls the approximate shape and position of the shadow under a black-box threat model [2203.03818]. The perturbation is modeled in LAB space by attenuating only the lightness channel inside a polygonal shadow region, with a coefficient $k \in (0,1)$ restricted to realistic ranges such as $0.2$–$0.7$, and with empirical analysis on the SBU Shadow dataset indicating a mean ratio $L_{\text{shadow}} / L_{\text{non-shadow}} \approx 0.43$ while $A$ and $B$ remain approximately unchanged [2203.03818].

The attack optimizes shadow geometry rather than additive pixel noise. For a sign image $x$, true label $y_{\text{true}}$, and classifier $f$, the adversarial goal is
$$
\arg\max_i f_i(x) \neq \arg\max_i f_i(x_{\text{adv}}),
$$
with the shadowed image produced by a shadow-generation pipeline $\mathcal{S}(x,\mathcal{P}_{\mathcal{V}},\mathcal{M},k)$ [2203.03818]. Because gradients are unavailable and polygon membership is discrete, the paper uses Particle Swarm Optimization to minimize the true-class score over polygon vertices, with random restarts when needed [2203.03818]. For physical robustness, it adds Expectation Over Transformation over downsampling, brightness adjustment, perspective transformations, motion blur, and variation in $k$, followed by a stabilization phase that maximizes a dominant wrong class under the same transformation distribution [2203.03818].

On traffic sign recognition, this shadow attack reached $98.23\%$ and $90.47\%$ success rates on the LISA and GTSRB test sets, respectively, for $k=0.43$, with average queries until success of approximately $91$ on LISA and $127$ on GTSRB [2203.03818]. In an outdoor real-world test on a US speed limit 25 sign, $100\%$ of video frames were misclassified and $95.91\%$ were misclassified as the same wrong class, “speed limit 35” [2203.03818]. The paper also simulated a scheduled attack driven by solar geometry, showing successful windows around a planned time, with durations ranging from tens of seconds to several minutes [2203.03818].

Although this work does not instantiate a neural attention layer, it explicitly motivates a shadowAttn interpretation: localized intensity changes in key sign regions can shift the classifier’s effective focus and redirect its decision toward another class [2203.03818]. That interpretation becomes more concrete in follow-up defense work. A defense for road sign recognition augments RGB inputs with a fourth channel containing either a binary adaptive threshold map or a Canny edge map, thereby steering the classifier toward structural features such as contours, text strokes, and foreground/background separation [2208.09285]. This defense achieved approximately $76.6\%$ robustness at $k=0.43$ for adaptive thresholding and approximately $78.0\%$ robustness at $k=0.43$ for edge maps on GTSRB, while maintaining approximately $98\%$ benign test accuracy; the abstract reports $78\%$ robustness and $98\%$ benign test accuracy for the edge defense [2208.09285]. The same paper reformulates the shadow attack as an $\ell_\infty$-bounded perturbation after LAB-to-RGB conversion, with
$$
\|x_{\text{adv}} - x\| = \varepsilon_k \le \|M\| \cdot 100|k-1|,
$$
which links shadow-based attacks to more conventional perturbation-based robustness analysis [2208.09285].

## 3. Shadow-aware attention for natural-image restoration

In shadow removal, the central architectural idea is that only part of the image is corrupted, and the most informative cues for restoration lie in nearby or globally corresponding non-shadow regions. ShadowFormer formulates this explicitly through a Retinex-based shadow model in which a shadowed image is
$$
\mathbf{I}_s = \mathbf{I}_m \odot \mathbf{L}_s \odot \mathbf{R} + (1 - \mathbf{I}_m) \odot \mathbf{L}_{ns} \odot \mathbf{R},
$$
with shared reflectance $\mathbf{R}$ across shadow and non-shadow regions [2302.01650]. This motivates a transformer-based encoder-decoder with a multi-scale channel attention framework and a Shadow-Interaction Module containing Shadow-Interaction Attention at the bottleneck [2302.01650].

Shadow-Interaction Attention is mask-guided self-attention biased toward shadow–non-shadow interactions. After max-pooling the shadow mask to the bottleneck scale, ShadowFormer constructs a patch-wise correlation map
$$
\mathbf{\Sigma}^{ij} = \mathbf{M}^{i} \oplus \mathbf{M}^{j},
$$
where XOR marks whether two patches belong to different illumination regimes [2302.01650]. Attention then becomes
$$
\text{SIA}(\mathbf{X}, \mathbf{\Sigma}) =
\text{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^{\top}}{d}\right)\mathbf{V}
\left[\sigma \mathbf{\Sigma} + (1-\sigma)\mathbf{1}\right],
$$
so shadow–non-shadow pairs are retained while same-region interactions are attenuated [2302.01650]. On ISTD at $256 \times 256$, the large variant achieved all-image PSNR $32.21$, SSIM $0.968$, and RMSE $4.09$ with $9.3$M parameters; on SRD it achieved all-image PSNR $32.90$, SSIM $0.958$, and RMSE $4.04$; on ISTD+ it achieved all-image PSNR $35.46$ and RMSE $2.8$ [2302.01650]. The paper characterizes this as state-of-the-art performance with up to $150\times$ fewer model parameters [2302.01650].

RASM, introduced later, replaces global attention with a regional attention mechanism tailored to the local structure of shadows [2411.14201]. Operating at the bottleneck on a downsampled feature map $Y \in \mathbb{R}^{C \times H' \times W'}$, it defines a local dilated neighborhood for each token and computes neighborhood attention
$$
NA_r(i) = \text{softmax}\!\left(\frac{A_i^r}{\sqrt{d}}\right)^{T} V_i^r,
$$
with relative positional bias and regional keys and values [2411.14201]. The intended effect is that shadow tokens attend primarily to nearby non-shadow tokens that carry reliable local illumination, color, and texture cues [2411.14201]. RASM uses a U-shaped encoder-decoder with channel attention modules, a Regional Attention Module only at the bottleneck, and a loss consisting of Charbonnier content loss plus VGG-19 perceptual loss with $\alpha_1 = 0.001$ and $\alpha_2 = 1$ [2411.14201].

The computational argument is explicit. RASM uses embedding dimension $C=32$, regional attention only at the bottleneck, and local neighborhoods such as $11\times11$ with dilation $2$, so complexity is $O(nr)$ rather than $O(n^2)$ [2411.14201]. On $256\times256$ SRD inputs, RASM has $5.2$M parameters and $25.2$ GFLOPs, compared with ShadowFormer at $11.4$M parameters and $63.1$ GFLOPs, and ShadowDiffusion at $55.2$M parameters and $896.7$ GFLOPs [2411.14201]. On ISTD+, RASM achieved best reported values across all-image PSNR $36.16$ dB, SSIM $0.976$, and RMSE $2.53$, together with shadow-region PSNR $40.73$ dB, SSIM $0.993$, and RMSE $4.41$ [2411.14201]. On SRD, it achieved best non-shadow PSNR $38.70$, SSIM $0.992$, and RMSE $2.72$, and best all-image SSIM $0.976$ and RMSE $3.37$ [2411.14201].

A different design point appears in SHAU, the Soft-Hard Attention U-Net for multiscale shadow removal [2408.03734]. SHAU uses a U-Net backbone with multiscale feature extraction blocks employing parallel $3\times3$, $5\times5$, and $7\times7$ convolutions, hard attention in the encoder via binary shadow-mask gating, and soft attention in the decoder for fine-grained feature fusion [2408.03734]. The hard attention uses downsampled masks produced by max-pooling, while soft attention follows each decoder multiscale block and a final soft-attention layer combines shallow encoder features with the original image [2408.03734]. On the synthetic MSRD benchmark, SHAU achieved $\text{PSNR}_S = 32.36$, $\text{PSNR}_N = 36.14$, $\text{PSNR}_A = 28.83$, with $\text{RMSE}_S = 5.76$, $\text{RMSE}_N = 4.96$, and $\text{RMSE}_A = 5.55$ [2408.03734]. The paper states that SHAU improves the PSNR and RMSE performance metrics for the shadow image area by $25.1\%$ and $61.3\%$, respectively, over the state of the art across all datasets [2408.03734].

## 4. Document shadow removal and explicit shadow-centric masking

Document shadow removal sharpens the role of shadowAttn because the dominant ambiguity is not only shadow versus non-shadow, but also shadow versus text. ShaDocFormer addresses this with a two-module transformer-based pipeline comprising a Shadow-attentive Threshold Detector and a Cascaded Fusion Refiner [2309.06670]. The detector begins with a classical Otsu threshold on grayscale intensities, selecting
$$
t^* = \arg\max_t \sigma_B^2(t),
$$
and binarizing pixels into dark versus light regions [2309.06670]. Convolutional layers then encode local patterns, and transformer blocks with multi-head self-attention refine the representation so that dark regions can be disambiguated using global document layout, repeated text lines, and background consistency [2309.06670]. The output is a refined shadow probability mask $M(x,y)\in[0,1]$ [2309.06670].

The Cascaded Fusion Refiner uses the original shadowed image together with the detected mask in a U-shaped encoder-decoder with Convolutional Depth Grouped Fusion Net blocks, multi-head attention, a dual-gated feed-forward network, and spatial pyramid pooling [2309.06670]. The loss is
$$
\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{MSE}} + 0.3 \times \mathcal{L}_{\text{SSIM}} + 0.7 \times \mathcal{L}_{P},
$$
combining pixel-wise reconstruction, SSIM, and perceptual loss [2309.06670]. On the RDD dataset, ShaDocFormer achieved PSNR $29.46$, SSIM $0.92$, and RMSE $8.9$; on the Kligler dataset it achieved PSNR $26.36$, SSIM $0.90$, and RMSE $13.17$ [2309.06670]. In the ablation reported in the text, removing the detector degrades PSNR from $29.46$ to $25.39$ and increases RMSE, which the paper uses to attribute a substantial role to the shadow-attentive masking stage [2309.06670].

ARGAN provides an earlier explicit learnable shadow attention mechanism in a recurrent GAN [1908.01323]. At each progressive step $i$, a shadow attention detector produces a soft attention map $A_i$, supervised against a shadow matte $M$ using
$$
L_{\text{det}} = \sum_{i=1}^{N} \beta_i \, V_{\text{MSE}}(A_i, M),
$$
with $\beta_i = 0.7^{N-i+1}$ and $N=3$ in practice [1908.01323]. A shadow removal encoder then produces a residual map that is multiplied by $A_i$ so that the correction is concentrated on shadow regions [1908.01323]. On shadow detection, ARGAN achieved BER $3.09$ on SBU, $3.76$ on UCF, and $2.01$ on ISTD, while ARGAN+SS improved these to $2.56$, $3.03$, and $1.75$, respectively [1908.01323]. On shadow removal, ARGAN achieved RMSE $5.74$ on SRD and $6.68$ on ISTD over the entire image, with ARGAN+SS improving to $5.31$ and $5.89$ [1908.01323].

DocShaDiffusion moves document shadow removal into latent-space diffusion and implements shadow awareness without explicit self-attention layers [2507.01422]. Its Shadow Soft-mask Generation Module produces a continuous mask $I^m$ from a single shadowed image using grayscale conversion, dilation, median filtering, darkest-pixel statistics, and a final median filtering step [2507.01422]. The mask then modulates both the initial noisy state and the diffusion process through a Shadow Mask-aware Guided Diffusion Module. The forward and reverse SDEs are
$$
dx = \theta_t (\mu - x)\, dt - \sigma_t (1 - I^m)\, dw
$$
and
$$
dx = \Big[ \theta_t (\mu - x) - \sigma_t^2 \nabla_x \log p_t(x) \Big]\, dt - \sigma_t (1 - I^m)\, d\hat{w},
$$
so stochasticity is spatially modulated by the shadow mask [2507.01422]. Training uses
$$
\mathcal{L}_{\text{total}} = \lambda\, \mathcal{L}_{\text{diff}} + (1-\lambda)\,\mathcal{L}_{\text{fea}},
$$
with $\lambda = 0.5$, where $\mathcal{L}_{\text{fea}}$ is a shadow-robust perceptual feature loss built from weighted VGG-19 slices [2507.01422]. A plausible implication is that this realizes shadowAttn at the level of generative dynamics rather than feature-level dot-product attention.

## 5. Robustness analysis, synthetic data, and region-adaptive perturbation

A robustness-centered branch of the literature studies shadow-aware attacks and defenses on restoration networks themselves. The shadow-adaptive adversarial attack for shadow removal argues that a uniform perturbation budget is mismatched to the spatially varying illumination of shadow images [2403.10076]. Instead of constraining $\|\delta\|_\infty \le \epsilon$, it uses an intensity-normalized constraint
$$
\left\| \frac{\delta}{I} \right\|_\infty \le \epsilon,
$$
equivalently $|\delta^{(i)}| \le \epsilon I^{(i)}$ per pixel [2403.10076]. The attack is optimized with PGD for $T=20$ iterations and step size $\alpha=\epsilon/4$, with adaptive budgets $\epsilon_a \in \{8/255,16/255\}$ in the direct comparison and $\epsilon_a \in \{1/255,2/255,4/255,8/255,16/255\}$ in the robustness benchmark [2403.10076].

The paper also proves an $\ell_1$-energy equivalence result: choosing the uniform budget as $\epsilon_u = \epsilon_a \bar I$ makes the maximum achievable average $\ell_1$ perturbation equal under uniform and adaptive attacks [2403.10076]. On ShadowFormer, for ISTD with $\epsilon_a = 8/255$, uniform and adaptive attacks yield PSNR $19.40$ versus $19.68$ and SSIM $0.9156$ versus $0.9168$; for ISTD+ at the same budget they yield PSNR $26.44$ versus $27.28$ and SSIM $0.9404$ versus $0.9395$ [2403.10076]. The attack strength is therefore comparable, but the adaptive attack is visually less conspicuous in shadow regions because $\delta/I$ is more uniform [2403.10076]. The broader benchmark shows that as $\epsilon_a$ increases to $16/255$, all evaluated models on ISTD drop by approximately $10$ dB in PSNR, and robustness differs systematically between shadow and non-shadow regions [2403.10076].

Synthetic data have played a major role in making shadow-aware models trainable. SynShadow provides a large-scale synthetic dataset of shadow image, shadow-free image, and matte triplets, built from $10{,}000$ shadow matte images synthesized from 3D models and approximately $1{,}800$ shadow-free background images from USR, with on-the-fly composition governed by a physically grounded affine illumination model [2101.01713]. The underlying relation is
$$
I^{lit}_k = \alpha_k + \gamma I^{dark}_k,
$$
and the final shadow image is formed by alpha composition,
$$
x^{s}_{ijk} = (1 - m_{ij}) x^{ns}_{ijk} + m_{ij} x^{dark}_{ijk},
$$
with non-binary mattes capturing penumbra [2101.01713]. Pre-training on SynShadow improves several existing models. For example, on ISTD+, SP+M improves all-image RMSE from $4.4$ to $4.0$ and DHAN improves from $5.2$ to $4.6$ after SynShadow pre-training and fine-tuning; on SRD+, SP+M improves from $5.3$ to $5.2$ and DHAN from $7.4$ to $6.6$ [2101.01713]. For shadow detection on ISTD, BDRAR-ft and DSDNet++-ft achieve all-image BER $1.10$ and $1.09$, compared with prior DSDNet at $2.17$ [2101.01713].

The defense side of the original physical shadow attack also includes shadow-aware adversarial training. Augmenting training images with random shadow polygons and random shadow intensity $k$ yields robust models $\text{LISA-CNN}_{rob}$ and $\text{GTSRB-CNN}_{rob}$ [2203.03818]. At $k=0.43$, on LISA, clean accuracy changes from $99.63\%$ to $99.56\%$, robustness increases from $1.73\%$ to $40.93\%$, and average queries until success increase from approximately $91$ to approximately $850$ [2203.03818]. On GTSRB, clean accuracy changes from $99.00\%$ to $98.91\%$, robustness increases from $9.53\%$ to $25.57\%$, and queries increase from approximately $127$ to approximately $465$ [2203.03818]. This suggests that even simple random-shadow augmentation can substantially harden a classifier against naturalistic shadow perturbations.

## 6. shadowAttn as a named sparse attention module for mobile SoCs

A separate usage appears in systems research, where shadowAttn is the name of a dynamic sparse attention layer for LLM inference on mobile SoCs [2508.16703]. The problem setting is not shadow removal or illumination, but the mismatch between quantization-sensitive attention and NPU hardware: in existing on-device frameworks, attention often falls back from the NPU to CPU/GPU because activations $Q$, $K$, and $V$ are much more quantization-sensitive than weights, and mobile NPUs typically support only per-tensor static quantization [2508.16703]. Empirically, putting full attention on the NPU with INT8 produces an average accuracy drop of approximately $18$ percentage points on mobile tasks [2508.16703].

shadowAttn addresses this by splitting attention into a pilot estimation phase and an exact sparse phase. Standard attention is
$$
\text{Attention}(Q, K, V) = \text{softmax}\left( \frac{Q K^\top}{\sqrt{d_k}} + \text{mask} \right) V.
$$
shadowAttn first computes an approximate dense score matrix $\tilde S = Q_{\text{INT8}}K_{\text{INT8}}^\top$ on the NPU, uses top-$k$ selection per head on CPU/GPU to identify important positions, and then computes exact high-precision attention only on those retained tokens [2508.16703]. The kept-token ratio is controlled globally and per head. Head-specific ratios are derived from offline head and layer importance measurements on a calibration set of $128$ WikiText-2 samples, with a formula that allocates larger kept-token ratios to more important heads [2508.16703].

Several system techniques make the design practical. NPU compute graph bucketing precompiles multiple static Q·K graphs for different scale-factor buckets; the paper uses $9$ buckets and reports that each Q·K graph is only $10$–$100$ KB [2508.16703]. A head-wise NPU–CPU/GPU pipeline overlaps three stages per head: NPU Q·K estimation, CPU/GPU top-$k$, and CPU/GPU sparse QKV [2508.16703]. With fused NPU launches, a single-head Q·K takes $2$ ms on MI14, two fused heads take $3$ ms, and four fused heads take $4$ ms [2508.16703]. A global sparsity ratio of $20\%$ is identified as a knee point giving minimal accuracy loss with substantial latency reduction [2508.16703].

The reported outcomes are primarily systems metrics. Compared with full float attention on CPU/GPU, shadowAttn yields an average accuracy of $36.4$ versus $36.8$, a drop of only $0.4$ percentage points across four models and three datasets [2508.16703]. It delivers up to $6.9\times$ and on average $3.5\times$ speedup for attention kernel latency, and up to $4.5\times$ and on average $2.9\times$ end-to-end speedup on prefill-heavy workloads [2508.16703]. On Redmi K60, single attention kernel energy drops from $3.72$ J to $0.66$ J for PhoneLM-0.5B and from $8.59$ J to $1.12$ J for PhoneLM-1.5B when comparing full CPU/GPU attention with shadowAttn [2508.16703]. In this usage, “shadow” refers not to illumination but to a low-precision pilot computation that “shadows” the exact attention to identify important tokens.

This second lineage is terminologically separate from shadow-aware vision models, yet structurally related: both senses of shadowAttn use a lightweight mechanism to decide where expensive computation or representational capacity should be concentrated. In vision, that selectivity is driven by shadow masks, boundaries, non-shadow references, or luminance-adaptive perturbation structure [2302.01650] [2411.14201] [2403.10076]. In mobile LLM systems, it is driven by pilot attention scores used to retain only a tiny portion of tokens [2508.16703].

Source: https://www.emergentmind.com/topics/shadowattn