Illumination-Guided MSA in Low-Light Image Enhancement

• The paper introduces IG-MSA, a module that embeds a spatially varying illumination prior directly into the query for channel-wise self-attention.
• IG-MSA employs depthwise-separable convolutions and a learnable scalar α to calibrate attention scores, ensuring brightness restoration adapts to local illumination.
• Empirical ablations on LOL-v2-Real show that combining IG-MSA with SAI²E boosts PSNR by up to +0.83 dB, validating its effectiveness in non-uniform lighting scenarios.

Illumination-Guided Multi-head Self-Attention (IG-MSA) is the illumination-conditioned attention mechanism introduced in the SAIGFormer framework for low-light image enhancement, specifically to address non-uniform lighting scenarios such as backlit and shadow, which can otherwise appear as over-exposure or inadequate brightness restoration. Within SAIGFormer, IG-MSA leverages a spatially varying illumination map to calibrate lightness-relevant features, and it is formulated as the first sub-layer in each Spatially-Adaptive Illumination-Guided Transformer (SAIGT) block (Li et al., 21 Jul 2025).

1. Position within the SAIGT block

Within each SAIGT block, IG-MSA is applied before the dual-gated feed-forward network (DG-FFN). Denoting the input feature map at stage $i$ by $F_i\in\mathbb{R}^{H\times W\times C}$ and the corresponding illumination map by $I_{L_i}\in\mathbb{R}^{H_i\times W_i\times3}$, the block is defined as

$$F_i' \;=\; F_i \;+\;\mathrm{IG\text{-}MSA}\bigl(\mathrm{LN}(F_i),\,I_{L_i}\bigr),$$

$$F_{i+1}\;=\;F_i' \;+\;\mathrm{DG\text{-}FFN}\bigl(\mathrm{LN}(F_i')\bigr).$$

This places IG-MSA in the residual attention pathway, where it receives the LayerNorm-normalized feature tensor together with the stage-aligned illumination map. In functional terms, the module is not an auxiliary branch detached from the main representation stream; it directly participates in the feature update that precedes the feed-forward transformation. A plausible implication is that illumination guidance is injected at the point where contextual interactions are formed, rather than only at reconstruction time.

2. Feature projections and illumination embedding

Let $$\hat F=\mathrm{LN}(F)\in\mathbb{R}^{H\times W\times C}$$. IG-MSA first extracts query, key, and value embeddings by applying a pointwise convolution followed by a depthwise-separable convolution: $$T \;=\; W_d\bigl(W_p\,\hat F\bigr)\;\in\;\mathbb{R}^{H\times W\times(3C)}, \quad [\,Q,K,V\,]\;=\;\mathrm{split}_\mathrm{chan}(T).$$

Here, $W_p$ is a $1\times1$ convolution mapping $\mathbb{R}^{H\times W\times C}$ to $F_i\in\mathbb{R}^{H\times W\times C}$0, $F_i\in\mathbb{R}^{H\times W\times C}$1 is a $F_i\in\mathbb{R}^{H\times W\times C}$2 depthwise-separable convolution, and $F_i\in\mathbb{R}^{H\times W\times C}$3 partitions the $F_i\in\mathbb{R}^{H\times W\times C}$4 channels into three tensors $F_i\in\mathbb{R}^{H\times W\times C}$5.

The illumination signal is processed separately. The illumination map is first brought to the same spatial resolution as the feature representation and then channel-aligned: $F_i\in\mathbb{R}^{H\times W\times C}$6

The resulting illumination embedding is concatenated to the query: $F_i\in\mathbb{R}^{H\times W\times C}$7

The defining structural feature of IG-MSA is therefore not merely the presence of an illumination prior, but its explicit insertion into the query tensor. This means that illumination information modulates affinity formation rather than being fused only after attention has been computed.

3. Channel-wise self-attention formulation

To perform channel-wise self-attention, IG-MSA flattens the spatial dimensions so that $F_i\in\mathbb{R}^{H\times W\times C}$8: $F_i\in\mathbb{R}^{H\times W\times C}$9

The attention scores are then formed with a learnable scalar $I_{L_i}\in\mathbb{R}^{H_i\times W_i\times3}$0: $I_{L_i}\in\mathbb{R}^{H_i\times W_i\times3}$1

$I_{L_i}\in\mathbb{R}^{H_i\times W_i\times3}$2

$I_{L_i}\in\mathbb{R}^{H_i\times W_i\times3}$3

Finally, the attended representation is projected back to $I_{L_i}\in\mathbb{R}^{H_i\times W_i\times3}$4 channels and reshaped to the spatial layout: $I_{L_i}\in\mathbb{R}^{H_i\times W_i\times3}$5

Several aspects of this formulation are technically distinctive. First, the attention is channel-wise rather than spatial-token-wise: the interaction tensor has shape $I_{L_i}\in\mathbb{R}^{H_i\times W_i\times3}$6, not $I_{L_i}\in\mathbb{R}^{H_i\times W_i\times3}$7. Second, the extra $I_{L_i}\in\mathbb{R}^{H_i\times W_i\times3}$8 dimensions introduced by illumination guidance persist through the score formation and output aggregation steps before the final projection restores the original channel dimension. Third, the learnable scalar $I_{L_i}\in\mathbb{R}^{H_i\times W_i\times3}$9 explicitly calibrates the magnitude of the attention scores. In the paper’s summary, this is presented as a mechanism for fine-tuning the impact of illumination guidance on the attention scores (Li et al., 21 Jul 2025).

4. Forward-pass realization

The forward pass is specified procedurally as follows:

$$\hat F=\mathrm{LN}(F)\in\mathbb{R}^{H\times W\times C}$$1

This implementation-level view makes explicit that the illumination pathway is shallow but not trivial: it includes a $$F_i' \;=\; F_i \;+\;\mathrm{IG\text{-}MSA}\bigl(\mathrm{LN}(F_i),\,I_{L_i}\bigr),$$0 depthwise-separable convolution followed by a $$F_i' \;=\; F_i \;+\;\mathrm{IG\text{-}MSA}\bigl(\mathrm{LN}(F_i),\,I_{L_i}\bigr),$$1 convolution before concatenation. It also makes clear that the output projection is applied after attention has already mixed the original feature channels with the illumination-augmented query space. This suggests that the illumination signal is used to steer feature interactions while still allowing the final representation to remain dimensionally compatible with the residual backbone.

5. Distinction from vanilla multi-head self-attention

The paper contrasts IG-MSA with vanilla multi-head self-attention as used, for example, in ViT. Vanilla MSA constructs $$F_i' \;=\; F_i \;+\;\mathrm{IG\text{-}MSA}\bigl(\mathrm{LN}(F_i),\,I_{L_i}\bigr),$$2, $$F_i' \;=\; F_i \;+\;\mathrm{IG\text{-}MSA}\bigl(\mathrm{LN}(F_i),\,I_{L_i}\bigr),$$3, and $$F_i' \;=\; F_i \;+\;\mathrm{IG\text{-}MSA}\bigl(\mathrm{LN}(F_i),\,I_{L_i}\bigr),$$4 by three independent linear projections or $$F_i' \;=\; F_i \;+\;\mathrm{IG\text{-}MSA}\bigl(\mathrm{LN}(F_i),\,I_{L_i}\bigr),$$5 convolutions of $$F_i' \;=\; F_i \;+\;\mathrm{IG\text{-}MSA}\bigl(\mathrm{LN}(F_i),\,I_{L_i}\bigr),$$6, flattens spatial tokens, and computes $$F_i' \;=\; F_i \;+\;\mathrm{IG\text{-}MSA}\bigl(\mathrm{LN}(F_i),\,I_{L_i}\bigr),$$7. IG-MSA differs in four stated respects (Li et al., 21 Jul 2025).

First, $$F_i' \;=\; F_i \;+\;\mathrm{IG\text{-}MSA}\bigl(\mathrm{LN}(F_i),\,I_{L_i}\bigr),$$8, $$F_i' \;=\; F_i \;+\;\mathrm{IG\text{-}MSA}\bigl(\mathrm{LN}(F_i),\,I_{L_i}\bigr),$$9, and $$F_{i+1}\;=\;F_i' \;+\;\mathrm{DG\text{-}FFN}\bigl(\mathrm{LN}(F_i')\bigr).$$0 are generated via depthwise-separable convolutions to better preserve local context in $$F_{i+1}\;=\;F_i' \;+\;\mathrm{DG\text{-}FFN}\bigl(\mathrm{LN}(F_i')\bigr).$$1. Second, the mechanism performs channel-wise self-attention, treating the $$F_{i+1}\;=\;F_i' \;+\;\mathrm{DG\text{-}FFN}\bigl(\mathrm{LN}(F_i')\bigr).$$2 feature channels as tokens rather than attending over spatial tokens. Third, it explicitly concatenates the illumination embedding $$F_{i+1}\;=\;F_i' \;+\;\mathrm{DG\text{-}FFN}\bigl(\mathrm{LN}(F_i')\bigr).$$3 into the queries, allowing the spatially varying illumination map to modulate the dot-product affinities. Fourth, it introduces a learnable scale $$F_{i+1}\;=\;F_i' \;+\;\mathrm{DG\text{-}FFN}\bigl(\mathrm{LN}(F_i')\bigr).$$4 for fine-tuning the impact of illumination guidance on the attention scores.

The intended consequence of these modifications is also stated directly: they permit the module to steer the Transformer’s focus onto channels that are relevant to reconstructing the correct local brightness, thereby avoiding the over- or under-exposure artifacts common in uniform, end-to-end approaches. A common misconception would be to treat IG-MSA as a standard attention block with a side-channel prior added after attention. Its defining property is stronger than post-hoc conditioning: the illumination map participates in the query representation that determines affinity itself.

6. Empirical effects and ablation evidence

The ablation study reported on LOL-v2-Real isolates the contribution of IG-MSA and its interaction with the Spatially-Adaptive Integral Illumination Estimator (SAI$$F_{i+1}\;=\;F_i' \;+\;\mathrm{DG\text{-}FFN}\bigl(\mathrm{LN}(F_i')\bigr).$$5E). The reported PSNR/SSIM results are as follows (Li et al., 21 Jul 2025):

Configuration	PSNR	SSIM
baseline (pure U-shaped transformer)	23.01 dB	0.867
+ IG-MSA (with simple mean-RGB prior instead of SAI$$F_{i+1}\;=\;F_i' \;+\;\mathrm{DG\text{-}FFN}\bigl(\mathrm{LN}(F_i')\bigr).$$6E)	23.22 dB	0.871
+ IG-MSA + SAI$$F_{i+1}\;=\;F_i' \;+\;\mathrm{DG\text{-}FFN}\bigl(\mathrm{LN}(F_i')\bigr).$$7E (full illumination guidance)	23.84 dB	0.873

On these figures, IG-MSA alone yields a $$F_{i+1}\;=\;F_i' \;+\;\mathrm{DG\text{-}FFN}\bigl(\mathrm{LN}(F_i')\bigr).$$8 dB PSNR uplift over the baseline, and the combination of IG-MSA with SAI$$F_{i+1}\;=\;F_i' \;+\;\mathrm{DG\text{-}FFN}\bigl(\mathrm{LN}(F_i')\bigr).$$9E yields a $$\hat F=\mathrm{LN}(F)\in\mathbb{R}^{H\times W\times C}$$0 dB gain over the baseline. Qualitative residual-map visualizations in Fig. 8 are described as showing that IG-MSA produces residuals that tightly follow the non-uniform illumination pattern, whereas a model without IG-MSA brightens uniformly and loses detail in shadow and backlit regions.

Within the larger SAIGFormer framework, these observations are presented as evidence that illumination-guided attention is particularly relevant when enhancement quality depends on spatially adaptive brightness restoration rather than global exposure correction. A plausible implication is that the module’s value is highest in settings where the dominant failure mode is not insufficient global context, but incorrect local lightness calibration across heterogeneous illumination fields.