ISALux: Illumination & Segmentation Transformer

• ISALux is a transformer-based low-light image enhancement method that integrates illumination and semantic priors for region-specific exposure correction.
• It employs a compact U-shaped encoder–decoder with HISA-MSA for prior-guided attention and a Mixture-of-Experts feed-forward design to tackle heterogeneous degradations.
• Extensive ablations and benchmarks demonstrate that ISALux achieves superior PSNR/SSIM with high parameter efficiency and faster inference compared to state-of-the-art methods.

Searching arXiv for the specific paper on ISALux to ground the article in the cited source. ISALux, short for Illumination and Segmentation Aware Transformer Employing Mixture of Experts for Low Light Image Enhancement, is a transformer-based method for low-light image enhancement (LLIE) that integrates illumination and semantic priors into a compact encoder–decoder architecture (Balmez et al., 25 Aug 2025). It is designed to recover bright images from dark inputs while addressing noise, color distortion, artifacts, loss of local detail, poor global visibility, and misinterpretation of scene structures. The model’s central claim is that LLIE should not be performed uniformly across the image: different regions require different exposure corrections, and structurally meaningful enhancement therefore benefits from explicit conditioning on both illumination and semantic context.

1. Research setting and problem formulation

ISALux is situated in LLIE, where the input is a low-light RGB image and the target is an enhanced image that is visually brighter while remaining structurally faithful and semantically coherent (Balmez et al., 25 Aug 2025). The paper motivates the problem by noting that low-light degradations affect not only perceptual quality but also downstream tasks such as object detection and segmentation. Within that framing, the method emphasizes two complementary forms of prior information.

The first is illumination awareness, which indicates where and to what extent the scene is underexposed. The second is semantic awareness, which identifies region- and object-level structure so that enhancement does not treat roads, sky, people, trees, or indoor objects identically. This suggests a departure from LLIE methods that rely on generic global enhancement or feature transformation alone. A plausible implication is that ISALux is best understood not as a purely restoration-oriented transformer, but as a prior-conditioned enhancement model in which exposure regulation and structural preservation are co-optimized.

The end-to-end enhancement function is written as

$$\mathcal{I}_E = \text{ISA-}\mathcal{T}(\mathcal{I}, \mathcal{P}^s, \mathcal{P}^i),$$

where $\mathcal{I} \in \mathbb{R}^{H \times W \times 3}$ is the low-light input, $\mathcal{P}^i$ is the illumination prior, $\mathcal{P}^s$ is the semantic prior, and $\mathcal{I}_E \in \mathbb{R}^{H \times W \times 3}$ is the enhanced output.

2. Backbone architecture and prior construction

The backbone, denoted ISA-$\mathcal{T}$, is a U-shaped encoder-decoder transformer with two encoder levels, a bottleneck, and two decoder levels (Balmez et al., 25 Aug 2025). The input image is first projected by a $3 \times 3$ convolution into

$$\mathcal{F}_{in} \in \mathbb{R}^{H \times W \times C}.$$

The encoder produces

$$\mathcal{F}_{enc}^k \in \mathbb{R}^{\frac{H}{2^k} \times \frac{W}{2^k} \times C \cdot 2^k}, \quad k \in \{0,1\},$$

the bottleneck is

$$\mathcal{F}_{bot} \in \mathbb{R}^{\frac{H}{4} \times \frac{W}{4} \times 4C},$$

and the decoder reconstructs same-scale features through skip connections, yielding

$\mathcal{I} \in \mathbb{R}^{H \times W \times 3}$0

A final $\mathcal{I} \in \mathbb{R}^{H \times W \times 3}$1 convolution maps $\mathcal{I} \in \mathbb{R}^{H \times W \times 3}$2 to the enhanced RGB output.

The illumination prior is hand-derived from the input image:

$\mathcal{I} \in \mathbb{R}^{H \times W \times 3}$3

The text describes $\mathcal{I} \in \mathbb{R}^{H \times W \times 3}$4 as returning the maximum channel value for each pixel. The model then forms a three-level illumination pyramid,

$\mathcal{I} \in \mathbb{R}^{H \times W \times 3}$5

so that the prior is available at the spatial scales used by the transformer. At each scale, only a $\mathcal{I} \in \mathbb{R}^{H \times W \times 3}$6 convolution is used to align channel depth.

The semantic prior is produced externally:

$\mathcal{I} \in \mathbb{R}^{H \times W \times 3}$7

The paper specifies MobileNetV3, a parameter count of 11.03M, training on a subset of COCO, and a 21-channel class-probability map output. Unlike the illumination prior, the semantic prior is adapted to each transformer scale using strided convolutions with stride $\mathcal{I} \in \mathbb{R}^{H \times W \times 3}$8. This division is technically significant: illumination is a simple image-derived cue, whereas semantics is supplied by a separate segmentation backbone.

3. HISA-MSA: illumination- and semantics-aware self-attention

The central module of ISALux is HISA-MSA, the Hybrid Illumination and Semantics-Aware Multi-Headed Self-Attention block (Balmez et al., 25 Aug 2025). It differs from standard multi-head self-attention in five reported ways: it uses two parallel self-attention branches, injects illumination and semantic priors, combines them with learnable weighted fusion, augments $\mathcal{I} \in \mathbb{R}^{H \times W \times 3}$9 through LoRA, and replaces fixed attention scaling with a learnable temperature.

Given

$\mathcal{P}^i$0

a $\mathcal{P}^i$1 convolution produces

$\mathcal{P}^i$2

which is split into $\mathcal{P}^i$3, $\mathcal{P}^i$4, and $\mathcal{P}^i$5. Attention is then computed as

$\mathcal{P}^i$6

where $\mathcal{P}^i$7 is a learnable temperature.

The model uses two parallel attention outputs, $\mathcal{P}^i$8 and $\mathcal{P}^i$9, corresponding to illumination-aware and semantics-aware streams. The paper states that ISALux “employs two self-attention modules to independently process illumination and semantic features,” but does not fully specify whether the branches have separate parameterizations or whether they diverge only through prior modulation. That omission matters for strict reproducibility.

The prior-guided enrichment is expressed as

$\mathcal{P}^s$0

with $\mathcal{P}^s$1 denoting element-wise multiplication and $\mathcal{P}^s$2 learnable fusion weights. The text conceptually describes this as selective enrichment, but the explicit equation is a weighted sum of separately modulated branches rather than a formal cross-attention construction. A careful reading therefore supports the weaker claim that the branches interact at the fusion stage.

Within each transformer block, HISA-MSA is followed by an MoE-based feed-forward block, LayerNorm, and residual connections. The paper gives the block-level relations as

$\mathcal{P}^s$3

and

$\mathcal{P}^s$4

The source typesetting is explicitly noted as corrupted, but the intended composition is clear: prior-aware attention first, MoE feed-forward processing second.

4. Mixture-of-Experts feed-forward design and LoRA adaptation

ISALux replaces the usual dense FFN with a sparse Mixture-of-Experts (MoE)-based FFN (Balmez et al., 25 Aug 2025). The rationale is that LLIE involves heterogeneous degradations: some regions require luminance correction, others detail recovery, noise handling, or structural refinement. Sparse expert selection is therefore introduced as a specialization mechanism.

Each expert is an independent CNN:

$\mathcal{P}^s$5

where $\mathcal{P}^s$6 is GELU. The output remains in $\mathcal{P}^s$7.

Routing is based on globally pooled features:

$\mathcal{P}^s$8

with $\mathcal{P}^s$9. The model then performs Top-k expert selection:

$\mathcal{I}_E \in \mathbb{R}^{H \times W \times 3}$0

and

$\mathcal{I}_E \in \mathbb{R}^{H \times W \times 3}$1

the final MoE output is

$\mathcal{I}_E \in \mathbb{R}^{H \times W \times 3}$2

The paper states that top-2 experts are used in the final model and that a third expert increased computational cost with only about 0.05 dB PSNR gain. It does not explicitly state the total number of experts $\mathcal{I}_E \in \mathbb{R}^{H \times W \times 3}$3, and it does not mention load balancing loss, auxiliary routing regularizers, entropy penalties, or capacity constraints.

LoRA is applied inside HISA-MSA to the attention projections. After reshaping

$\mathcal{I}_E \in \mathbb{R}^{H \times W \times 3}$4

the low-rank updates are

$\mathcal{I}_E \in \mathbb{R}^{H \times W \times 3}$5

with

$\mathcal{I}_E \in \mathbb{R}^{H \times W \times 3}$6

and

$\mathcal{I}_E \in \mathbb{R}^{H \times W \times 3}$7

The adapted projections are then

$\mathcal{I}_E \in \mathbb{R}^{H \times W \times 3}$8

The paper frames this as a way to improve enhancement under varying illumination and to reduce overfitting to dataset-specific lighting patterns. The reported ablation on LOL shows Illum + Seg (no LoRA) at 27.41 / 0.880 and Illum + Seg at 27.63 / 0.881, implying about 0.22 dB PSNR and 0.001 SSIM gain.

5. Optimization, losses, and evaluation protocol

Training is described on paired LOL images, using random $\mathcal{I}_E \in \mathbb{R}^{H \times W \times 3}$9 patches, the Adam optimizer, $\mathcal{T}$0, $\mathcal{T}$1, 300k iterations, and a staged learning-rate schedule (Balmez et al., 25 Aug 2025). The initial learning rate is

$\mathcal{T}$2

it increases to

$\mathcal{T}$3

after 92k iterations, decreases back to

$\mathcal{T}$4

by 208k, and finally decays to

$\mathcal{T}$5

at the end. The batch size is 8, and augmentation consists of random rotation and random flipping.

The loss is a weighted combination of three terms: an $\mathcal{T}$6 reconstruction loss, a perceptual loss using pretrained VGG-19, and an MS-SSIM loss. The explicit components are

$\mathcal{T}$7

$\mathcal{T}$8

and

$\mathcal{T}$9

The total loss is described only as a weighted sum,

$3 \times 3$0

but the coefficients are not reported.

The evaluation suite covers paired, synthetic, static-scene, blur-corrupted, and no-reference settings. The listed datasets are LOL, LOL-v2-real, LOL-v2-synthetic, SDSD indoor/outdoor, LOL-Blur, and the no-reference datasets LIME, MEF, DICM, and NPE. Metrics are PSNR and SSIM on reference datasets and NIQE on no-reference datasets. At the same time, the implementation details state only that training data consists of patches cropped from paired LOL images; the paper does not state multi-dataset joint training. It also does not specify the base channel count $3 \times 3$1, the number of transformer blocks per stage, the number of attention heads, the LoRA rank $3 \times 3$2, the total number of experts $3 \times 3$3, the exact loss weights, the hardware used, or FLOPs and memory usage.

6. Reported results, ablations, and stated limitations

The principal quantitative result is that ISALux reports the best average paired-benchmark score among the methods shown, with 30.08 dB / 0.910 across LOL-v1, LOL-v2-R, LOL-v2-S, SDSD-in, and SDSD-out (Balmez et al., 25 Aug 2025). It does so with 1.85M parameters, compared with 44.04M for GLARE and 1.61M for Retinexformer. The paper states that this is about 96% more parameter efficient than GLARE. Average inference time on LOL test images at $3 \times 3$4 is reported as 105 ms for ISALux, versus 300 ms for Retinexformer, 378 ms for LLFlow, and 650 ms for GLARE. This suggests practical deployment potential, although the hardware is not specified.

Setting	Reported result	Comparator context
Avg paired benchmarks	30.08 / 0.910	GLARE: 29.42 / 0.905; Retinexformer: 28.71 / 0.884
Model size	1.85M	GLARE: 44.04M; Retinexformer: 1.61M
Runtime at $3 \times 3$5	105 ms	Retinexformer: 300 ms; LLFlow: 378 ms; GLARE: 650 ms
LOL-Blur	28.01 / 0.903	Retinexformer: 22.90 / 0.824; VQCNIR: 27.79 / 0.875
No-reference NIQE average	3.34	GLARE: 3.99

The paper also reports dataset-specific paired results: 27.63 / 0.881 on LOL-v1, 29.76 / 0.908 on LOL-v2-R, 30.78 / 0.956 on LOL-v2-S, 30.67 / 0.909 on SDSD-in, and 31.58 / 0.895 on SDSD-out. On no-reference evaluation, it reports NIQE scores of 3.58 on MEF, 3.91 on LIME, 3.21 on DICM, 3.40 on NPE, and an average of 3.34. The paper states that ISALux ranks first on MEF, DICM, and NPE, and second on LIME, while also noting a formatting inconsistency: the tabulated LIME value is numerically the best among the values shown.

The ablations attribute the gains primarily to prior integration and the hybrid loss. On LOL, No Priors gives 27.04 / 0.870, Illum Only gives 27.15 / 0.878, Segmentation Only gives 27.36 / 0.879, Illum + Seg (no LoRA) gives 27.41 / 0.880, and Illum + Seg gives 27.63 / 0.881. For loss terms, $3 \times 3$6 only gives 26.95 / 0.850, $3 \times 3$7 gives 27.08 / 0.869, $3 \times 3$8 gives 27.14 / 0.872, and $3 \times 3$9 gives 27.63 / 0.881.

Ablation setting	PSNR	SSIM
No Priors	27.04	0.870
Illum Only	27.15	0.878
Segmentation Only	27.36	0.879
Illum + Seg (no LoRA)	27.41	0.880
Illum + Seg	27.63	0.881

Several limitations are stated or directly implied by omissions in the specification. The semantic prior is externally generated by MobileNetV3 rather than learned end-to-end with enhancement, which introduces dependence on segmentation quality under low-light input. The paper does not provide an explicit MoE-vs-dense-FFN ablation, does not discuss routing regularization, and does not report the total number of experts, the LoRA rank, or the number of attention heads. It also acknowledges that future work should address noise patterns and low-light artifacts. These caveats do not negate the reported gains, but they delimit what can presently be concluded about the relative contributions of sparse routing, low-rank adaptation, and prior conditioning.

In technical terms, the key contribution of ISALux is the claim that LLIE benefits from conditioning transformer attention on both how dark a region is and what that region represents. Within the evidence reported, that claim is supported by the prior ablations, the compact parameter count of 1.85M, and the cross-benchmark results on paired, no-reference, and low-light-plus-blur settings (Balmez et al., 25 Aug 2025).