ModalFormer: Multimodal Low-Light Image Enhancement

• ModalFormer is a multimodal architecture that enhances low-light images by leveraging a U-shaped Cross-modal Transformer and nine auxiliary modality pathways.
• It utilizes modality-specific self-attention maps fused into hybrid attention to restore contrast, color, and structural details in degraded images.
• Experiments show state-of-the-art performance on multiple benchmarks while achieving competitive parameter efficiency and robust results in noisy conditions.

ModalFormer is a multimodal architecture for low-light image enhancement (LLIE) that combines a U-shaped Cross-modal Transformer with nine auxiliary modality pathways in order to restore visibility, contrast, structural detail, and color fidelity under insufficient illumination. The method is presented as the first large-scale multimodal framework for LLIE to fully exploit nine auxiliary modalities, and it centers on a Cross-modal Multi-headed Self-Attention mechanism that constructs modality-specific attention maps and fuses them into hybrid attention maps for RGB restoration (Brateanu et al., 27 Jul 2025).

1. Problem formulation and defining contributions

LLIE addresses degradations induced by insufficient illumination, including noise, poor contrast, color shift, and structural loss. In the formulation used by ModalFormer, a low-light RGB image $I \in \mathbb{R}^{H \times W \times 3}$ is mapped to an enhanced image $\hat{I}$ through a multimodal restoration pipeline. The paper argues that purely RGB, pixel-level methods remain limited under compound degradations such as noise, blur, and low illumination because they do not exploit strong contextual priors, even when based on Vision Transformers. ModalFormer is designed around the premise that semantics, segmentation boundaries, geometry, and color statistics provide complementary priors for restoration (Brateanu et al., 27 Jul 2025).

The framework couples two major components. The first is a Cross-modal Transformer, denoted CM-$\mathcal{T}$, which performs the enhancement itself. The second is a set of nine auxiliary subnetworks, $U_1,\ldots,U_9$, which reconstruct modality features and inject them into CM-$\mathcal{T}$ at matched encoder-decoder levels. The principal architectural claim is that the Cross-modal Multi-headed Self-Attention module, or CM-MSA, produces distinct attention maps for each modality and then fuses them into a hybrid attention map. The principal empirical claim is state-of-the-art performance across multiple benchmark datasets, together with favorable parameter and runtime trade-offs (Brateanu et al., 27 Jul 2025).

At a high level, a fixed Modality Extractor based on 4M-21 produces eight modality maps $$M_1,\ldots,M_8 \in \mathbb{R}^{H \times W \times 22}$$ and one luminance map $M_9 \in \mathbb{R}^{H \times W \times 1}$. Each modality is refined by its own auxiliary U-Net, yielding reconstructed features $\{\hat{M}_1,\ldots,\hat{M}_9\}$. CM-$\mathcal{T}$ then lifts the RGB image into feature space and injects the reconstructed modality features at multiple scales through Multimodal Transformer Blocks (MMTBs), ultimately producing $\hat{I}$.

2. Auxiliary modalities and reconstruction subnetworks

A defining property of ModalFormer is the use of nine auxiliary modalities. Eight are extracted by the pre-trained MIM foundation model 4M-21 and PCA-reduced to low-dimensional, rank-preserving feature maps; the ninth is luminance computed by NTSC conversion. Spatial resolution matches the input for all modalities, and normalization beyond PCA is not specified (Brateanu et al., 27 Jul 2025).

Modality	Type	Source or preprocessing
CLIP features	Feature embeddings	4M-21
ImageBind features	Feature embeddings	4M-21
DINOv2 features	Feature embeddings	4M-21
SAM Instances	Segmentation information	4M-21
SAM Edges	Segmentation or boundary information	4M-21
Depth Map	Geometric information	4M-21
Surface Normals	Geometric information	4M-21
Color Palette	Color information	4M-21
Luminance	Color or illumination information	NTSC conversion

For modalities 1-8, the resulting tensors satisfy $\hat{I}$0. For depth and surface normals, the extractor yields RGB-like or higher-dimensional encodings suitable for fusion. The color palette is represented as a five-color RGB array. Luminance forms a single-channel map $\hat{I}$1.

Each modality is processed by a dedicated auxiliary subnetwork $\hat{I}$2. These subnetworks are four-level convolutional U-Nets with skip connections and are explicitly deeper than the three-stage CM-$\hat{I}$3, so that the auxiliary towers and the main transformer have independent bottlenecks. Each $\hat{I}$4 takes $\hat{I}$5 as input and outputs $\hat{I}$6, where $\hat{I}$7 is aligned to the channel dimension used by CM-$\hat{I}$8 at the corresponding injection scale. The exact numerical value of $\hat{I}$9 is not specified.

The auxiliary subnetworks are trained jointly with CM-$\mathcal{T}$0 using the multimodal reconstruction loss $\mathcal{T}$1. There is no separate pretraining for $\mathcal{T}$2; by contrast, 4M-21 and PCA are fixed and non-differentiable. The paper states that the supervision target for each auxiliary modality is the “corresponding ground truth.” A plausible implication is that these targets are re-extracted from the normal-light reference through the same extractor, but this step is not explicitly stated in the paper (Brateanu et al., 27 Jul 2025).

3. Cross-modal Transformer architecture

The Cross-modal Transformer is a U-shaped, three-stage encoder-decoder with skip connections and a bottleneck. The design is fully convolutional rather than a patchified ViT: a $\mathcal{T}$3 convolution lifts the input image into feature space, after which attention is computed over pixel tokens within MMTBs. No explicit positional encoding is specified (Brateanu et al., 27 Jul 2025).

The encoder begins by mapping $\mathcal{T}$4 to $\mathcal{T}$5 using a $\mathcal{T}$6 convolution. It then applies one MMTB, followed by a $\mathcal{T}$7 stride-2 downsampling operation, then two MMTBs, then another $\mathcal{T}$8 stride-2 downsampling operation, producing $\mathcal{T}$9. The bottleneck contains two additional MMTBs. The decoder mirrors the encoder using $U_1,\ldots,U_9$0 stride-2 deconvolutions and MMTBs, with skip connections that concatenate encoder and decoder streams at matched scales. A final $U_1,\ldots,U_9$1 convolution maps $U_1,\ldots,U_9$2 to the enhanced image $U_1,\ldots,U_9$3.

Multimodal feature injection occurs at multiple resolutions. Reconstructed modality features are injected into MMTBs at the $U_1,\ldots,U_9$4 and $U_1,\ldots,U_9$5 scales in both encoder and decoder. At the bottleneck resolution $U_1,\ldots,U_9$6, features propagated through the downsampling and upsampling paths are combined and injected into the bottleneck MMTB.

The basic computational unit is the Multimodal Transformer Block. Given an input feature map $U_1,\ldots,U_9$7 and injected modality features $U_1,\ldots,U_9$8 resized to the current scale, the block applies a residual CM-MSA stage followed by a residual feed-forward stage:

$U_1,\ldots,U_9$9

$\mathcal{T}$0

The feed-forward network uses GELU activations and an expansion ratio $\mathcal{T}$1:

$\mathcal{T}$2

Here the first $\mathcal{T}$3 convolution expands to $\mathcal{T}$4, the $\mathcal{T}$5 convolution extracts spatial structure, and the final $\mathcal{T}$6 convolution compresses back to $\mathcal{T}$7. Exact values for $\mathcal{T}$8, the number of heads $\mathcal{T}$9, MLP ratios beyond $$M_1,\ldots,M_8 \in \mathbb{R}^{H \times W \times 22}$$0, and block counts per stage are not numerically specified.

4. Cross-modal Multi-headed Self-Attention and training objective

CM-MSA is the mechanism that integrates RGB features with modality-specific features. The paper situates it relative to standard scaled dot-product attention, where for tokens $$M_1,\ldots,M_8 \in \mathbb{R}^{H \times W \times 22}$$1, attention is computed as

$$M_1,\ldots,M_8 \in \mathbb{R}^{H \times W \times 22}$$2

with $$M_1,\ldots,M_8 \in \mathbb{R}^{H \times W \times 22}$$3, $$M_1,\ldots,M_8 \in \mathbb{R}^{H \times W \times 22}$$4, and $$M_1,\ldots,M_8 \in \mathbb{R}^{H \times W \times 22}$$5 (Brateanu et al., 27 Jul 2025).

ModalFormer instead uses Restormer-style transposed attention computed in channel space, reducing complexity from $$M_1,\ldots,M_8 \in \mathbb{R}^{H \times W \times 22}$$6 to $$M_1,\ldots,M_8 \in \mathbb{R}^{H \times W \times 22}$$7. The input feature map is reshaped to $$M_1,\ldots,M_8 \in \mathbb{R}^{H \times W \times 22}$$8 and split into $$M_1,\ldots,M_8 \in \mathbb{R}^{H \times W \times 22}$$9 heads, although $M_9 \in \mathbb{R}^{H \times W \times 1}$0 is not specified. A shared $M_9 \in \mathbb{R}^{H \times W \times 1}$1 convolution projects $M_9 \in \mathbb{R}^{H \times W \times 1}$2 to RGB queries, keys, and values. RGB self-attention is then formed in channel space:

$M_9 \in \mathbb{R}^{H \times W \times 1}$3

$M_9 \in \mathbb{R}^{H \times W \times 1}$4

where the exact scaling for RGB is not explicitly given.

For each modality $M_9 \in \mathbb{R}^{H \times W \times 1}$5, a dedicated $M_9 \in \mathbb{R}^{H \times W \times 1}$6 convolution maps $M_9 \in \mathbb{R}^{H \times W \times 1}$7 to $M_9 \in \mathbb{R}^{H \times W \times 1}$8, and the modality attention map is

$M_9 \in \mathbb{R}^{H \times W \times 1}$9

with learned $\{\hat{M}_1,\ldots,\hat{M}_9\}$0. The modality maps are then fused multiplicatively with learned scalar gates $\{\hat{M}_1,\ldots,\hat{M}_9\}$1:

$\{\hat{M}_1,\ldots,\hat{M}_9\}$2

The paper states that the product is the matrix product over $\{\hat{M}_1,\ldots,\hat{M}_9\}$3 maps and that the second softmax stabilizes $\{\hat{M}_1,\ldots,\hat{M}_9\}$4. The final output reweights the RGB-attended features by the hybrid attention:

$\{\hat{M}_1,\ldots,\hat{M}_9\}$5

This construction yields modality-specific projections $\{\hat{M}_1,\ldots,\hat{M}_9\}$6 and $\{\hat{M}_1,\ldots,\hat{M}_9\}$7 and introduces learned scalars $\{\hat{M}_1,\ldots,\hat{M}_9\}$8 and $\{\hat{M}_1,\ldots,\hat{M}_9\}$9 to improve robustness when some modalities are unreliable or missing. A common misconception would be to treat ModalFormer as simple feature concatenation; the ablation study instead frames CM-MSA as a structured attention-level fusion mechanism distinct from additive, concatenative, or partial replacement strategies.

The total training loss is

$\mathcal{T}$0

with

$\mathcal{T}$1

Implementation details specify $\mathcal{T}$2, $\mathcal{T}$3, and $\mathcal{T}$4. The component losses are

$\mathcal{T}$5

$\mathcal{T}$6

$\mathcal{T}$7

where $\mathcal{T}$8 denotes VGG-19 features. The multimodal reconstruction loss $\mathcal{T}$9 is defined as the average MSE between each predicted modality $\hat{I}$0 and its corresponding ground truth across all nine modalities.

5. Experimental evaluation and ablation evidence

Training uses paired low/normal-light datasets: LOL-v1, LOL-v2-real, LOL-v2-synthetic, SDSD-indoor, and SDSD-outdoor. The exact splits are 485/15 for LOL-v1, 689/100 for LOL-v2-real, 900/100 for LOL-v2-synthetic, 62/6 for SDSD-indoor, and 116/10 for SDSD-outdoor. No-reference evaluation uses LIME, NPE, MEF, and DICM. Preprocessing consists of random $\hat{I}$1 crops, random rotation, and flipping. Optimization uses PyTorch with Adam$\hat{I}$2 for 300K iterations; the learning rate starts at $\hat{I}$3 and decays to $\hat{I}$4 with cosine annealing; batch size is 6 (Brateanu et al., 27 Jul 2025).

On paired benchmarks, ModalFormer is reported as best on all five datasets in the main quantitative comparison. It achieves 27.97/0.897 on LOL-v1, 29.33/0.915 on LOL-v2-real, 30.15/0.951 on LOL-v2-syn, 31.37/0.917 on SDSD-indoor, and 31.73/0.904 on SDSD-outdoor in PSNR/SSIM. Relative to GLARE, the gains are reported as +0.62 dB and +0.014 SSIM on LOL-v1, +0.35 dB and +0.010 SSIM on LOL-v2-real, +0.31 dB in PSNR on LOL-v2-syn while SSIM ranks third at 0.951 compared with 0.958, +1.27 dB and +0.021 SSIM on SDSD-indoor, and +0.88 dB and +0.020 SSIM on SDSD-outdoor.

The parameter count in the main comparison is 19.81 M. The paper contrasts this with GLARE at 44.04 M, LL-Flow at 37.68 M, LL-SKF at 39.91 M, and IPT and EnGAN at above 100 M. A close value of 19.79 M appears in the ablation table, but the main state-of-the-art table lists 19.81 M.

On no-reference evaluation, ModalFormer achieves the best mean NIQE of 3.61 across MEF, LIME, DICM, and NPE, outperforming SNR-Net at 4.66, URetinex at 4.12, LLFlow at 4.29, LL-SKF at 4.24, RFR at 3.90, and GLARE at 4.00. It is also reported as lowest on MEF at 3.44.

Qualitative comparisons on paired and no-reference datasets attribute to ModalFormer sharper detail, better exposure balance, more faithful colors, and fewer artifacts than methods such as SNR-Net, LL-Flow, Retinexformer, GLARE, LL-SKF, and EnlightenGAN. Because these observations are taken from the paper’s qualitative figures, they should be understood as figure-grounded comparative characterizations rather than separately quantified perceptual metrics.

The ablation study isolates several design choices. For multimodal feature injection, a plain MSA baseline reaches 26.73 dB. Alternative fusion strategies yield 26.44 dB for Add, 26.93 dB for Concat, 27.25 dB for Q-replace, and 27.43 dB for V pointwise multiplication, while CM-MSA reaches 27.97 dB, a gain of +1.24 dB over plain MSA. For modality groups, four categories are evaluated: Feature Embeddings, Segmentation, Geometric, and Color. Each improves performance; feature embeddings contribute +0.85 dB, geometric cues +0.69 dB, color +0.60 dB, and segmentation +0.53 dB, with all four together reaching 27.97 dB. For the hybrid loss, adding MS-SSIM and perceptual loss to $\hat{I}$5 improves by approximately +0.5 dB, and adding $\hat{I}$6 yields a further +0.45 dB.

The comparison with previous multimodal LLIE baselines reports that ModalFormer surpasses earlier multimodal approaches by 4.20 dB on LOL-v1 and 6.69 dB on LOL-v2-real. The baselines listed include EYV, NeRCo, LLMR, DACLIP, GG-L, Four, and DMF. The paper interprets this as evidence for the value of scaling to nine modalities together with CM-MSA.

6. Positioning, limitations, reproducibility, and prospective extensions

Within LLIE, ModalFormer is positioned against RGB-only transformer-based methods and against earlier multimodal approaches. Its central distinction is that it injects external priors spanning perceptual embeddings, segmentation, geometry, and color through modality-specific attention maps rather than relying on RGB self-attention alone (Brateanu et al., 27 Jul 2025).

Several practical trade-offs are explicitly noted. Modality extraction through 4M-21 adds inference latency, and training requires multimodal reconstruction supervision through $\hat{I}$7. The use of non-differentiable PCA also reduces end-to-end transparency. These costs are balanced against reported efficiency figures: on LOL-v1 test images at $\hat{I}$8, GLARE achieves 27.35 dB in 650 ms, Retinexformer and LLFlow are approximately 25 dB in 300-380 ms, and ModalFormer reaches 27.97 dB in 728 ms total, decomposed into 395 ms for CM-$\hat{I}$9 and 295 ms for 4M-21 extraction. With parallel 4M-21 extraction, the total drops to approximately 443 ms while preserving 27.97 dB; the paper describes this as 0.62 dB higher than GLARE, 32% faster, and using 55% fewer parameters. FLOPs and memory are not reported.

The paper does not explicitly enumerate failure cases, and several implementation details remain unspecified, including exact widths $\hat{I}$00, the number of attention heads $\hat{I}$01, block counts per stage, FLOPs, mixed precision, random seeds, licenses, and exact evaluation scripts. Ground-truth modality formation for $\hat{I}$02 is also not explicitly described. These omissions may hinder exact architectural reproduction.

At the same time, the available evidence suggests robustness to severe noise, illumination artifacts, and varying color casts, based on SDSD results, no-reference evaluation, and the learned gates $\hat{I}$03 and $\hat{I}$04. This suggests resilience to imperfect modalities, although a dedicated stress-test across noise levels or domain shifts is not separately reported.

Reproducibility resources include pre-trained models and results at the project repository linked by the paper. The experimental setup identifies the datasets, exact train/test splits, hyperparameters, and an RTX 3090 for the complexity analysis. Those details provide a substantial but not exhaustive basis for replication.

The application scope described for LLIE includes night-time autonomous driving, surveillance, photography, and downstream vision systems sensitive to illumination, including detection and segmentation. Geometry- and segmentation-aware enhancement is highlighted as especially relevant for safety-critical and scene-understanding tasks. The paper also points toward broader applicability of the CM-MSA fusion principle to denoising, deblurring, dehazing, and low-light video enhancement. Future directions are not explicitly stated as formal claims, but the framework naturally suggests end-to-end learnable modality extraction, lightweight real-time variants, modality pruning, and domain adaptation as plausible extensions.