Configurable Multi-Bayer LoRA Module

• The paper introduces a sensor-aware adaptation mechanism that integrates LoRA groups into RAW-domain VAE and diffusion networks for diverse Bayer mosaics.
• The approach reduces computational overhead by fine-tuning only pattern-specific LoRA layers, improving image fidelity and mitigating artifacts.
• Experimental results reveal enhanced PSNR, SSIM, LPIPS, and FID metrics while maintaining parameter efficiency across RAW and sRGB domains.

The Configurable Multi-Bayer (CMB) LoRA module is a specialized adaptation strategy for image restoration models operating in the RAW domain, specifically within the RDDM (RAW Domain Diffusion Model) framework. The CMB LoRA module addresses the distinct challenge of sensor-dependent Bayer mosaic patterns—such as RGGB, BGGR, etc.—which pose problems for models trained on fixed mosaic configurations. By injecting low-rank adaptation (LoRA) groups into both the RAW-domain VAE (RVAE) encoder and the diffusion network, the CMB module enables plug-and-play adaptation to diverse sensor mosaics while maintaining optimization efficiency and strong prior generalization.

1. Motivation and Problem Statement

Bayer mosaics define the spatial arrangement of color filters on image sensors, resulting in unique RAW data structures per sensor. Conventional restoration networks trained on a fixed pattern (e.g., only RGGB) tend to perform suboptimally or require full re-training when deployed on images from other sensor mosaics. The CMB LoRA module is designed to resolve this lack of mosaic generality. By providing distinct LoRA parameter groups per Bayer pattern, the system can activate the appropriate adaptation group for any given sensor’s RAW data, sidestepping catastrophic interference and reducing the need for exhaustive network retraining.

2. CMB LoRA Module Architecture and Workflow

The CMB LoRA module operates through targeted architectural augmentation:

• Independent LoRA layers (low-rank adaptation modules) are inserted into the RVAE encoder and the diffusion backbone.
• Each Bayer pattern (e.g., RGGB, BGGR, etc.) is matched to a unique LoRA group. During inference or training, the system activates the relevant LoRA parameters as determined by the input pattern.
• Parameter updates are restricted to the active LoRA group and associated diffusion layers, with the RVAE decoder weights frozen during adaptation. This design leverages the pretrained prior while updating only a minimal parameter subset.

This modularity allows the feature extraction pipeline to remain broadly generalizable, while the pattern-specific LoRA layers absorb the structural idiosyncrasies introduced by different Bayer configurations.

3. Integration within RDDM: RAW-Domain VAE and Differentiable PTP

In the RDDM pipeline, the CMB LoRA module is integrated into two central components:

• RAW-domain VAE (RVAE) Encoder: Initially, the encoder is pretrained on linear image pairs to establish a latent space representation in the linear domain. The CMB LoRA layers are then introduced and fine-tuned per Bayer pattern, yielding feature codes that better match the statistics imposed by sensor mosaic structure.
• Pretrained Diffusion Network: The same adaptation strategy is applied, with distinct LoRA groups inserted to adjust the network’s transformation operators per mosaic variant.
• Differentiable Post Tone Processing (PTP) Module: After diffusion-driven restoration and decoding, the linear domain output is tone-mapped to sRGB via a differentiable module. Gradients are thus propagated synergistically across RAW and sRGB domains, supporting joint fidelity and perceptual optimization.

The objective for the end-to-end system is framed as:

$$θ^* = \operatorname*{argmin}_{θ} \mathbb{E}_{x^{RAW}, x^{LIN} \in S}\left[ L(G_{θ}^{RAW}(x^{RAW}), x^{LIN}) \right]$$

where $G_{θ}^{RAW}$ comprises the RVAE encoder + CMB LoRA, diffusion network, and a fixed RVAE decoder.

4. Training Objectives and Relevant Equations

The loss function combines components to enforce both RAW-domain and sRGB-domain quality:

$$L = L_{VSD}(\hat{X}_H^{(lin)}, X_H^{(lin)}) + \lambda_1 L_{RAW}(\hat{X}_H^{(lin)}, X_H^{(lin)}) + \lambda_2 L_{rgb}\left(F_{PTP}(\hat{X}_H^{(lin)}), F_{PTP}(X_H^{(lin)})\right)$$

with:

• $\hat{X}_H^{(lin)}$: reconstructed linear image
• $F_{PTP}(\cdot)$: post tone processing operator mapping linear to sRGB
• $\lambda_1$, $\lambda_2$: domain loss weights
• Loss terms utilize MSE, perceptual, or VSD metrics as appropriate.

Normalization of the latent space, with importance for LoRA group consistency, is defined as:

$$\sigma^2 = \frac{1}{b c h w}\sum_{b, c, h, w}(z^{(b, c, h, w)} - \hat{\mu})$$

where

$$\hat{\mu} = \frac{1}{b c h w}\sum_{b, c, h, w}z^{(b, c, h, w)}$$

5. Quantitative Performance and Experimental Findings

RDDM models equipped with the CMB LoRA module demonstrate:

• Superior fidelity across PSNR, SSIM, LPIPS, and FID metrics, maintaining top-three rankings relative to state-of-the-art sRGB-based competitors.
• Markedly reduced artifacts and color deviation in restored outputs, attributed to direct handling of native sensor RAW data and Bayer-aware adaptation.
• High parameter efficiency: Only the LoRA layers associated with each Bayer configuration are fine-tuned, resulting in a model that is lightweight compared to re-training entire diffusion networks.
• Robust optimization in both RAW (physical) and sRGB (perceptual) domains owing to full gradient flow through the differentiable PTP module.

A plausible implication is that sensor-specific adaptation pipelines substantially improve both fidelity and generalization without incurring significant computational burden.

6. Technical Significance and Broader Impact

The CMB LoRA methodology decouples restoration learning from mosaic-specific adaptation, resolving the distribution shift and OOD limitation imposed by sensor variance. The design circumvents the need for large-scale retraining and is modularly extensible as new Bayer patterns arise in imaging hardware. The combination of joint RAW-sRGB optimization, parameter-efficient adaptation, and competitive perceptual results positions this approach as technically robust for both academic research and deployment scenarios in edge devices or real-world camera systems.

7. Modular Adaptation and Future Directions

The CMB LoRA architecture foregrounds adaptability: new or rare Bayer patterns can be accommodated simply by provisioning additional LoRA groups, without interference to existing trained parameters. Future research may explore extensions to multi-spectral mosaics or further modularization for dynamic camera sensor environments. This suggests settings where restoration models may generalize across varied imaging devices with minimal model reconfiguration.

In summary, the Configurable Multi-Bayer (CMB) LoRA module delivers sensor-aware, parameter-efficient adaptation for RAW-domain restoration models, validated by consistent improvements in fidelity and visual quality, while sustaining efficiency and extensibility within the end-to-end RDDM restoration pipeline (Chen et al., 26 Aug 2025).