RawGen: Learning Camera Raw Image Generation

Abstract: Cameras capture scene-referred linear raw images, which are processed by onboard image signal processors (ISPs) into display-referred 8-bit sRGB outputs. Although raw data is more faithful for low-level vision tasks, collecting large-scale raw datasets remains a major bottleneck, as existing datasets are limited and tied to specific camera hardware. Generative models offer a promising way to address this scarcity -- however, existing diffusion frameworks are designed to synthesize photo-finished sRGB images rather than physically consistent linear representations. This paper presents RawGen, to our knowledge the first diffusion-based framework enabling text-to-raw generation for arbitrary target cameras, alongside sRGB-to-raw inversion. RawGen leverages the generative priors of large-scale sRGB diffusion models to synthesize physically meaningful linear outputs, such as CIE XYZ or camera-specific raw representations, via specialized processing in latent and pixel spaces. To handle unknown and diverse ISP pipelines and photo-finishing effects in diffusion-model training data, we build a many-to-one inverse-ISP dataset where multiple sRGB renditions of the same scene generated using diverse ISP parameters are anchored to a common scene-referred target. Fine-tuning a conditional denoiser and specialized decoder on this dataset allows RawGen to obtain camera-centric linear reconstructions that effectively invert the rendering pipeline. We demonstrate RawGen's superior performance over traditional inverse-ISP methods that assume a fixed ISP. Furthermore, we show that augmenting training pipelines with RawGen's scalable, text-driven synthetic data can benefit downstream low-level vision tasks.

• The paper introduces a many-to-one diffusion-based pipeline that maps diverse sRGB outputs to a canonical CIE XYZ representation, overcoming hardware constraints.
• It leverages a pretrained DiT backbone with conditional denoising and VAE fine-tuning to ensure high-fidelity, device-agnostic raw image synthesis.
• Empirical evaluations show significant improvements in ISP optimization, illuminant estimation, and denoising tasks, confirming statistical alignment with real sensor data.

RawGen: Diffusion-Based Camera Raw Image Generation Across Arbitrary Devices

Introduction and Motivation

Conventional generative models in computer vision and computational photography predominantly target synthesis in the 8-bit sRGB color space. Despite the prevalence of sRGB for display and storage, raw sensor measurements in linear space (i.e., camera raw or CIE XYZ) are essential for low-level vision tasks, including denoising, color constancy, and ISP optimization. However, the acquisition of large-scale, diverse raw data is hampered by data collection costs and hardware lock-in, restricting generalizability and dataset scale.

"RawGen: Learning Camera Raw Image Generation" (2604.00093) presents a unified framework that overcomes data scarcity and hardware constraints by enabling both text-to-raw and image-to-raw generation pathways. The key technical advance is a many-to-one diffusion-based pipeline that produces device-agnostic scene-referred linear representations and supports mapping to arbitrary camera raw domains, thereby decoupling content generation from device specificity.

Methodology and Technical Contributions

RawGen leverages pretrained large-scale sRGB diffusion models and introduces a robust pipeline for physically meaningful linear data generation. The approach comprises a series of architectural and training innovations that address prior methods’ deficiencies in handling diverse, unknown ISP-induced photo-finishing effects:

• Many-to-One Inverse-ISP Objective: Instead of learning a one-to-one mapping from sRGB to linear (as in conventional inverse ISP frameworks), RawGen anchors multiple sRGB renditions—produced through wide-ranging, randomized software ISP photo-finishing—to a single canonical CIE XYZ target. This formulation endows the model with strong invariance to heterogeneous and unknown photo-finishing transformations.
• Backbone and Conditional Denoising: RawGen builds on a pretrained rectified-flow DiT backbone, exploiting its image conditioning pathway. The framework fine-tunes the conditional denoiser via LoRA adapters (applied solely to attention projections) to denoise CIE XYZ VAE latents conditioned on sRGB input latents, ensuring invariance to sRGB domain variability.
• VAE Decoder Fine-Tuning: The decoder is further fine-tuned in the CIE XYZ domain to guarantee high-fidelity reconstruction from the denoised latent, alleviating degradation typically observed when directly transferring pretrained sRGB decoders to the XYZ space.
• Unified Inference for I2R and T2R: RawGen supports conditioning either on VAE-encoded sRGB images (I2R) or text prompts via the standard text-to-latent path (T2R), thus enabling both image-driven and prompt-driven raw data synthesis.
• Device-Specific Mapping and Noise Modeling: Mapping from CIE XYZ to target camera color spaces utilizes DNG calibration metadata with interpolated color matrices (based on selected CCTs). Optional heteroscedastic noise simulation approximates sensor-specific acquisition noise, boosting realism for downstream learning tasks.

  Figure 1: Overview of the RawGen framework, including many-to-one training strategy, fine-tuning of denoiser and decoder, and device-specific inference.

Evaluation: Many-to-One sRGB-to-XYZ Consistency

To quantify RawGen's robustness in projecting sRGB images with arbitrary photo-finishing to a device-independent canonical representation, two test domains are evaluated:

• Expert-Retouched MIT-Adobe FiveK Variations: Using sRGB images retouched by five distinct experts, RawGen consistently outperforms CIE XYZ Net, InvISP, and Raw-Diffusion across all splits. Peak PSNR/SSIM improvement over baselines for unseen retouch styles affirms effective domain-gap suppression and domain-level generalization. The ablation with one-to-one supervision (RawGen_{\text{one-to-one}}) confirms the necessity of many-to-one training for robust inversion.

  Figure 2: sRGB input images (with different expert edits) and corresponding CIE XYZ reconstructions by competing methods.

• Text-Augmented Synthetic Variants: Conditioning on text-generated sRGBs with systematically varied color grading, RawGen achieves maximal latent space compactness (i.e., lowest L2 distances in PCA/t-SNE/UMAP projections), indicating superior convergence to a unique canonical representation and effective suppression of entangled photo-finishing styles.

  Figure 3: Left: Comparative latent space compactness across methods; Right: t-SNE visualization demonstrating RawGen's clustering of sRGB-to-XYZ projections.

Device-Specific Raw Synthesis and Distribution Alignment

RawGen’s canonical CIE XYZ output is mapped deterministically to any target camera’s raw space using device calibration data and customizable illuminant parameters. Synthesis supports optional sensor noise simulation for realistic domain alignment.

Empirical validation is provided by feeding RawGen-generated raw data into pretrained neural ISP models (e.g., PyNet, Modular Neural ISP) trained exclusively on real device data. Models render plausible, noise-consistent sRGB outputs from RawGen-synthesized raw, signifying strong statistical alignment with sensor-specific distributions.

Figure 4: Input sRGB images (A), ground-truth raw (B), RawGen reconstructions (C), and RawGen-generated raw images driven by semantic prompts (D).

Impact on Downstream Low-Level Vision Tasks

RawGen-generated data significantly improves learning-based performance across several downstream raw-based tasks compared to prior synthetic data pipelines (e.g., EnlightenGAN, UPI, Graphics2RAW):

• Illuminant Estimation: On the challenging NUS-8 dataset, RawGen-augmented training achieves mean/median angular error reductions close to those obtained from real raw data—critical for color constancy under device diversity and limited real data per camera.
• Neural ISP Learning: Networks trained with RawGen data reach PSNR/SSIM parity with real-data-trained models.
• Raw-Domain Denoising: On real nighttime sensor data at multiple ISO levels, denoisers trained on RawGen data achieve SOTA PSNR/SSIM, exhibiting robust noise generalization.

These empirical findings demonstrate RawGen's synthetic data utility, alleviating the hardware and labor constraints hindering data scalability for low-level tasks.

Generative Editing in Linear Space

Conventional generative pipelines support either limited (and implicitly learned) camera parameter control, or require iterative regeneration for each edit. RawGen decouples content generation from rendering: After generating a single scene-referred XYZ (or raw)—either from text or image—downstream parametric edits (e.g., exposure, white balance, tone mapping) are efficiently realized using conventional ISP logic without extra inference cost or retraining.

Figure 5: Device- and illumination-specific renderings of a scene from the same canonical CIE XYZ image, showcasing color temperature and noise parameter variation.

Implications and Future Directions

RawGen introduces a scalable, unified pipeline for semantically controlled, camera-agnostic generation of sensor raw data, shifting generative modeling for low-level vision from display-referred imagery to physically consistent, device-resolved outputs.

• Practical Impact: Immediate applications include data augmentation for training camera-agnostic ISPs, illumination-invariant denoising, and general-purpose vision systems requiring robust raw image synthesis.
• Theoretical Implications: The many-to-one training paradigm establishes a new operational standard for learning invariance to uncontrolled ISP effects, crucial for generative tasks targeting upstream domains like sensor space, in contrast to conventional one-to-one mappings.
• Future Work: Incorporating richer, device-specific priors (e.g., PSF, spatially-varying noise, lens shading) and advancing text-driven control over acquisition conditions (e.g., dynamic range, exposure) will further bridge the gap toward comprehensive physical camera simulation.

Conclusion

RawGen provides a robust, diffusion-based framework for camera-agnostic raw image generation, validated via both semantic and quantitative measures. It enables flexible, physically grounded data generation and editing, supporting generalizable low-level vision research and downstream photonics applications. By separating content synthesis from device rendering, RawGen establishes a foundation for scalable, cross-device raw image modeling unconstrained by hardware or manual data collection.

---

Reference:

"RawGen: Learning Camera Raw Image Generation" (2604.00093)