DRIFT: Deep Restoration, ISP Fusion, and Tone-mapping

Abstract: Smartphone cameras have gained immense popularity with the adoption of high-resolution and high-dynamic range imaging. As a result, high-performance camera Image Signal Processors (ISPs) are crucial in generating high-quality images for the end user while keeping computational costs low. In this paper, we propose DRIFT (Deep Restoration, ISP Fusion, and Tone-mapping): an efficient AI mobile camera pipeline that generates high quality RGB images from hand-held raw captures. The first stage of DRIFT is a Multi-Frame Processing (MFP) network that is trained using a adversarial perceptual loss to perform multi-frame alignment, denoising, demosaicing, and super-resolution. Then, the output of DRIFT-MFP is processed by a novel deep-learning based tone-mapping (DRIFT-TM) solution that allows for tone tunability, ensures tone-consistency with a reference pipeline, and can be run efficiently for high-resolution images on a mobile device. We show qualitative and quantitative comparisons against state-of-the-art MFP and tone-mapping methods to demonstrate the effectiveness of our approach.

• The paper presents a unified pipeline that leverages DRIFT-MFP for multi-frame restoration and DRIFT-TM for adaptive tone-mapping, achieving state-of-the-art metrics (LPIPS 0.05, FID 10.73) and high perceptual quality.
• It employs a tailored NAFNet architecture with a novel Adversarial Perceptual Loss to efficiently process 11-frame 12MP bursts in just 3.2 seconds on Snapdragon hardware.
• The integrated design enables precise HDR and contrast modulation with inference-time tunability, reducing artifacts and supporting real-time mobile computational photography.

Authoritative Summary of "DRIFT: Deep Restoration, ISP Fusion, and Tone-mapping" (2604.03402)

Introduction and Motivation

DRIFT introduces a unified Deep Learning-based Image Signal Processing (ISP) pipeline optimized for mobile camera applications. It addresses fundamental challenges in handheld multi-frame image restoration, exposure fusion for HDR, and high-resolution tone-mapping—all within constraints of computational efficiency suitable for modern NPUs in smartphones. The pipeline is logically and algorithmically organized into two principal neural modules: DRIFT-MFP for restoration and super-resolution, and DRIFT-TM for tone-mapping. Restoration artifacts often propagate and amplify through tone-mapping, motivating the pipeline’s integrated design.

Figure 1: A schematic detailing the sequential coupling of multi-frame deep restoration (DRIFT-MFP), fusion ISP, and deep tone-mapping (DRIFT-TM) with provision for inference-time output tuning.

DRIFT-MFP: Multi-Frame Restoration and Super-Resolution

DRIFT-MFP leverages a tailored NAFNet architecture, accepting 11 noisy RGB frames as input (covering typical handheld burst acquisitions) and outputting a single restored RGB frame. The restoration subsystem is designed for maximal compatibility with Qualcomm Snapdragon NPUs, omitting computational bottlenecks such as deformable convolutions and transformers. Training data is synthesized from tripod and handheld captures, with realistic human handshake modeled by temporally correlated homographies applied to clean tripod data, exposing the model to authentic artifacts seen in practice.

Figure 2: Visual denoising comparisons across challenging scenes, DRIFT-MFP outperforms prior art in artifact suppression and fidelity to ground truth.

A novel Adversarial Perceptual Loss (APL) stabilizes GAN training and avoids artifact generation characteristic of VGG-based perceptual losses and LPIPS. Discriminator features are matched pre-activation at multiple layers, reflecting domain-adaptivity through continual updates—a marked deviation from static classification losses.

Strong numerical results are demonstrated:

• For denoising, DRIFT-MFP achieves LPIPS 0.05, FID 10.73, PSNR 37.49, SSIM 0.97, outperforming Restormer, Burstormer, and NAFNet variants on almost all metrics.
• A 4x SR evaluation confirms robustness: FID 20.84 and LPIPS 0.10 (see supplementary Table).

A user study with 11 image quality experts confirms perceptual superiority (DRIFT-MFP was preferred in 63% of cases), and the LPIPS-trained NAFNet produced visually distracting grid artifacts despite “strong” FID scores.

Figure 3: Comparative 4x super-resolution; DRIFT-MFP maintains GT fidelity without grid-like artifacts seen in LPIPS-optimized competitors.

DRIFT-TM: Adaptive Deep Tone-Mapping Network

DRIFT-TM innovates in high-resolution tone-mapping by decomposing the operation into residual learning over a lightweight “Tone-map Lite” baseline rather than direct regression to RGB. The network outputs fusion weights and point-wise gain maps, enabling precise modulation of local contrast and HDR effect at inference, without retraining.

Figure 4: Overview of DRIFT-TM training; the network learns enhancements as residuals to Tone-map Lite, facilitating robust learning and inference-time tunability.

Full-resolution image processing and global encoding prevent tile-to-tile inconsistencies that degrade prior deep tone-mappers—this is critical for high-res (12MP+) mobile images. Metadata encoding (ISO, exposure time, sensor/pipeline) further tailors output for diverse capture conditions.

Figure 5: DRIFT-TM architecture: integration of local, global, and metadata encoders enables scalable and consistent tone-mapping for arbitrary input sizes and conditions.

Reference tone-map outputs from a complex, non-DL pipeline serve as GT for supervised learning. DRIFT-TM is trained to match two GT targets (with and without contrast blocks), permitting separate adjustment of HDR and contrast during inference.

Figure 6: Qualitative comparison of non-reference tone-mapping algorithms with DRIFT-TM; tiled methods (Self-TMO) produce visible artifacts absent in DRIFT-TM.

DRIFT-TM delivers PSNR 40.59 and SSIM 0.99 relative to GT tone-mapping, outperforming LLF-LUT and TMO-GAN. Structural metrics (TMQI-Q 0.845) closely approach ground-truth.

Figure 7: Tone-mapping comparison and difference heatmaps; DRIFT-TM yields closer matches to GT in both brightness and texture than LLF-LUT and TMO-GAN.

Ablation studies confirm the necessity of global encoding, metadata input, and residual learning for optimal quality; removal of these components introduces color deviations and tiling artifacts.

Figure 8: Ablation visualizations; missing global/meta encodings and maps degrade fidelity and consistency.

Inference-time tunability is demonstrated by modulating LUTs for local contrast and HDR strength, enabling customizable outputs from a single trained model.

Figure 9: Output variations illustrating tunability; local contrast and HDR strength can be modulated on demand.

Practical Implications

DRIFT computes full DRIFT-MFP restoration for 12MP/11-frame bursts in 3.2 seconds on Snapdragon 8 Elite; tone-mapping adds 0.5 seconds. Tone-map Lite and tuning operations are CPU-parallelizable. The pipeline’s efficiency is highly relevant for on-device mobile implementations, supporting real-time or near-real-time workflows for consumer devices.

The architectural choices favor scalability and hardware compatibility for current and next-generation NPUs, reducing barriers for integration into commercial smartphone ISP software stacks.

Theoretical Implications and Future Directions

DRIFT demonstrates that integrated, task-adaptive restoration and tone-mapping, with explicit domain-specific loss functions and modular residual learning, are superior to disjoint, direct regression approaches. The use of APL and residual enhancement learning opens avenues for further refinement in GAN-optimized multi-frame processing.

Explicit encoding of global features and metadata points toward future architectures that are context-aware, predicting per-scene processing requirements and allowing deployment across heterogeneous sensor and capture environments.

Inference-time tuning, achieved without retraining, is a step toward user- and device-adaptive ISP pipelines. Extending such control to semantic content or user preference, coupled with robust artifact avoidance, suggests broad potential for interactive computational photography.

Conclusion

DRIFT offers a unified pipeline for deep restoration, super-resolution, fusion, and tone-mapping, optimized for mobile platforms. It achieves state-of-the-art performance in quantitative and human-centric metrics, provides flexible user control, and deploys efficiently on commodity hardware. The integration of adversarial perceptual loss, global encoding, and residual tone enhancement sets a new standard for AI mobile ISP, with pragmatic advances for real-world handheld imaging and theoretical implications for robust, controllable deep pipelines in computational photography.