FastSHADE: Fast Self-augmented Hierarchical Asymmetric Denoising for Efficient inference on mobile devices

Abstract: Real-time image denoising is essential for modern mobile photography but remains challenging due to the strict latency and power constraints of edge devices. This paper presents FastSHADE (Fast Self-augmented Hierarchical Asymmetric Denoising), a lightweight U-Net-style network tailored for real-time, high-fidelity restoration on mobile GPUs. Our method features a multi-stage architecture incorporating a novel Asymmetric Frequency Denoising Block (AFDB) that decouples spatial structure extraction from high-frequency noise suppression to maximize efficiency, and a Spatially Gated Upsampler (SGU) that optimizes high-resolution skip connection fusion. To address generalization, we introduce an efficient Noise Shifting Self-Augmentation strategy that enhances data diversity without inducing domain shifts. Evaluations on the MAI2021 benchmark demonstrate that our scalable model family establishes a highly efficient speed-fidelity trade-off. Our base FastSHADE-M variant maintains real-time latency (<50 ms on a modern mobile GPU) while preserving structural integrity, and our scaled-up FastSHADE-XL establishes a new state-of-the-art for overall image quality. Ultimately, FastSHADE successfully bridges the gap between theoretical network efficiency and practical deployment for real-world mobile ISP pipelines.

• The paper introduces FastSHADE, which leverages asymmetric frequency denoising and spatially gated upsampling to achieve high PSNR with low latency on mobile devices.
• It employs a modified U-Net with specialized AFDB and SGU modules to efficiently separate structural details from noise, optimizing both performance and power usage.
• Ablation studies and frequency domain analysis validate its design choices, establishing a new state-of-the-art speed-fidelity trade-off on benchmarks like MAI2021.

FastSHADE: An Efficient Hierarchical Asymmetric Denoising Architecture for Real-Time Mobile Inference

Introduction

Efficient and high-fidelity image denoising on mobile devices is constrained by real-time inference requirements and strict power budgets, particularly given the small sensor sizes and high-resolution demands of contemporary mobile photography. Current solutions largely rely on either heavy-weight models that compromise latency or lightweight approaches with inadequate perceptual quality. The paper "FastSHADE: Fast Self-augmented Hierarchical Asymmetric Denoising for Efficient inference on mobile devices" (2604.10275) proposes a systematized approach to resolve this trade-off by combining architectural innovations, frequency-aware processing, and deployment-level optimizations.

Figure 1: The FastSHADE model family maintains a superior latency-vs-quality Pareto frontier as measured by PSNR and on-device inference time on a Qualcomm Adreno 840 GPU, outclassing prior state-of-the-art approaches.

Architecture

The foundation of FastSHADE is a U-Net-style topology subjected to principled modifications at both macro and micro levels to maximize both deployment efficiency and in-distribution denoising capacity. Two key architectural elements define the approach:

1. Asymmetric Frequency Denoising Block (AFDB): Unlike channel bifurcation or naive convolutional stacks, AFDB separates the processing of low-frequency (LF) and high-frequency (HF) components by allocating only 25% of channels for deep spatial context extraction (using $5 \times 5$ convolutions) and the remaining 75% for localized noise suppression (using $3 \times 3$ convolutions). This division arises from the hypothesis, confirmed by spectral analysis, that broader context and structural details are less parameter-intensive compared to the diverse, fine-detailed statistics of noise.
2. Spatially Gated Upsampler (SGU): Standard skip connections produce significant overhead on resource-limited devices. SGU improves upon this by employing spatial attention-like gating of upsampled features, modulating high-resolution bypasses at a per-pixel level, preserving fidelity in upsampling without unnecessary channel inflation.

   Figure 2: Overview of the FastSHADE model architecture, highlighting the hierarchical processing and integration points for the AFDB and SGU modules.

These innovations are enabled by a design that uses learnable $2 \times 2$ strided convolutions for efficient, task-adaptive downsampling, discarding fixed transforms like Haar wavelets in favor of robust end-to-end learned representations.

Frequency-Aware Feature Decoupling

To empirically validate the frequency-routing efficacy of AFDB, the internal representations were subjected to frequency domain analysis. The broader energy spectrum in the $5 \times 5$ path evidences that it preserves structural detail, while the $3 \times 3$ path assumes a flatter spectrum—indicating focus on unstructured, isotropic noise.

Figure 3: Frequency domain analysis within AFDB; the $5\times5$ branch maintains energy in low-to-mid spatial frequencies, while the $3\times3$ branch focuses on high-fidelity noise extraction as characterized by its flat spectral response.

This confirms the architectural inductive bias toward separating structure and noise processing, validated without explicit frequency regularization in the loss, and is directly attributable to the efficiency and quality gains observed.

Training, Self-Augmentation, and Deployment

FastSHADE incorporates a self-augmentation scheme named Noise Shifting Self-Augmentation. By extracting and spatially shifting the true noise residual from dataset pairs, the method enriches the effective in-distribution diversity of training data while strictly avoiding domain-shift errors associated with synthetic or mismatched noise injection. This augmentation, combined with standard geometric transformations and the occasional injection of Gaussian noise, demonstrably improves PSNR during ablation.

On the deployment side, FastSHADE leverages multi-branch re-parameterization (MBRConv), fusing complex training-time topologies into single convolutions for inference. Quantization is handled via an STE approach to achieve direct integer-valued outputs suitable for 8-bit image processing pipelines without runoff artifacts. All deployment optimizations are carefully constructed to be compatible with FP16 execution on mobile GPUs, with further potential for INT8-optimized inference using Quantization-Aware Training (QAT) in future extensions.

Experimental Results and Ablation

FastSHADE establishes a new state-of-the-art in speed-fidelity trade-off on the MAI2021 benchmark and in competition against baselines, notably SplitterNet, TeamPAK, and TLG. The base FastSHADE-M model achieves a PSNR of 37.52 dB at only 48.1 ms per image on Adreno 840, outpacing SplitterNet both in speed and in practical denoising fidelity.

Ablations confirm the following:

• The asymmetric channel split of AFDB (25% LF, 75% HF) achieves the same fidelity as a symmetric split but with lower latency due to reduced heavy computation on broad contexts.
• The SGU upsampling block outperforms both vanilla concatenation and conventional attention blocks (Squeeze-and-Excitation, CBAM) in both runtime and PSNR, with its depthwise refinement variant yielding the highest score.
• Inclusion of the self-augmentation pipeline outperforms both random noise profile swapping and vanilla augmentation, further increasing PSNR robustness.

Qualitative results indicate the preservation of structural integrity and superior perceptual denoising across all tested benchmarks, as shown by the outputs below.

Figure 4: Representative qualitative denoising results from the FastSHADE family, demonstrating structural preservation and effective noise suppression.

Implications, Limitations, and Future Directions

FastSHADE’s modular approach, particularly the AFDB and SGU components, are readily transferable to other vision tasks requiring a balance of fine spatial recovery and large-scale efficiency—such as low-light enhancement, super-resolution, or even real-time video enhancement on resource-constrained hardware.

The work exposes several promising directions:

• Fine-grained quantization using QAT and hardware-aligned numerical schemes (INT8, custom NPU kernels), which could push inference latency even lower.
• Hardware-stream optimization for advanced mobile NPUs and maximizing on-chip SRAM utilization can further narrow runtime and energy gaps between theoretical and operational performance.

Despite its strengths, the present architecture is specifically tuned for FP16/FP32; quantitative evaluation of INT8 deployment and adaptation to hardware-specific graph optimizations remain open.

Conclusion

FastSHADE introduces a series of algorithmic and systems-level innovations that establish a new Pareto frontier for real-time image denoising on mobile devices. By systematically addressing the frequency-dependence of image noise and optimizing for edge-aware structural and noise feature decoupling, FastSHADE achieves a high-fidelity, low-latency solution suitable for integration into mobile ISP pipelines. Its comprehensive evaluation and ablation studies rigorously justify architectural decisions and provide high-confidence guidelines for system designers seeking fast, robust, and accurate image denoising in real-world deployment environments.