Learned Nonlocal Feature Matching and Filtering for RAW Image Denoising

Abstract: Being one of the oldest and most basic problems in image processing, image denoising has seen a resurgence spurred by rapid advances in deep learning. Yet, most modern denoising architectures make limited use of the technical knowledge acquired researching the classical denoisers that came before the mainstream use of neural networks, instead relying on depth and large parameter counts. This poses a challenge not only for understanding the properties of such networks, but also for deploying them on real devices which may present resource constraints and diverse noise profiles. Tackling both issues, we propose an architecture dedicated to RAW-to-RAW denoising that incorporates the interpretable structure of classical self-similarity-based denoisers into a fully learnable neural network. Our design centers on a novel nonlocal block that parallels the established pipeline of neighbor matching, collaborative filtering and aggregation popularized by nonlocal patch-based methods, operating on learned multiscale feature representations. This built-in nonlocality efficiently expands the receptive field, sufficing a single block per scale with a moderate number of neighbors to obtain high-quality results. Training the network on a curated dataset with clean real RAW data and modeled synthetic noise while conditioning it on a noise level map yields a sensor-agnostic denoiser that generalizes effectively to unseen devices. Both quantitative and visual results on benchmarks and in-the-wild photographs position our method as a practical and interpretable solution for real-world RAW denoising, achieving results competitive with state-of-the-art convolutional and transformer-based denoisers while using significantly fewer parameters. The code is available at https://github.com/MIA-UIB/nonlocal-matchfilter .

• The paper introduces a novel RAW image denoising network that integrates classical nonlocal methods into a learnable multiscale UNet architecture.
• It employs a CNN-based Nonlocal Feature Matching and Filtering block that adaptively selects and aggregates features to improve noise removal and image quality.
• The method achieves competitive PSNR and SSIM with fewer parameters, demonstrating robust, sensor-agnostic performance on both benchmark and in-the-wild images.

Learned Nonlocal Feature Matching and Filtering for RAW Image Denoising

Introduction and Motivation

The paper "Learned Nonlocal Feature Matching and Filtering for RAW Image Denoising" (2604.17453) presents a principled reinterpretation of classical nonlocal, self-similarity-based denoising methodologies within a fully learnable neural architecture for RAW-to-RAW image denoising. The work is conceptually motivated by the disconnect between the interpretability and domain knowledge underpinning classical methods (e.g., BM3D, NL-means) and the remarkable performance but architectural opacity of deep learning-based denoisers, especially for RAW sensor data with heterogeneous noise characteristics. The authors aim to synthesize the theoretical grounding and modularity of patch-based denoising pipelines with the representational power, parameter efficiency, and adaptability of neural networks, specifically targeting sensor-agnostic RAW image denoising.

Network Architecture

The proposed network is structured as a three-scale UNet architecture, where at each scale, a single Nonlocal (NL) block is embedded. The NL block encapsulates the key operations of nonlocal methods: feature matching, collaborative filtering, and aggregation, all as learnable modules that operate in a multiscale learned feature space rather than on raw pixel patches. The NL block itself is sandwiched between two convolutional blocks (simplified ConvNeXt layers), with the overall pipeline efficiently increasing both receptive field and expressivity.

Figure 1: Architecture of the proposed denoising network, a three-scale UNet backbone with a single NL block per scale, where each NL block envelops a Nonlocal Feature Matching and Filtering block between two simplified ConvNeXt layers.

The input to the network consists of a 4-channel packed Bayer RAW image concatenated with a pixelwise noise standard deviation map (inferred from shot/readout noise models), totaling 8 channels. Outputs are denoised packed RAW images.

Nonlocal Feature Matching and Filtering Block

The central innovation is the Nonlocal Feature Matching and Filtering (NLFeMF) block, which implements a learned analog to the neighbor selection, transformation, and collaborative filtering aggregation pipeline. Feature matching is performed by a CNN-based offset prediction module, which, for each pixel, regresses $K$ 2D offsets pointing to optimal neighbor locations within a search window. Features at these positions are stacked, and nonlinear collaborative filtering is facilitated by learnable groupwise $1\times1$ convolutions (analogous to linear transformations in BM3D), followed by data-dependent, learned modulation via depthwise convolutions outputting per-coefficient attenuation maps. Aggregation to the central feature is performed by a learnable $1\times1$ convolution rather than a deterministic average, supporting subpixel alignment and nonuniform support.

Figure 2: Outline of the proposed Nonlocal Feature Matching and Filtering block, with all feature matching, transformation, modulation, and aggregation operations implemented as learned, differentiable modules.

The receptive field is effectively expanded in a nonlocal manner not only via explicit neighbor search but also by local convolutions on the stack of nonlocal features, enabling both flexible contextual integration and efficient learning.

Figure 3: Local convolutional aggregation on the stacked nonlocal neighbors effectively expands the receptive field in a nonlocal manner.

Noise Modeling and Dataset Construction

The authors address the variability and complexity inherent to sensor noise in practical RAW imaging. Rather than rely on unprocessing-based synthetic datasets with sRGB artifacts, the paper advocates curation of a high-quality dataset comprised of real clean RAW data from multiple sources, augmented with synthetic Poisson-Gaussian noise modeled after a diverse set of empirically estimated sensor noise curves (across devices/ISOs). Noise standard deviation maps for conditioning are generated per-image and per-channel to maximize sensor-agnostic generalization.

Figure 4: Estimated noise level curves (standard deviation as a function of intensity) for multiple real camera sensors and ISO values, underpinning realistic noise synthesis for training.

Analytical Experiments

An extensive ablation on the CBSD68 dataset with synthetic AWGN (additive white Gaussian noise) elucidates the impact of various design choices:

• Neighbor count: Increasing $K$ improves PSNR up to a saturation point, consistent with diminishing marginal gains in classical patch-based denoising. The multiscale variant achieves further improvements with only moderate parameter increase.

  Figure 5: Visual impact of different neighbor counts/scales, demonstrating improved texture and structure recovery with moderate $K$ and multiscale aggregation.

• Feature matching strategy: The learned CNN-based matching notably outperforms differentiable PatchMatch and fixed local windows, particularly at higher noise levels, confirming the advantage of adaptive neighbor selection in a learned feature domain.

Figure 6: Qualitative comparison of feature matching strategies, with CNN-based offsets yielding superior texture recovery and artifact suppression compared to PatchMatch or local baselines.

Comparison with State-of-the-Art Denoisers

On classical AWGN benchmarks, the proposed network achieves results comparable to state-of-the-art transformer-based models (e.g., Restormer, CTNet, DSCA-Former) and the high-parameter DRUNet, while using significantly fewer parameters (e.g., 7.5M vs. 32.6M for DRUNet). The architecture’s inductive bias, leveraging explicit self-similarity priors, accounts for its high sample efficiency and robustness.

Figure 7: Visual comparison with DRUNet, Restormer, CTNet, and the proposed method with 25/15/9 neighbors on Urban100, highlighting improved structure reconstruction and artifact suppression at reduced complexity.

Experiments on RAW Datasets and In-The-Wild Images

DND RAW Benchmark

On the DND RAW denoising benchmark, the method achieves PSNR and SSIM on par with or superior to all prior art except the dual-domain RAW-RGB DualDn (which uses a substantially more complex backbone). Notably, the proposed architecture avoids the residual structured or low-frequency noise sometimes visible in the output of competing methods, especially in dark or flat regions.

Figure 8: DND visual crops showing superior noise removal and artifact reduction by the proposed method compared to DualDn, CycleISP, and UPI.

In-The-Wild Generalization

On uncurated, high-ISO RAW captures from commercial phones across various brands and conditions (including extreme low-light), the network successfully removes both grainy and structured noise with minimal texture suppression and no color bias, outperforming transformer-based pipelines (e.g., DualDn/Restormer) in regions with low SNR and non-stationary noise, without suffering from color bias or noise underfitting.

Figure 9: Qualitative comparison on in-the-wild images—proposed method shows robust denoising, even in extreme SNR conditions and with diverse noise statistics, compared to DualDn.

Figure 10: Detailed crops on low-light outdoor images highlighting effective removal of residual grain, low-frequency noise, and color biases in the proposed method.

Implications, Limitations, and Future Directions

The integration of an interpretable, model-inspired nonlocal processing module into a modern neural pipeline positions this architecture as a highly efficient, robust, and sensor-agnostic alternative to deep denoisers relying on opaque, high-capacity transformer mechanisms. The demonstrated parameter efficiency, adaptability to noise statistics via explicit noise map conditioning, and competitive performance on challenging real-world data make the approach compelling, especially for resource-constrained devices or transparent deployment contexts.

The method's modularity suggests it may be extensible to other restoration tasks (e.g., joint denoising/demosaicking, burst or video denoising), potentially with adjustments for different CFA patterns or incorporation into vision transformer backbones. The explicit separation and interpretability of the stages could facilitate hybrid model-based–data-driven approaches and could support plug-and-play optimization schemes.

The primary limitation is the fixed neighbor count per block, which may be suboptimal for highly nonstationary scene content. The approach also assumes sufficiently accurate noise maps for conditioning; failure modes are possible if sensor noise statistics deviate significantly from those encountered during training or are estimated erroneously.

Conclusion

The paper offers a rigorous and technically substantiated bridge between classical and modern denoising paradigms via learnable nonlocal feature matching and filtering in a multiscale architecture. The resulting network achieves strong empirical performance across both standard and challenging in-the-wild conditions, with a clear reduction in model complexity, improved interpretability, and high adaptability to real sensor characteristics. The methodology opens the way for principled neural modeling in other image restoration applications where explicit modularity, data efficiency, and theoretical tractability are valued.