TM-BSN: Triangular-Masked Blind-Spot Network for Real-World Self-Supervised Image Denoising

Abstract: Blind-spot networks (BSNs) enable self-supervised image denoising by preventing access to the target pixel, allowing clean signal estimation without ground-truth supervision. However, this approach assumes pixel-wise noise independence, which is violated in real-world sRGB images due to spatially correlated noise from the camera's image signal processing (ISP) pipeline. While several methods employ downsampling to decorrelate noise, they alter noise statistics and limit the network's ability to utilize full contextual information. In this paper, we propose the Triangular-Masked Blind-Spot Network (TM-BSN), a novel blind-spot architecture that accurately models the spatial correlation of real sRGB noise. This correlation originates from demosaicing, where each pixel is reconstructed from neighboring samples with spatially decaying weights, resulting in a diamond-shaped pattern. To align the receptive field with this geometry, we introduce a triangular-masked convolution that restricts the kernel to its upper-triangular region, creating a diamond-shaped blind spot at the original resolution. This design excludes correlated pixels while fully leveraging uncorrelated context, eliminating the need for downsampling or post-processing. Furthermore, we use knowledge distillation to transfer complementary knowledge from multiple blind-spot predictions into a lightweight U-Net, improving both accuracy and efficiency. Extensive experiments on real-world benchmarks demonstrate that our method achieves state-of-the-art performance, significantly outperforming existing self-supervised approaches. Our code is available at https://github.com/parkjun210/TM-BSN.

• The paper proposes a blind-spot network that uses triangular-masked convolutions to precisely align with the diamond-shaped correlation of real sRGB noise.
• It avoids downsampling by leveraging input rotations and feature-map shifts to retain rich contextual information and fine texture details.
• Experimental results on SIDD and DND benchmarks demonstrate improved PSNR/SSIM metrics with lower computational cost and effective knowledge distillation.

TM-BSN: Triangular-Masked Blind-Spot Network for Real-World Self-Supervised Image Denoising

Problem Setting and Limitations of Prior Approaches

Self-supervised image denoising has become a major research direction, primarily because acquiring paired clean-noisy real-world datasets is fundamentally expensive or infeasible in many scenarios. While blind-spot networks (BSNs) avoid reliance on paired data by excluding the center pixel from the receptive field, their conventional assumption of pixel-wise noise independence is systematically violated in realistic camera pipelines. In sRGB images, spatial correlation arises from the camera's demosaicing process, introducing a diamond-shaped noise correlation pattern that invalidates the naive blind-spot principle.

Downsampling strategies (e.g., pixel-shuffle) attempt to decorrelate the noise but at the cost of altered noise statistics, reduced contextual information, and the introduction of inference-train domain gaps, often necessitating further post-processing. Recent asymmetric architectures (e.g., AT-BSN) and patch-masked designs (e.g., LG-BPN) have attempted to sidestep downsampling, but their blind-spot geometry misaligns with the actual correlation pattern, leading to suboptimal utilization of the signal context and loss of fine texture fidelity.

Figure 1: Effective receptive field (ERF) visualization for several BSNs: (a) AP-BSN, (b) LG-BPN, (c) AT-BSN, (d) TM-BSN. The proposed TM-BSN aligns its blind spot with the diamond-shaped spatial correlation of real sRGB noise, unlike competing BSN designs.

Triangular-Masked Convolution and Network Architecture

Correlation Geometry of Real sRGB Noise

A key insight motivating TM-BSN is that demosaicing interpolates missing RGB values by assigning spatially decaying weights to neighboring pixels. This process results in highly structured, diamond-shaped local spatial correlations, as directly visualized via empirical correlation maps from SIDD image data.

Figure 2: (a) Demosaicing filter coefficients emphasize nearby pixels; (b) the empirical correlation map reveals the characteristic diamond-shaped dependency inherent to sRGB noise after ISP processing.

Conventional architectures with square or rectangular blind spots fail to match this shape, either including correlated pixels (inducing identity mapping/trivial solutions) or needlessly discarding independent signal.

Triangular-Masked Convolutional Layer

TM-BSN instantiates a blind-spot architecture that precisely excludes the correlated pixels identified by demosaicing geometry. This is achieved by enforcing an upper-triangular (rather than full or square) mask on all $3 \times 3$ convolution kernels. Only elements where the row index is less than or equal to the column index are retained. By stacking these layers, the receptive field grows in a triangular manner; with suitable input rotations and feature-map shifts, the final blind spot aligns into a central diamond at every spatial location.

Figure 3: Stacked triangular-masked convolutions progressively expand the receptive field in a triangular lattice; the subsequent feature-map shift produces a central blind spot conforming to the spatial correlation pattern.

The filter stack is applied to four rotated versions (0°, 90°, 180°, 270°) of the input, feature shifts are used to precisely tune the excluded context (controllable by the shift offset $s$), and the rotated outputs are merged to finalize the denoised prediction.

Figure 4: TM-BSN’s architecture leverages four rotations, triangular-masked convolutions, and feature-map shifts. The ensemble reconstructs the full-resolution, diamond-shaped blind spot to robustly exclude correlated noise contributions.

By avoiding downsampling, TM-BSN processes images at native resolution and does not require post-processing, and its masked convolutional operations avoid the computational overhead of methods such as DSPMC.

Knowledge Distillation for Performance-Efficiency Trade-Off

TM-BSN’s feature extraction is shared across multiple shift offsets; thus, it can efficiently generate predictions for different blind-spot sizes in a single pass (about 15% extra FLOPs relative to one offset). By collecting complementary outputs across offsets, TM-BSN serves as a multi-teacher ensemble whose knowledge is distilled into a lightweight U-Net student using the recharged distillation (RD) scheme. The RD framework randomly reinjects noisy pixels to produce a stable, diverse supervision signal while preventing the student from collapsing to trivial identity mappings.

Experimental Analysis

Quantitative and Qualitative Evaluation

TM-BSN was extensively benchmarked on SIDD and DND real-world denoising datasets. The distilled TM-BSN (D) achieves highest PSNR/SSIM across all self-supervised methods, e.g., on SIDD validation, TM-BSN (D) attains 38.08 dB and 0.952 SSIM, outperforming the second-best APR(RD) (38.00 dB, 0.947 SSIM) and all recent BSN/PD-based competitors. On the DND benchmark, TM-BSN (D) surpasses TBSN by +0.33 dB in the self-supervised setting, with no compromise in visual quality or runtime efficiency.

Figure 5: Qualitative results on SIDD. TM-BSN (D) preserves wood grain and subtle image textures while effectively removing sensor noise, outperforming alternative self-supervised methods both visually and numerically.

Figure 6: Qualitative comparison on DND. TM-BSN (D) robustly suppresses non-Gaussian noise while retaining fabric patterns—most other methods oversmooth fine structure.

Complexity Comparison

TM-BSN (D) achieves top-tier accuracy with significantly lower parameter count and FLOPs (1.02M parameters, 26.74 GFLOPs, 3.21 ms per $256\times256$ patch) relative to recent patch-masked or transformer BSNs. The design precision and effective use of all uncorrelated pixels avoid the exponential cost of large-kernel or downsampled methods.

Ablation: Shift Offset and Blind-Spot Size Sensitivity

Extensive ablation studies show:

• Too small a shift offset ($s=4$ or less) leads to insufficient exclusion of correlated pixels and degenerate identity mapping.
• Excessively large offsets ($s\geq 6$) prune useful context, impeding texture recovery.
• The optimal training offset is $s=5$; the knowledge distillation set $S=\{2,3,4,5,6\}$ delivers the strongest validation performance, balancing context diversity and stability.

  Figure 7: Ablation curves for training and inference shift offsets. Models trained with $s=5$ optimally block correlated regions while leveraging uncorrelated adjacent pixels.

  

Figure 8: Output images with different shift offsets demonstrate visual degradation outside the optimal shift regime; TM-BSN (D) exhibits the most faithful denoising with preserved detail.

Implications and Future Directions

TM-BSN establishes a rigorous principle for blind-spot construction in self-supervised denoising: the blind-spot must geometrically align with the true signal dependency structure. This finding affects future architecture designs for both imaging and potentially other signal processing domains where artifacts are introduced by domain-specific sensor pipelines.

Practically, TM-BSN demonstrates that high-fidelity, real-world denoising can be achieved without downsampling, complex post-processing, or clean data—substantially improving the accessibility and deployability of denoising models in uncontrolled, cost-sensitive environments (e.g., mobile photography, medical imaging).

Theoretically, TM-BSN’s design suggests that further exploiting the precise geometry of noise and signal correlation, possibly tailored to additional ISP stages or more complex, non-local processes, may yield even greater gains. The architectural integration of advanced context masking, adaptive shift policies, or more sophisticated ensemble distillation schemes (beyond U-Net backbones) are relevant extensions.

Conclusion

TM-BSN introduces a triangular-masked convolutional architecture that constructs a diamond-shaped, data-driven blind spot, precisely reflecting the noise correlation pattern found in real sRGB images produced by demosaicing. This approach overcomes the limitations of previous self-supervised denoising paradigms by optimally filtering correlated context while exploiting the maximum available uncorrelated signal. By further integrating an efficient multi-offset knowledge distillation regime, TM-BSN delivers state-of-the-art performance on real-world denoising benchmarks while remaining computationally efficient. The demonstrated design principles have direct implications for future advancements in both generic and domain-adapted self-supervised restoration architectures.

---

Reference: "TM-BSN: Triangular-Masked Blind-Spot Network for Real-World Self-Supervised Image Denoising" (2604.04484)