CGFformer: Cluster-Guidance Frequency Transformer for Pansharpening

Abstract: Pansharpening aims to generate high-resolution multispectral (HRMS) images by fusing low-resolution multispectral (LRMS) images with high-resolution panchromatic (PAN) images. However, the current mainstream frequency-based pansharpening methods employ fixed frequency filters, which cannot precisely adapt to complex and spatially diversified frequency distributions in PAN and MS images. Furthermore, existing denoising strategies insufficiently exploit frequency components for denoising and struggle to suppress various noise types accurately. To address these challenges, we propose CGFformer, a cluster-guidance frequency Transformer that focuses on varying frequency distribution and interactions between frequency and spatial components. Specifically, we design an adaptive separation module that integrates local features and non-local information through K-means clustering, enabling more precise separation of high- and low-frequency components. Subsequently, we introduce a dual-stream refinement module combined with Transformer-based cross-attention to remove various noise, allowing the network to jointly suppress frequency-relevant and irrelevant disturbances. In addition, we develop a frequency-spatial fusion module designed to enhance detail and facilitate spatial-frequency interaction, ensuring more effective reconstruction of spatial structures in the fused results. Extensive experiments on multiple benchmark datasets demonstrate that the proposed CGFformer achieves notable improvements over existing pansharpening approaches.

• The paper introduces a novel cluster-guided adaptive frequency separation method that outperforms conventional filters in differentiating high- and low-frequency components.
• It details a dual-stream refinement module that employs mutual cross-attention for effective noise suppression in both frequency branches.
• Evaluated on WorldView-3 and GaoFen-2 datasets, the approach achieves superior PSNR, SSIM, and reduced spectral distortions compared to state-of-the-art methods.

CGFformer: Cluster-Guidance Frequency Transformer for Pansharpening

Introduction

The pansharpening task in remote sensing targets the synthesis of high-resolution multispectral (HRMS) images by fusing low-resolution multispectral (LRMS) images with high-resolution panchromatic (PAN) images. Mainstream approaches leverage frequency-based fusion techniques, typically employing fixed-domain frequency separation and deep learning models to enhance spatial fidelity or spectral consistency. However, conventional frequency filters demonstrate poor adaptivity to diverse spatial-frequency patterns, resulting in suboptimal component separation and inadequate noise suppression. The "CGFformer: Cluster-Guidance Frequency Transformer for Pansharpening" (2605.01490) introduces a unified Transformer-based framework explicitly designed to address these limitations by integrating spatially adaptive frequency separation, dual-frequency denoising guided by mutual cross-attention, and an effective spatial-frequency fusion mechanism.

Methodology

Adaptive Frequency Separation with Cluster Guidance

The cornerstone of CGFformer is the Cluster Adaptive Frequency Separation (CAFS) module, which implements content-adaptive, spatially non-local frequency decomposition. The CAFS module employs a content-adaptive non-local (CAN) block, which leverages K-means clustering over local feature windows to partition image regions into non-local similarity groups. Dedicated CNN-generated adaptive filters are applied to each cluster, enabling accurate, position-adaptive separation of high- and low-frequency components based on both local and global feature correlations.

Figure 1: Comparative overview of frequency separation strategies, including the proposed non-local cluster-based adaptive filtering, with associated visual and quantitative error maps.

The efficacy of this clustering-based frequency separation is demonstrated to surpass traditional Gaussian or Fourier filtering approaches, as fixed and patch-based filters neglect non-local structure. As further evidenced by quantitative and qualitative results, CAFS yields superior high- and low-frequency component discrimination crucial for robust pansharpening.

Figure 3: Detailed schematic of the CAFS module, including feature similarity partitioning and partition-guided filter generation via MLP layers for each cluster.

Dual-Stream Refinement for Frequency-Guided Denoising

High- and low-frequency features generated from CAFS are susceptible to both frequency-relevant and frequency-irrelevant noise, often exacerbated by the separation process. The Dual-Stream Refinement (DSR) module addresses this through a cascaded architecture:

• Noise Calibration Block (NCB): Jointly estimates and suppresses frequency-relevant noise in both branches using spatial and channel attention mechanisms.
• Mutual Guidance Block (MGB): Applies cross-attention between the high- and low-frequency streams, guided by the denoised features, enabling dual-frequency noise removal and promoting structure consistency.
• Feature Gating Block (FGB): Further refines output via adaptive gating for information compaction and redundancy suppression.

  Figure 5: DSR and SFA module architectures, highlighting the calibration, mutual guidance, and feature gating operations, as well as the spatial-fusion branches.

This dual-stream, attention-driven approach provides comprehensive noise suppression not achievable by independent denoising or attention-alone models. The resulting features are clean, robust, and structurally consistent.

Spatial-Frequency Attention (SFA) Module for Fusion

To prevent frequency-only fusion from causing blurring and spatial loss, the SFA module incorporates complementary spatial and spatial-frequency attention mechanisms:

• SFA-S (Spatial Enhancement): Enhances structural detail by direct spatial channel refinement using linear MLPs.
• SFA-F (Space-Frequency Fusion): Integrates the outputs of DSR (high- and low-frequency) with enhanced spatial features via triplet attention. Queries are defined by the refined frequency features, while keys and values are derived from spatial information, ensuring strong spatial-frequency interaction and superior HRMS reconstruction.

Experimental Evaluation

Quantitative and Qualitative Results

Comprehensive experiments on WorldView-3 (WV3) and GaoFen-2 (GF-2) datasets provide strong empirical justification for each component of CGFformer. Across all reduced- and full-resolution settings, CGFformer yields the highest PSNR, SSIM, SCC, and the lowest SAM and ERGAS, outperforming both traditional and state-of-the-art deep learning or transformer-based pansharpening networks. Gains are most substantial in high-fidelity spatial and spectral indices, demonstrating the advantage of adaptive, non-local frequency separation and sophisticated dual-frequency denoising.

Figure 2: Visual results on WV3 reduced-resolution dataset, showing that CGFformer (t) reconstructs sharper contours, spectral consistency, and more faithful spatial details compared to state-of-the-art benchmarks.

Figure 4: Mean absolute error maps for the same setting, highlighting minimum residual errors for CGFformer, indicative of superior fusion accuracy.

Similar trends hold on GaoFen-2 data (Figures 8, 9, and 10), where CGFformer consistently reconstructs natural spatial boundaries and accurate colors, with minimal spectral distortion.

Ablation Studies

Frequency Separation: Quantitative ablation against Gaussian, Fourier, and local adaptive methods reveals that the cluster-guided strategy yields the best spatial-spectral metrics (PSNR 38.35, SSIM 0.978, SAM 2.57, and ERGAS 1.86). Increasing the cluster count improves adaptivity up to a point, with optimal separation observed near $K = 128$ for test images of typical size.

Figure 6: Visualization of cluster index matrices at multiple training epochs, demonstrating progression toward semantically meaningful groupings and spatial adaptation.

Figure 7: Performance metrics as a function of cluster number, elucidating the trade-off between global and local adaptivity.

Fusion and Network Modules: Removal of the DSR module or spatial-frequency fusion branches causes notable decreases in all performance indices, validating the necessity for each architectural component for optimal denoising and detail preservation.

Discussion and Implications

CGFformer establishes a new methodological paradigm for pansharpening by demonstrating that frequency separation must be both spatially adaptive and capable of non-local information aggregation. The cluster-based filter assignment in CAFS moves beyond local or global fixed convolution strategies, directly addressing the heterogeneous structure of remote sensing images. Dual-frequency attention-guided denoising in the DSR module ensures that noise and artifacts, particularly those introduced by frequency separation, are accurately removed, preventing the accumulation of distortions in the fusion stage. The tripartite SFA module ensures high-fidelity integration of spatial and frequency cues, ameliorating the typical loss of fine structure or spectral content.

Practically, the network is computationally efficient and modular, lending itself to deployment on large-scale or real-time pansharpening tasks across diverse remote sensing platforms. Theoretically, the findings generalize to cross-modal or multi-sensor fusion scenarios, suggesting that non-local adaptive filtering and joint attention-based denoising are transferable to other spatial-frequency fusion problems, including medical imaging or hyperspectral restoration.

Conclusion

CGFformer presents a robust, frequency-aware Transformer framework that unifies spatially adaptive frequency separation, sophisticated dual-frequency denoising, and complementary spatial-frequency fusion to realize state-of-the-art pansharpening performance. The approach establishes clear empirical superiority over both conventional and recent transformer-based models on multiple benchmarks, confirming the criticality of non-local adaptivity and joint noise suppression in frequency fusion. This work anticipates future research in cross-domain adaptive filtering, learned frequency analysis, and more generalized multi-modal fusion architectures for remote sensing