RAFNet: Region-Aware Fusion Network for Pansharpening

Abstract: Pansharpening aims to generate high-resolution multispectral (HRMS) images by fusing low-resolution multispectral (LRMS) and high-resolution panchromatic (PAN) images. Although deep learning has advanced this field, mainstream frequency-based methods relying on standard scaled dot-product attention suffer from quadratic computational complexity and fail to exploit the inherent regional sparsity of remote sensing imagery. Furthermore, existing spatial enhancement strategies typically employ static convolution kernels, which struggle to adapt to the complex frequency and regional variations of PAN and MS images. To address these bottlenecks, we propose a Region-Aware Fusion (RAFNet) Network that synergistically models spatial and frequency information. Specifically, we design a Spatial Adaptive Refinement (SAR) module that leverages the discrete wavelet transform (DWT) for directional frequency separation and K-means clustering for regional partitioning, which enables the dynamic construction of region-specific adaptive convolution kernels, achieving spatially and frequency-adaptive feature enhancement. Moreover, we introduce a Clustered Frequency Aggregation (CFA) module based on a sparse attention mechanism guided by the semantic clusters, which executes a region-aware sparse attention strategy that drastically reduces computational redundancy while ensuring high-quality frequency feature extraction. In addition we integrated these modules into a progressive, multi-level spatial-frequency network architecture to facilitate robust interaction and accurate image reconstruction. Extensive experiments on multiple benchmark datasets demonstrate that the proposed RAFNet significantly outperforms state-of-the-art pansharpening methods in both reduced- and full-resolution assessments. The code is available at https://github.com/PatrickNod/RAFNet.

• The paper introduces RAFNet, which uses a Spatial Adaptive Refinement module based on discrete wavelet transform and soft K-Means clustering to enable region-specific dynamic convolutions.
• It incorporates a Clustered Frequency Aggregation module with region-aware sparse attention to reduce computational load while enhancing spectral-spatial consistency.
• Quantitative results on WorldView-3 and GaoFen-2 datasets show that RAFNet outperforms existing methods in PSNR, SSIM, and other key metrics, delivering minimal distortion with optimal efficiency.

RAFNet: Region-Aware Fusion Network for Pansharpening

Motivation and Problem Formulation

The pansharpening task is critical in remote sensing, aiming to reconstruct high-resolution multispectral (HRMS) images by fusing low-resolution multispectral (LRMS) data with high-resolution panchromatic (PAN) observations. Existing methods fall into component substitution, multi-resolution analysis, or variational optimization paradigms, but all suffer from notable spectral distortion, spatial artifacts, or reliance on domain-specific priors limiting adaptability. Recent advances in deep learning have delivered improvements, but mainstream spatial enhancement relies on static convolutions, inherently lacking spatial adaptability and imposing spectral bias favoring low-frequency details. Frequency-domain fusion with transformers promises global dependency modeling, but quadratic complexity and information redundancy severely hinder practical deployment.

Spatial and Frequency Adaptivity: SAR Module Design

RAFNet primarily leverages a Spatial Adaptive Refinement (SAR) module, parameterized by frequency and spatial region. The SAR employs 2D discrete wavelet transform (DWT) to decouple input imagery into orthogonal frequency sub-bands. To capture non-local similarities and regional heterogeneity, it applies a differentiable soft K-Means clustering (with spatial penalty) over the low-frequency (LL) feature embedding, yielding a cluster matrix as regional mask, enabling spatial adaptivity.

Dynamic convolution kernels are generated for each identified cluster using centroid feature vectors mapped through low-rank MLPs, efficiently producing region-specific filters for each frequency sub-band (LL, LH, HL, HH). This synergy enables joint spatial-frequency enhancement, extracting localized details while preserving region-wise semantics.

Figure 1: Paradigmatic progression from static, dynamic, context-aware, and frequency-adaptive convolutions, culminating in region-adaptive kernels tailored per frequency component via DWT.

Region-Aware Sparse Attention: CFA Module Construction

The Clustered Frequency Aggregation (CFA) module exploits the spatial clustering mask from SAR to guide attention routing, drastically mitigating quadratic computational burden typical of global dot-product attention. RAFNet’s Cluster-Routed Sparse Attention (CRSA) enforces dense, pixel-wise dot products only within clusters, while inter-cluster interactions are aggregated via cluster-mean keys, enhancing sparsity and computational efficiency.

Empirical analysis shows that 90% of attention energy is accounted for by only ~57% of tokens, confirming that spatial semantic alignment dominates attention distribution. The CRSA logic strictly follows these semantic boundaries, ensuring that intra-cluster interactions maintain detail, and inter-cluster fusion achieves efficient summary statistics.

Figure 2: CDF energy curves, cluster masks, and attention matrices illustrate the regional sparsity and block-diagonal structure underlying attention, motivating semantically routed sparse attention.

Network Architecture and Loss Function

RAFNet is instantiated as a hierarchical, multi-level framework, with progressive DWT decompositions forming a wavelet pyramid. At each scale, SAR and CFA process PAN and LRMS representations, operating jointly. The framework supports recursive spatial-frequency fusion, culminating in the reconstruction of HRMS imagery via a final convolutional mapping from the deep, fused representation.

The loss function is an $\ell_1$ criterion, providing strict pixelwise fidelity and stable convergence.

Figure 3: End-to-end architecture of RAFNet—joint spatial frequency decomposition, adaptive convolution, and region-aware sparse attention modules for HRMS image synthesis.

Quantitative and Qualitative Performance

Experimental results across WorldView-3 and GaoFen-2 datasets for both reduced- and full-resolution settings are reported. RAFNet achieves the highest PSNR, SSIM, SCC, lowest SAM and ERGAS, and outperforms the second-best FSGformer across all metrics. Full-resolution no-reference evaluation (HQNR, $D_\lambda$, $D_s$) further demonstrates excellent spectral and spatial consistency, with minimal distortion.

Figure 4: Visual results and residuals for all approaches on WorldView-3 show RAFNet’s sharper edges and minimal error.

Figure 5: Visual results and residuals for Gaofen-2 emphasize RAFNet’s precise spatial reconstruction.

Figure 6: Detailed qualitative results on full-resolution GaoFen-2 data validate the network’s structural and spectral performance.

Computational Efficiency and Ablation Study

RAFNet achieves optimal trade-off between parameter count and performance. Its CRSA module, by restricting dense dot-products to $\mathcal{O}\left(\frac{N^2}{K}\right)$ intra-cluster interactions and using lightweight $\mathcal{O}(N K)$ inter-cluster keys, maintains transformer-level fidelity with dramatically fewer parameters and FLOPs.

Ablation studies confirm the superiority of CRSA over local block, banded, or global cross attention, and underline the mutual indispensability of SAR and CFA. Cluster count analysis demonstrates optimal performance with $K=64$ for inference, adapting to spatial extent and detail.

Figure 7: Cluster index matrices evolve through training, indicating non-local semantic structure captured by SAR’s clustering.

Figure 8: PSNR, SAM, ERGAS metrics vary with $K$; optimal values achieved around $K=64$.

Theoretical and Practical Implications

RAFNet establishes a generalized spatial-frequency fusion paradigm, synergizing region-adaptive convolution and region-aware sparse attention. The end-to-end differentiable K-Means clustering enables semantic region partitioning at scale, aligning convolution and attention operations with the innate heterogeneity of remote sensing data. The framework rigorously prunes computational redundancy, enabling robust and efficient high-resolution synthesis, and is extensible to broader image fusion scenarios with similar spatial-frequency complexity. Semantic clustering-based routing mechanisms are well poised to become standard in computationally constrained high-fidelity image processing pipelines.

Conclusion

RAFNet delivers a technically advanced solution to pansharpening, integrating spatial adaptive refinement and clustered frequency aggregation to achieve state-of-the-art spectral-spatial fusion. The architecture achieves strong quantitative and qualitative results, superior computational efficiency, and theoretical generalizability. These results endorse the region-aware spatial-frequency paradigm for scalable remote sensing imagery fusion and set the foundation for future sparse attention mechanisms tailored to semantic regionality in large-scale image synthesis (2605.02184).