Clustered Frequency Aggregation for Pansharpening

• Clustered Frequency Aggregation (CFA) is a frequency-fusion module that employs region-aware sparse attention to selectively aggregate wavelet-domain features in pansharpening.
• It integrates Cluster-Routed Sparse Attention (CRSA) with a Gated Depthwise Forward Network to compress inter-region interactions while preserving intra-cluster details.
• By leveraging SAR-derived semantic clusters, CFA reduces computational complexity and enhances pansharpening performance, as evidenced by improved PSNR, SSIM, and other metrics.

Searching arXiv for the exact term and closely related uses of “CFA” to ground the article in current literature. Clustered Frequency Aggregation (CFA) is the frequency-fusion module of RAFNet, a Region-Aware Fusion Network for pansharpening, where it replaces full global self-attention with a region-aware sparse attention mechanism over wavelet-domain features (Zhang et al., 4 May 2026). In RAFNet, CFA operates on the frequency features $\bm{P}_k$ produced within a progressive spatial-frequency pipeline, performs Cluster-Routed Sparse Attention (CRSA) followed by a Gated Depthwise Forward Network (GDFN), and uses semantic cluster masks inherited from the Spatial Adaptive Refinement (SAR) module to restrict dense interactions to semantically coherent regions while compressing cross-region interactions through cluster summaries (Zhang et al., 4 May 2026). Its stated purpose is to reduce the quadratic computational cost and redundant semantically irrelevant interactions of standard scaled dot-product attention for high-resolution remote sensing imagery.

1. Definition and design objective

CFA is introduced to address two linked limitations of frequency-based pansharpening: the quadratic complexity of global attention and the fact that many query-key interactions in remote sensing scenes are redundant because large areas are homogeneous or semantically weakly related (Zhang et al., 4 May 2026). The module is motivated by an empirical observation reported in RAFNet: the attention distribution is long-tailed, with 90% of the attention energy concentrated in only 57.1% of the tokens, and the attention matrix exhibits a block-diagonal sparse structure. In the formulation of the paper, this indicates that attention is regionally sparse and should be routed mainly within semantically coherent regions rather than computed exhaustively over all token pairs.

Within pansharpening, this objective is tied to the dual requirement of preserving spectral consistency from low-resolution multispectral imagery and fine spatial detail from panchromatic imagery. CFA addresses this by aggregating frequency information selectively. Dense interactions are preserved where local semantic consistency is high, and semantically weaker cross-region interactions are summarized rather than expanded into full pairwise attention. A plausible implication is that the module is designed not merely as an efficiency device, but as a structural prior for spatial-frequency fusion in remote sensing scenes.

2. Placement within RAFNet’s progressive spatial-frequency pipeline

RAFNet is described as a two-stage progressive wavelet pyramid architecture, motivated by the fact that one discrete wavelet transform (DWT) halves spatial resolution (Zhang et al., 4 May 2026). At stage $k$, the network processes an intermediate multispectral feature $\bm{M}_k$ together with a panchromatic frequency feature $\bm{P}_k$, with stage transition written as

$\bm{M}_{k+1} = \mathcal{F}(\bm{M}_k,\bm{P}_k),$

where $\mathcal{F}$ denotes the combined SAR + CFA pipeline.

CFA is not an isolated block. It receives the frequency features already decomposed into four Haar-wavelet sub-bands,

$$\bm{P}_b = (\bm{P}_k \ast \bm{f}_b)\downarrow_2,\quad b\in\{LL,LH,HL,HH\},$$

and these sub-bands provide the frequency structure on which sparse aggregation is performed. The workflow described in RAFNet is: frequency features at stage $k$ are processed by CRSA; the resulting sub-band outputs are inverse-wavelet-transformed back to the spatial domain; the fused result is refined by GDFN; and, in deeper stages, CFA is cascaded again with a fresh clustering step on the updated $LL$ features (Zhang et al., 4 May 2026).

This placement is significant because it explains why CFA is framed as a fusion module rather than a generic sparse Transformer block. Its inputs, routing signals, and outputs are all defined relative to wavelet-domain decomposition and stagewise pansharpening reconstruction. The module therefore couples global frequency aggregation to the same multi-level reconstruction process that SAR uses for spatially adaptive enhancement.

3. Cluster-Routed Sparse Attention (CRSA)

The core computational component of CFA is Cluster-Routed Sparse Attention. For each wavelet band $b\in\{LL,LH,HL,HH\}$, the queries come from the corresponding frequency band $k$0, the key comes from the low-frequency band $k$1, and the value comes from a fused representation of the intermediate multispectral feature $k$2 with $k$3 (Zhang et al., 4 May 2026). The preprocessing is

$k$4

CRSA departs from standard scaled dot-product attention by using the semantic cluster mask $k$5 to route interactions. If query and key belong to the same cluster, attention is dense at the pixel level. If they belong to different clusters, pixel-wise interactions are replaced by interactions with a cluster-mean key,

$k$6

The sparse attention score is written as

$k$7

and the attention output for each band is

$k$8

The four reconstructed frequency features are then reassembled by inverse DWT:

$k$9

This mechanism defines the specific sense in which the module is “clustered.” The clustering does not remove global context entirely; rather, it changes the granularity of global interaction. Intra-cluster relations are preserved exactly at pixel level, while inter-cluster relations are compressed into cluster-level summaries. The result is a sparse attention operator whose sparsity pattern is semantic rather than purely geometric.

4. Semantic clustering, SAR dependence, and post-attention refinement

A frequent misunderstanding is to treat CFA as if it performs semantic clustering internally. In RAFNet, the clusters are inherited from the SAR module rather than created inside CFA itself (Zhang et al., 4 May 2026). SAR performs differentiable soft K-means on the low-frequency wavelet band $\bm{M}_k$0, using both feature similarity and spatial proximity. For a pixel with embedding $\bm{M}_k$1 and spatial coordinate $\bm{M}_k$2, the soft assignment to cluster $\bm{M}_k$3 is

$\bm{M}_k$4

The final hard cluster index matrix $\bm{M}_k$5 is obtained by argmax, with straight-through estimation used so that hard assignments remain trainable.

SAR and CFA have complementary roles. SAR performs spatially adaptive refinement through DWT, clustering, region-specific dynamic convolutions, and IDWT, while CFA uses the resulting region semantics to guide sparse global frequency aggregation (Zhang et al., 4 May 2026). The paper explicitly states that removing CFA harms performance and that removing SAR also harms performance. This supports a precise interpretation: CFA is not a substitute for adaptive local processing, and SAR is not a substitute for global cross-frequency modeling.

After CRSA, CFA applies a Restormer-style GDFN:

$\bm{M}_k$6

and

$\bm{M}_k$7

where $\bm{M}_k$8 and $\bm{M}_k$9 are $\bm{P}_k$0 depthwise convolutions, $\bm{P}_k$1 is elementwise multiplication, and $\bm{P}_k$2 is a $\bm{P}_k$3 projection. This post-attention refinement is part of the module definition rather than an auxiliary add-on: CFA in RAFNet consists of CRSA followed by GDFN.

5. Computational profile, implementation, and empirical behavior

Relative to standard attention,

$\bm{P}_k$4

which incurs $\bm{P}_k$5 time and memory for $\bm{P}_k$6 tokens, CFA reduces complexity by computing dense interactions only inside clusters and approximating inter-cluster links with cluster-mean keys (Zhang et al., 4 May 2026). The complexity analysis reported in RAFNet gives $\bm{P}_k$7 for intra-cluster dense interactions and $\bm{P}_k$8 for inter-cluster modeling, so the total cost is far below full quadratic attention when $\bm{P}_k$9.

The implementation details reported for RAFNet are specific. The framework is PyTorch; training uses an NVIDIA RTX 5090, Adam, an initial learning rate of $\bm{M}_{k+1} = \mathcal{F}(\bm{M}_k,\bm{P}_k),$0, a learning-rate schedule halved every 90 epochs, 500 training epochs, batch size 32, and 8 attention heads in CRSA (Zhang et al., 4 May 2026). The number of clusters is $\bm{M}_{k+1} = \mathcal{F}(\bm{M}_k,\bm{P}_k),$1 during training and $\bm{M}_{k+1} = \mathcal{F}(\bm{M}_k,\bm{P}_k),$2 during testing. The paper also notes that $\bm{M}_{k+1} = \mathcal{F}(\bm{M}_k,\bm{P}_k),$3 materially affects behavior: values that are too small make clustering overly coarse, whereas values that are too large make clusters too sparse, with best inference performance observed around $\bm{M}_{k+1} = \mathcal{F}(\bm{M}_k,\bm{P}_k),$4.

The empirical evidence presented for CFA is both ablative and architectural. In the attention ablation comparing Local Block Attention, Banded Attention, Global Cross Attention, and CRSA, CRSA is reported as best, with PSNR 38.3770, SSIM 0.9775, SCC 0.9857, SAM 2.5386, and ERGAS 1.9687 (Zhang et al., 4 May 2026). In module ablation, an Only-Spatial variant without CFA is worse than the full model, and an Only-Frequency variant without SAR is also worse, indicating complementarity rather than redundancy. The paper further reports complexity figures of 3.09M total parameters for RAFNet, 2.92M Transformer parameters, and 1.38G FLOPs, and states that extensive experiments on multiple benchmark datasets show RAFNet significantly outperforms state-of-the-art pansharpening methods in both reduced- and full-resolution assessments (Zhang et al., 4 May 2026).

6. Scope, misconceptions, and acronym disambiguation

Within RAFNet, Clustered Frequency Aggregation has a narrow technical meaning: it is a region-aware sparse attention module for wavelet-domain feature fusion in pansharpening (Zhang et al., 4 May 2026). It is therefore distinct from several other uses of the acronym “CFA” in arXiv literature.

One distinct usage appears in time-series meta-forecasting, where CFA denotes the “Continuous Frequency Adapter,” an adversarial domain-adaptation mechanism for learning frequency-invariant representations across sampling frequencies (Ness et al., 2023). Another appears in ensemble learning, where CFA denotes “Combinatorial Fusion Analysis,” a framework using rank-score characteristic functions and cognitive diversity for combining multiple scoring systems (Roginek et al., 9 Mar 2026). In programming-languages theory, $\bm{M}_{k+1} = \mathcal{F}(\bm{M}_k,\bm{P}_k),$5-CFA denotes $\bm{M}_{k+1} = \mathcal{F}(\bm{M}_k,\bm{P}_k),$6-level control-flow analysis, whose control-flow decision problem is EXPTIME-complete for every fixed $\bm{M}_{k+1} = \mathcal{F}(\bm{M}_k,\bm{P}_k),$7 (Horn et al., 2013). In image processing, “CFA” commonly abbreviates color filter array, as in Bayer CFA denoising and demosaicking pipelines (Buades et al., 2018).

Related optical terminology introduces a further source of ambiguity. In Kerr microresonator physics, clustered frequency comb formation arises through a two-stage nonlinear process in which a primary widely detuned parametric sideband pair is created by phase-matched four-wave mixing and at least one sideband lies in anomalous dispersion, enabling further comb formation through modulational instability and additional four-wave mixing (Sayson et al., 2018). This is conceptually unrelated to RAFNet’s sparse attention module even though both involve “clustered frequency” language.

The most important misconception, therefore, is terminological rather than algorithmic: “CFA” is not a unique term across the literature. In the context of pansharpening and RAFNet, however, Clustered Frequency Aggregation refers specifically to the CRSA + GDFN frequency-fusion module guided by SAR-derived semantic clusters, designed to replace full attention with semantically routed sparse aggregation in a progressive wavelet architecture (Zhang et al., 4 May 2026).