RAPConv: Adaptive Pansharpening Convolution

• The paper introduces RAPConv, a novel convolution method that adapts kernels based on local feature context for improved pansharpening.
• RAPConv employs a dual-branch design with a local feature branch and a Global Harmonic Bias Module, leading to better metrics such as ERGAS, Q8, and SCC.
• Integrated into RAPNet, RAPConv supports dynamic feature fusion and precise spatial detail extraction to generate high-resolution multispectral outputs.

Receptive-field Adaptive Pansharpening Convolution (RAPConv) is a deep learning operation specifically developed for pansharpening, the process of fusing a high-resolution panchromatic (PAN) image with a lower-resolution multispectral (MS) image to produce a high-resolution multispectral output. RAPConv departs from conventional convolution by generating location-specific convolutional kernels that adapt to the local content at each spatial position, enhancing spatial detail extraction and mitigating the limitations of uniform kernel application that characterize standard convolutions (Tang et al., 14 Jul 2025).

1. Mathematical Formulation and Mechanism

RAPConv operates on an input feature map $X \in \mathbb{R}^{C \times H \times W}$ (with $C$ channels, spatial size $H \times W$) and produces an output $Y \in \mathbb{R}^{P \times H \times W}$. Unlike standard convolution, which applies a fixed learnable kernel $W \in \mathbb{R}^{P \times C \times k \times k}$ to all positions, RAPConv generates a unique modulation tensor $K_p \in (0,1)^{P \times C \times k \times k}$ for each spatial location $p = (x, y)$. The resulting operation at each location $p$ is:

$$Y_p = \sum_{q \in R} [K_p \circ W]_{:,:,q} \cdot X_{:,p+q}$$

where $R$ is the $C$0 receptive field and $C$1 denotes the Hadamard product. The kernel modulation $C$2 is determined by local feature context, enabling location-dependent feature extraction. The local receptive field is modeled by:

$C$3

with $C$4 being the sigmoid activation, $C$5 realizing $C$6 filters grouped by channel, and $C$7 providing channel-wise spatial averages broadcast across the map.

A location-specific tensor $C$8 is reshaped from $C$9 and broadcast across output channels to form $H \times W$0.

2. Kernel Generation and Bias Integration

The RAPConv kernel-generation process consists of:

• Local Feature Branch:
  • Global average pooling of the input ($H \times W$1).
  • Group-wise $H \times W$2 convolution to generate kernel weights.
  • Sigmoid non-linearity to ensure modulation coefficients are in $H \times W$3.
  • Per-location rearrangement to construct $H \times W$4 from the coefficients.
• Global Harmonic Bias Module (GHBM):
  • Channel pooling ($H \times W$5).
  • Two-stage bottleneck via $H \times W$6 convolutions and ReLU activations to produce a bias term $H \times W$7, broadcast spatially as $H \times W$8.
  • The final RAPConv output is $H \times W$9.

This design allows RAPConv to generate both content-adaptive kernel weights and a spatially broadcast bias, both derived from the input.

3. RAPConv in the RAPNet Pansharpening Architecture

RAPConv forms the backbone of the RAPNet network for pansharpening, with the architectural pipeline as follows:

1. PAN and MS images are processed by Edge Spatial Attention Modules (ESAM).
2. The MS image is upsampled to the PAN spatial resolution and concatenated with PAN-derived features.
3. An initial $Y \in \mathbb{R}^{P \times H \times W}$0 convolution is applied.
4. The feature map passes through $Y \in \mathbb{R}^{P \times H \times W}$1 RAP-ResBlocks (experiments use $Y \in \mathbb{R}^{P \times H \times W}$2), where each RAP-ResBlock consists of a sequence: [$Y \in \mathbb{R}^{P \times H \times W}$3 RAPConv $Y \in \mathbb{R}^{P \times H \times W}$4 PReLU $Y \in \mathbb{R}^{P \times H \times W}$5 $Y \in \mathbb{R}^{P \times H \times W}$6 RAPConv] + identity skip connection.
5. Features undergo Pansharpening Dynamic Feature Fusion (PAN-DFF) with the upsampled MS input.
6. Final reconstruction produces the high-resolution multispectral (HRMS) output.

RAPConv distinguishes itself from conventional convolution by adapting both kernels and additive bias at every spatial location, providing increased representational flexibility.

4. PAN-DFF: Adaptive Feature Fusion for Pansharpening

Pansharpening Dynamic Feature Fusion (PAN-DFF) complements RAPConv by adaptively balancing spatial detail and spectral fidelity. Given backbone features $Y \in \mathbb{R}^{P \times H \times W}$7 and upsampled MS features $Y \in \mathbb{R}^{P \times H \times W}$8, PAN-DFF computes attention gates:

• Spectral gate: $Y \in \mathbb{R}^{P \times H \times W}$9, broadcast to match $W \in \mathbb{R}^{P \times C \times k \times k}$0 spatial dimensions.
• Spatial gate: $W \in \mathbb{R}^{P \times C \times k \times k}$1, similarly broadcast.

Fusion is performed by:

$W \in \mathbb{R}^{P \times C \times k \times k}$2

where $W \in \mathbb{R}^{P \times C \times k \times k}$3 denotes element-wise multiplication. This mechanism enables the network to dynamically weight the contributions of detail-rich and spectrally faithful representations on a per-channel basis.

5. Empirical Evaluation and Performance Analysis

RAPNet, employing RAPConv, is benchmarked on the WorldView-3 simulated dataset (20 images, $W \in \mathbb{R}^{P \times C \times k \times k}$4). Key evaluation metrics include ERGAS (lower is better), SAM (spectral angle, lower is better), Q8 (higher is better), and SCC (spatial correlation coefficient, higher is better). The following results were observed (Tang et al., 14 Jul 2025):

Method	ERGAS ↓	SAM ↓	Q8 ↑	SCC ↑
FusionNet	2.492	3.372	0.899	0.979
RAPNet	2.353	3.369	0.902	0.982

RAPNet attains the highest Q8 and SCC, reflecting enhanced spatial detail extraction. Ablation studies demonstrate that replacing RAPConv with standard convolution yields substantial performance degradation (ERGAS: 2.639→2.353, Q8: 0.891→0.902), confirming the critical role of receptive-field adaptation.

In real-world WorldView-3 tests, the QNR metric increased from 0.9363 (FusionNet) to 0.9374 (RAPNet), while Da and Ds values further indicate improved spatial transfer with negligible spectral distortion.

6. Comparisons and Relations to Adaptive Convolution Methods

RAPConv shares conceptual ground with dynamic and content-adaptive convolutions but is tailored for the pansharpening context through spatially adaptive modulation and the inclusion of global harmonic bias. In contrast, techniques such as DSConv (Dynamic Splitting Convolution) dynamically select and activate a subset of sub-kernels via masking, focusing computation on the most informative offsets and achieving a different balance of efficiency and adaptivity (Liu et al., 8 Aug 2025).

Both RAPConv and DSConv aim to enlarge and vary the effective receptive field according to local context, though DSConv utilizes a binary mask and shifting mechanics, while RAPConv employs soft, sigmoid-based gating and spatially broadcast bias integration.

7. Limitations and Perspectives

While RAPConv enables localized adaptation critical for remote sensing fusion tasks, its reliance on sigmoid-parameterized gating can introduce smoothing effects. A plausible implication is that specialized training or additional sparsity constraints may further enhance localization or computational efficiency. Moreover, as RAPConv’s receptive field is position- and content-dependent, its interpretability and optimization may present additional challenges relative to standard convolutions. Ongoing work compares the tradeoffs between such adaptive mechanisms and alternative dynamic convolution schemes in varied vision domains (Tang et al., 14 Jul 2025, Liu et al., 8 Aug 2025).