HF-Aware Feature Fusion Module

• HF-aware feature fusion modules explicitly decompose and integrate high-frequency details using frequency-domain transforms and adaptive filtering.
• They effectively preserve crucial textures and boundaries, thereby improving dense predictions and multi-modal image fusion quality.
• Empirical studies show that architectures like FDFM and FreqFusion yield significant gains in metrics such as mIoU and ERGAS.

A high-frequency–aware (HF-aware) feature fusion module is a neural architectural component designed to preserve, enhance, and selectively integrate high-frequency details—such as edges, textures, and fine structures—across multiple feature maps or modalities. These modules are increasingly critical in dense prediction, multi-modal fusion, and image synthesis tasks, where accurate texture and precise boundary localization are essential to resolving semantic and spatial ambiguity. HF-aware fusion operators explicitly model and control the propagation of high-frequency information during feature combination, typically leveraging frequency-domain transforms, adaptive filtering, edge priors, or explicit frequency similarity constraints.

1. Principles and Motivations for HF-Aware Feature Fusion

Conventional feature fusion in deep neural networks often applies simple operations such as element-wise addition, concatenation, or channel-wise multiplications. While computationally efficient, these approaches are insensitive to the differing spectral content and structural roles of features from distinct modalities or hierarchies. Such insensitivity can result in blurred boundaries, inconsistent intra-category feature distributions, and insufficient enhancement of fine image details—especially detrimental in segmentation, image fusion, and scene parsing.

HF-aware modules are motivated by core challenges:

• High-frequency components are spatially sparse yet semantically critical for textures and boundaries;
• Naive fusion can corrupt or suppress these details, especially during upsampling or global aggregation;
• Different input sources (e.g., infrared vs. visible, PAN vs. multispectral) may offer complementary or conflicting HF cues.

As a result, HF-aware fusion modules seek to:

• Explicitly decompose, filter, or emphasize frequency sub-bands, particularly HF bands;
• Adaptively align and merge HF features from multiple sources;
• Constrain the loss landscape to preserve or reconstruct accurate HF content during learning.

2. Representative Architectures

2.1 Frequency Domain Fusion Module (FDFM)

The FDFM, as implemented in RPFNet (Zheng et al., 9 Jul 2025), is structured around a dual-branch design:

• Spatial shortcut branch: Retains local spatial detail via a standard 3×3 Conv–BN–ReLU block;
• Frequency branch: Achieves efficient, global, and true convolution in the frequency domain. A forward FFT is applied to the 3×3 convolved features, followed by a 1×1 convolution (acting as a learned complex filter), and then an inverse FFT reconstructs the result in the spatial domain.

Formally, for an input feature $\widehat{\mathcal{F}_F^0}$: $$\mathcal{F}_F^1 = \mathrm{CB}_{3\times3}(\widehat{\mathcal{F}_F^0}) + \mathcal{F}^{-1} \Big\{ \mathrm{CB}_{1\times1} \big( \mathcal{F}\{ \mathrm{CB}_{3\times3}(\widehat{\mathcal{F}_F^0}) \} \big) \Big\}$$

Residual priors encoding modality difference (from a cross-promotion module) pre-condition the fusion by focusing frequency-domain convolutional processing onto salient, typically high-frequency regions (Zheng et al., 9 Jul 2025).

2.2 Frequency-Aware Feature Fusion (FreqFusion)

The FreqFusion module (Chen et al., 2024) is organized as follows:

• Adaptive Low-Pass Filter (ALPF) generator: Predicts a spatially-variant smoothing kernel to suppress spurious HF inside objects and upsample high-level features;
• Offset generator: Predicts spatially-varying offsets to resample features, refining thin boundaries and reducing intra-class inconsistencies;
• Adaptive High-Pass Filter (AHPF) generator: Predicts spatially-variant high-pass filters to extract and restore fine boundary details lost by downsampling.

Fusion proceeds by smoothing and upsampling the coarse feature map, refining via offset-guided sampling, and residual-adding back high-pass refined details from low-level feature maps. This design enables decoupling of noise attenuation (low-pass) and boundary texture enhancement (high-pass).

2.3 Wavelet-Based Frequency-Aware Block

FAFNet (Xing et al., 2022) utilizes a multi-stage frequency-aware pipeline:

• Feature decomposition into low- and high-frequency subbands via discrete wavelet transforms (DWT);
• Parallel convolutional extraction of subband features;
• Fusion of multispectral and panchromatic HF components at multiple scales, guided by explicit high-frequency feature similarity loss to maximize alignment and sharpness.

2.4 Edge-Aware Multimodal Fusion

EGFNet (Zhou et al., 2021) extends HF awareness by directly injecting edge priors—computed as fused Sobel-filtered gradients of modalities—into the fusion process. Its multimodal fusion module interleaves cross-modal gating, residual convolutional refinement, and multi-scale dilated convolutions. The edge prior further modulates both semantic and boundary features, with multitask deep supervision reinforcing the preservation of high-frequency boundaries.

3. Mathematical Formulation and Theoretical Insights

A common foundation across HF-aware modules is the explicit frequency representation of signals. The 2D Discrete Fourier Transform (DFT) and its convolution theorem elucidate the equivalence between global convolution in spatial and local filtering in frequency domains:

$$\mathcal{F}\{ f * g \}(u,v) = \mathcal{F}\{ f \}(u,v) \cdot \mathcal{F}\{ g \}(u,v)$$

$$\mathbf{Y}_f(u,v) = \mathbf{X}_f(u,v) \cdot \mathcal{H}(u,v)$$

Learned convolutional filters in the frequency domain (as in FDFM) or adaptive high-pass/low-pass kernels (as in FreqFusion) allow the network to spatially localize (via attention or offset) and spectrally focus (via kernel weights) its fusion computation.

Losses targeting HF alignment explicitly regularize learning:

• Adaptive weight-based frequency contrastive loss aligns fused output HF to reference modalities via Fourier normed differences (Zheng et al., 9 Jul 2025);
• High-frequency feature similarity (HFS) loss matches cross-correlation of HF features between modalities, reducing spectral bias (Xing et al., 2022);
• Structural SSIM or boundary-aware losses preserve local spatial coherence and texture.

4. Integration into Deep Networks and Computational Considerations

HF-aware feature fusion modules are integrated into various backbone contexts:

• In FPN-style decoders for dense prediction, FreqFusion replaces standard upsample-add steps, jointly updating coarse and fine-resolution representations (Chen et al., 2024);
• In multi-modal fusion pipelines (e.g., infrared–visible or MS–PAN), frequency-aware modules align the complementary spatial detail content (Zheng et al., 9 Jul 2025, Xing et al., 2022);
• For RGB-Thermal scene parsing, edge-enhanced attention and supervision reinforce HF detail propagation (Zhou et al., 2021).

Efficient frequency-domain convolution (using FFTs) and spatially-variant adaptive filters ensure that the additional computational cost is moderate—FreqFusion, for example, increases per-fusion-stage compute by only ∼5–10% with ∼0.3M parameters and 2–3 GFLOPs per step (Chen et al., 2024).

5. Empirical Performance and Ablation Evidence

Extensive ablations demonstrate the superiority of HF-aware modules over conventional or Transformer-based fusions for tasks demanding sharp detail and robust contextual integration:

Study	Baseline Method	HF-Aware Module	Main Gains
RPFNet (Zheng et al., 9 Jul 2025)	Transformer block	FDFM	VIFF: 0.675→0.402, SF: significant drop prevented
FreqFusion (Chen et al., 2024)	Vanilla upsample+add	FreqFusion	mIoU: 41.7→44.5, bIoU: 27.8→32.8 in ADE20K
FAFNet (Xing et al., 2022)	SOTA pansharpening	FAFNet	ERGAS: 1.136 (vs. 1.503), Q4 and SAM improved
EGFNet (Zhou et al., 2021)	Plain addition	MFM+edge prior	mAcc: 68.1→72.7%, mIoU: 53.1→54.8%

These results confirm that such modules preserve high-frequency content, enhance boundaries, and provide better semantic and spatial accuracy, particularly in challenging multi-modal and high-resolution fusion tasks.

6. Summary of Key HF-Aware Fusion Strategies

• Frequency-domain convolution: True global context aggregation with negligible spatial attenuation, linear in $HW$, and directly compatible with attention modulation.
• Adaptive spatial filtering: Pixel-wise learned high- and low-pass filters enable targeted denoising and sharpening of frequency content.
• Wavelet-based feature decomposition: Provides multi-scale, interpretable splitting of LF and HF, suitable for highly structured fusion (e.g., pansharpening).
• Edge-guided fusion: Prioritizes boundary accuracy by seeding attention with explicit edge maps derived from the input sources.
• Spectral and spatial losses: Fusion regularized by loss terms directly sensitive to high-frequency mismatch, misclassification at boundaries, or feature similarity.

Together, these design elements constitute the current state of HF-aware feature fusion, which is now regarded as essential for high-fidelity prediction and multi-modal integration in contemporary computer vision pipelines (Zheng et al., 9 Jul 2025, Chen et al., 2024, Xing et al., 2022, Zhou et al., 2021).