Natural Spectral Fusion (NSF)

• Natural Spectral Fusion (NSF) is a framework that fuses multispectral and hyperspectral data to preserve spectral information and enhance spatial detail.
• NSF employs advanced modules like MSG, SpaCAM, SpeCAM, and SSCFM to integrate multi-scale, cross-modal features for improved image resolution and fidelity.
• In optimization, NSF uses cyclic p-scheduling to modulate spectral bias, leading to early decision-boundary alignment and accelerated accuracy gains.

Natural Spectral Fusion (NSF) encompasses a set of methodologies and frameworks for spectral information preservation and controlled integration across distinct spectral channels or frequency bands, particularly in multispectral/hyperspectral imaging and in optimization algorithms. Central to NSF is the premise that fusion—whether of spatial, spectral, or information-theoretic content—should be “natural,” i.e., both faithful to the physics or semantics of the underlying data and adaptive to context, preserving complementary insight and augmenting information utility. Recent research treats NSF either as a principled architectural strategy for image fusion or a mechanism for optimizer spectral bias modulation.

1. Core Principles and Problem Formulation

NSF, in the context of imaging, addresses the challenge of reconstructing high-resolution hyperspectral images by integrating diverse sources: high-resolution multispectral data (HRMSI) and low-resolution hyperspectral data (LRHSI). The goal is to recover a representation that combines HRMSI’s spatial richness with LRHSI’s spectral completeness, yielding a fused output with both fine spatial detail and spectral fidelity. For optimization in machine learning, NSF reframes parameter updates as dynamic spectral filtering, emphasizing distributed “coverage” of frequency bands via algorithmic means rather than model architecture alone (Li, 12 Apr 2026, Zhang et al., 5 Sep 2025).

2. Architectural Realizations in Imaging

2.1 Multiscale and Cross-Modal Integration

State-of-the-art NSF architectures incorporate multi-scale, cross-modal mechanisms to encode and merge complementary information:

• Multi-Scale Generator (MSG): Constructs a three-level pyramid, generating local and global proxy feature streams. Formally, at each scale $l$,

$$\mathbf{F}_c^l = \mathcal{E}_c(\downarrow^2(\mathbf{F}_c^{l-1})),\quad \mathbf{F}_{c'}^l = \mathcal{E}_{c'}(\uparrow^2(\mathbf{F}_{c'}^{l+1}))$$

with $\mathbf{F}_c^0=\mathcal{E}_c(\mathbf{Y})$ and $\mathbf{F}_{c'}^4=\mathcal{E}_{c'}(\mathbf{X})$, where $\mathbf{Y}$, $\mathbf{X}$ are HRMSI and LRHSI inputs, respectively (Li, 12 Apr 2026).

• Spatial Coordinate-Aware Mixing (SpaCAM): Enhances spatial branch features through multi-dilation depth-wise convolutions with softmax-gated fusion:

$$\overline{\mathbf{F}_{spa}^i} = \mathrm{Softmax}(\mathbf{D}_j^i) \odot \sigma(\mathrm{DWConv}(\mathbf{F}_{spa}^i)) + \mathbf{F}_{spa}^i$$

supporting robust multi-scale context aggregation and edge recovery.

• Spectral Coordinate-Aware Mixing (SpeCAM): Decouples spectral streams via DWT into low/high frequency, applies a $C \times C$ spectral coordinate attention mechanism, and adaptively fuses:

$$\mathbf{F}_{mid} = \sigma(\mathrm{DWConv}(\tilde{f}_{low} + \tilde{f}_{high})), \quad \overline{\mathbf{F}_{spe}^i} = \mathbf{F}_{mid} + \mathbf{F}_{spe}^i$$

where $\alpha$ balances low/high-frequency emphasis.

• Spatial-Spectral Cross-Fusion Module (SSCFM): Implements dynamic, gated cross-modal alignment using large-kernel attention and per-channel gating, with final feature refinement via residual convolution:

$$\mathbf{F}_c^l = \mathcal{E}_c(\downarrow^2(\mathbf{F}_c^{l-1})),\quad \mathbf{F}_{c'}^l = \mathcal{E}_{c'}(\uparrow^2(\mathbf{F}_{c'}^{l+1}))$$0

The combined pipeline enables NSF to model and exploit both spatial detail and spectral correlation naturally, markedly improving upon methods that treat spatial or spectral branches in isolation (Li, 12 Apr 2026).

3. Information-Theoretic NSF in First-Order Optimization

Within learning algorithms, NSF conceptualizes the optimizer as a spectral controller:

• p-Exponent Cyclic Scheduling: Generalizes momentum-style updates to

$$\mathbf{F}_c^l = \mathcal{E}_c(\downarrow^2(\mathbf{F}_c^{l-1})),\quad \mathbf{F}_{c'}^l = \mathcal{E}_{c'}(\uparrow^2(\mathbf{F}_{c'}^{l+1}))$$1

where $$\mathbf{F}_c^l = \mathcal{E}_c(\downarrow^2(\mathbf{F}_c^{l-1})),\quad \mathbf{F}_{c'}^l = \mathcal{E}_{c'}(\uparrow^2(\mathbf{F}_{c'}^{l+1}))$$2 can be negative or positive. Positive $$\mathbf{F}_c^l = \mathcal{E}_c(\downarrow^2(\mathbf{F}_c^{l-1})),\quad \mathbf{F}_{c'}^l = \mathcal{E}_{c'}(\uparrow^2(\mathbf{F}_{c'}^{l+1}))$$3 implements low-pass filtering, negative $$\mathbf{F}_c^l = \mathcal{E}_c(\downarrow^2(\mathbf{F}_c^{l-1})),\quad \mathbf{F}_{c'}^l = \mathcal{E}_{c'}(\uparrow^2(\mathbf{F}_{c'}^{l+1}))$$4 amplifies high-frequency directions (Zhang et al., 5 Sep 2025).

• Cyclic Scheduling: $$\mathbf{F}_c^l = \mathcal{E}_c(\downarrow^2(\mathbf{F}_c^{l-1})),\quad \mathbf{F}_{c'}^l = \mathcal{E}_{c'}(\uparrow^2(\mathbf{F}_{c'}^{l+1}))$$5 is periodically modulated:

$$\mathbf{F}_c^l = \mathcal{E}_c(\downarrow^2(\mathbf{F}_c^{l-1})),\quad \mathbf{F}_{c'}^l = \mathcal{E}_{c'}(\uparrow^2(\mathbf{F}_{c'}^{l+1}))$$6

This tidal pattern ensures all frequency bands receive concentrated optimization effort. Such cyclical spectral allocation has been shown to drive early decision-boundary alignment and accelerate accuracy gains even as the overall loss decays more slowly.

• Spectral Interpretation: The frequency-wise gain is $$\mathbf{F}_c^l = \mathcal{E}_c(\downarrow^2(\mathbf{F}_c^{l-1})),\quad \mathbf{F}_{c'}^l = \mathcal{E}_{c'}(\uparrow^2(\mathbf{F}_{c'}^{l+1}))$$7, where $$\mathbf{F}_c^l = \mathcal{E}_c(\downarrow^2(\mathbf{F}_c^{l-1})),\quad \mathbf{F}_{c'}^l = \mathcal{E}_{c'}(\uparrow^2(\mathbf{F}_{c'}^{l+1}))$$8 is the exponentially-averaged spectral power, directly linking optimizer dynamics to spectral response.

4. Signal-Level NSF and Physical Complementarity

Alternative NSF realizations act directly on the signal:

• Two-Scale Image Decomposition: Inputs are decomposed via weight-guided and guided filters into base (smooth) and detail (textural) layers per channel (Li et al., 2023).
• Complementarity Weight Maps: Difference maps and extended-DoG filters compute per-pixel, per-scale weighting between visible and NIR contributions. The arctanI mapping adaptively enhances NIR detail in low-light scenes without compromising color naturalness in well-lit regions.
• Fusion Equation:

$$\mathbf{F}_c^l = \mathcal{E}_c(\downarrow^2(\mathbf{F}_c^{l-1})),\quad \mathbf{F}_{c'}^l = \mathcal{E}_{c'}(\uparrow^2(\mathbf{F}_{c'}^{l+1}))$$9

$\mathbf{F}_c^0=\mathcal{E}_c(\mathbf{Y})$0

enabling fine-grained, non-iterative, physics-grounded fusion.

Empirically, this approach achieves superior Color Distance, Structural Similarity, and PSNR over prior methods, demonstrating high-fidelity, artifact-free fusion in real-world conditions (Li et al., 2023).

5. Quantitative Performance and Empirical Outcomes

NSF architectures and algorithms have reported performance advantages on standard benchmarks:

• Imaging Fusion (CoFusion):
  • Chikusei $\mathbf{F}_c^0=\mathcal{E}_c(\mathbf{Y})$1 upsampling: PSNR = 53.09 dB vs. next best $\mathbf{F}_c^0=\mathcal{E}_c(\mathbf{Y})$252.47 dB; SAM = 1.876° vs. next best $\mathbf{F}_c^0=\mathcal{E}_c(\mathbf{Y})$32.00°.
  • PaviaU $\mathbf{F}_c^0=\mathcal{E}_c(\mathbf{Y})$4 upsampling: PSNR = 38.32 dB vs. best baseline 37.83 dB; SAM = 2.564° vs. $\mathbf{F}_c^0=\mathcal{E}_c(\mathbf{Y})$52.72°.
  • QNR (no-reference): $\mathbf{F}_c^0=\mathcal{E}_c(\mathbf{Y})$6 on Chikusei, besting other methods (Li, 12 Apr 2026).
• Optimizer NSF:
  • TinyImageNet, ResNet-18: validation top-1 increases from 59.1% (baseline) to 62.9% (NSF), with NSF matching baseline accuracy at only 25% of training cost.
  • Early acceleration of accuracy before loss convergence—empirical signature of early decision-boundary alignment (Zhang et al., 5 Sep 2025).
• VIS-NIR Fusion:
  • SSIM: up to 93.5% (urban), correlation up to 95.9%, edge preservation up to 75.5% (Ofir et al., 2023).
  • Compared to five state-of-the-art signal-based baselines, NSF achieves best or near-best Color Distance, Spectrum Distortion Index, SSIM, and PSNR (Li et al., 2023).

6. Loss Functions, Training Protocols, and Limitations

NSF frameworks typically deploy loss formulations that jointly penalize pixel error and structural distortion, such as

$\mathbf{F}_c^0=\mathcal{E}_c(\mathbf{Y})$7

with $\mathbf{F}_c^0=\mathcal{E}_c(\mathbf{Y})$8 in image fusion settings, and rely on self-supervised or task-consistent loss functions in the absence of external fusion ground truth (Li, 12 Apr 2026, Ofir et al., 2023).

For the optimizer-NSF regime, no explicit loss modification is necessary; NSF operates entirely through the optimizer dynamics, with negligible computational overhead.

Limitations cited include the dataset-dependent tuning of cyclic scheduling parameters in first-order optimization, and susceptibility to registration errors or model bias in imaging applications. Current theory remains at the level of simplified models; extension to highly nonconvex architectures and broader modalities is open.

7. Perspectives and Future Directions

NSF exposes the importance of treating fusion as an inherently cross-modal, frequency-aware process. Its deployment in high-resolution spectral imaging, first-order optimization, and signal-level fusion demonstrates a unified underlying principle: dynamic allocation and context-sensitive integration of spectral components yields measurable gains in information preservation and utility.

Research directions include deeper integration of NSF with architectural frequency controls (such as learnable Fourier features), adaptive and task-conditioned cyclic scheduling in optimization, and generalization to additional modalities beyond imaging, such as text and time series. The full realization of NSF’s potential will require both theoretical generalization and systematic benchmarking under diverse, real-world scenarios.