Enhancing Unregistered Hyperspectral Image Super-Resolution via Unmixing-based Abundance Fusion Learning

Abstract: Unregistered hyperspectral image (HSI) super-resolution (SR) typically aims to enhance a low-resolution HSI using an unregistered high-resolution reference image. In this paper, we propose an unmixing-based fusion framework that decouples spatial-spectral information to simultaneously mitigate the impact of unregistered fusion and enhance the learnability of SR models. Specifically, we first utilize singular value decomposition for initial spectral unmixing, preserving the original endmembers while dedicating the subsequent network to enhancing the initial abundance map. To leverage the spatial texture of the unregistered reference, we introduce a coarse-to-fine deformable aggregation module, which first estimates a pixel-level flow and a similarity map using a coarse pyramid predictor. It further performs fine sub-pixel refinement to achieve deformable aggregation of the reference features. The aggregative features are then refined via a series of spatial-channel abundance cross-attention blocks. Furthermore, a spatial-channel modulated fusion module is presented to merge encoder-decoder features using dynamic gating weights, yielding a high-quality, high-resolution HSI. Experimental results on simulated and real datasets confirm that our proposed method achieves state-of-the-art super-resolution performance. The code will be available at https://github.com/yingkai-zhang/UAFL.

• The paper introduces UAFL, an unmixing-based framework that preserves spectral fidelity with fixed SVD-derived endmembers while learning spatially enhanced abundance residuals from unregistered RGB guidance.
• The method combines coarse-to-fine deformable aggregation, spatial-channel abundance cross-attention, and modulated fusion to handle misalignment without explicit pixel-level image warping.
• UAFL achieves 42.05, 37.23, and 32.28 dB PSNR on REAL at ×4, ×8, and ×16 scales, respectively, while using 5.94M parameters and 96.17G FLOPs—substantially less than leading alternatives.

This paper proposes an unmixing-based fusion framework for unregistered hyperspectral image (HSI) super-resolution (SR), addressing two shortcomings of existing approaches: the textural distortions introduced by explicit pre-alignment modules, and the constrained learning capacity of networks that fuse spatial-spectral data in a coupled manner. The method, referred to as UAFL, reformulates the fusion task as residual abundance map learning, and introduces three components: a coarse-to-fine deformable aggregation (CFDA) module, a spatial-channel abundance cross-attention (SCACA) block, and a spatial-channel modulated fusion (SCMF) module. Experiments on the simulated ICVL dataset and the real-world REAL dataset show state-of-the-art accuracy at a fraction of the computational cost of comparable methods.

Motivation and formulation

The starting observation is that HSIs are inherently low-rank due to strong spectral correlations, so spectral unmixing can decouple spatial structure from spectral signatures. Prior work on simultaneous SR and geometric alignment noted that unmixing-based formulations are more robust to misalignment than methods that impose explicit alignment constraints. The authors substantiate this with an empirical analysis: mixing LR HSI endmembers $\mathbf{E}_{lrhsi}$ with high-resolution, well-structured abundances $\mathbf{A}_{hrhsi}$ reconstructs a high-quality HSI, whereas directly mixing $\mathbf{E}_{lrhsi}$ with an unregistered HR RGB image produces a visually plausible but quantitatively poor result. This motivates reframing fusion as learning to enhance decoupled abundance maps rather than performing coupled spatial-spectral fusion, which the paper argues simplifies the optimization objective.

The pipeline is as follows. The LR HSI $\mathcal{X} \in \mathbb{R}^{h \times w \times B}$ is bilinearly upsampled to $\mathcal{X}_{\uparrow}$, then singular value decomposition is applied: the first $K$ left singular vectors form the endmember matrix $\mathbf{E} \in \mathbb{R}^{B \times K}$, and the initial abundance map is computed as $$\mathbf{A} = \mathbf{E}^{\mathrm{T}}\mathbf{X}_{\uparrow}$$. A network $f(\cdot\vert\theta)$ then predicts an enhanced abundance map conditioned on the unregistered HR RGB reference $\mathcal{I}$, the residual is mixed back through $\mathbf{A}_{hrhsi}$0, and the result is added to $\mathbf{A}_{hrhsi}$1 to produce the final HR HSI. Notably, the endmembers are fixed by the initial SVD and only the abundances are learned; this is what preserves spectral fidelity while delegating spatial enhancement to the network.

Coarse-to-fine deformable aggregation

The CFDA module performs implicit alignment at the feature level, avoiding explicit pixel warping of the reference image. A lightweight coarse pyramid predictor downsamples the abundance feature and reference feature, predicts a low-resolution flow, and upsamples it as a coarse motion prior. A residual flow and a similarity map (passed through a sigmoid) are then estimated from the warped and target features, yielding the prior flow $\mathbf{A}_{hrhsi}$2 and confidence map $\mathbf{A}_{hrhsi}$3.

Because the coarse flow lacks sub-pixel precision, a fine refinement stage encodes the decimal component of the flow using frequency positional encoding (sinusoidal features of increasing frequency, following NeRF-style encoding), and a refinement network predicts a residual deformable offset and modulation mask. The final offsets $\mathbf{A}_{hrhsi}$4 combine the prior flow with a tanh-bounded residual, and the modulation mask $\mathbf{A}_{hrhsi}$5 is the sigmoid of the similarity map element-wise multiplied with the predicted mask. A modulated deformable convolution then aggregates the reference feature. This design follows the implicit resampling alignment idea from video SR, adapted to cross-modal abundance fusion.

Spatial-channel abundance cross-attention

The SCACA block refines the aggregated features through hierarchical cross-attention. The reference feature is first self-modulated via a depth-wise $\mathbf{A}_{hrhsi}$6 convolutional gating. In the spatial branch, window-based cross-attention is applied where the Query, Key, and Value derive from the abundance feature, but the Value is modulated by the reshaped reference feature before attention aggregation; attention includes a learnable relative position bias. In the channel branch, the Value is modulated by the reference feature directly, adaptively recalibrating channel-wise responses. The paper credits this design with enabling fine-grained spatial correspondence learning while enhancing salient spectral signatures of the abundance map.

Spatial-channel modulated fusion

The SCMF module merges encoder-decoder features through two parallel gating pathways. The spatial pathway applies depth-wise convolutions to produce a spatial value feature and a sigmoid-gated pixel-wise weight from the concatenated encoder-decoder features. The channel pathway produces a channel-mixed value via $\mathbf{A}_{hrhsi}$7 convolution and channel-wise gates from global average pooling. The two modulated outputs are summed and added residually to the decoder feature, preserving information from the decoder stream and stabilizing training.

Experimental results

The method is evaluated on ICVL (201 image pairs with simulated viewpoint misregistration via virtual camera pose adjustment) and REAL (60 unregistered HSI–RGB pairs captured with a dual-camera SOC710-VP system), using PSNR, SSIM, and SAM. Training uses AdamW with an $\mathbf{A}_{hrhsi}$8 loss, a batch size of 1, on a single RTX 4090.

On ICVL at scale factor $\mathbf{A}_{hrhsi}$9, the method achieves 41.84 dB PSNR and 0.025 SAM, exceeding SSCH-S (41.38 dB) and HSIFN (41.14 dB), with the best SAM among all compared methods including bicubic interpolation.

On the REAL dataset, the method leads across all scale factors and metrics:

Method	PSNR $\mathbf{E}_{lrhsi}$0	PSNR $\mathbf{E}_{lrhsi}$1	PSNR $\mathbf{E}_{lrhsi}$2	Params (M)	FLOPs (G)
HSIFN	40.15	34.39	30.07	21.01	594.10
SSCH-S	41.16	36.19	31.91	11.01	165.68
Ours	42.05	37.23	32.28	5.94	96.17

The gains are 0.89 dB, 1.04 dB, and 0.37 dB over the second-best method at $\mathbf{E}_{lrhsi}$3, $\mathbf{E}_{lrhsi}$4, and $\mathbf{E}_{lrhsi}$5 respectively. The efficiency claim is notable: the model uses roughly half the parameters and 42% fewer FLOPs than SSCH-S while outperforming it, supporting the paper's argument that decoupling spatial-spectral fusion improves learnability. Visual comparisons show near-uniform error maps, whereas competing methods exhibit high-error patterns along sharp edges.

Ablation studies

The component-wise ablation on REAL ($\mathbf{E}_{lrhsi}$6) proceeds from a self-attention baseline at 41.26 dB. Adding the unmixing strategy yields 41.41 dB; adding SCACA yields 41.66 dB (while reducing parameters from 5.08M to 4.85M); adding CFDA yields 41.95 dB; and adding SCMF reaches 42.05 dB. Two further observations stand out. First, replacing CFDA with generic DCNv2 gives 41.80 dB versus 41.95 dB, and the visualized fused feature maps show that DCNv2 produces severe artifacts and blurred text while CFDA yields cleaner representations — evidence that the flow-guided, sub-pixel-refined offsets matter beyond the deformable mechanism itself. Second, the SCMF gain grows with scale factor (0.10 dB at $\mathbf{E}_{lrhsi}$7 but 0.38 dB at $\mathbf{E}_{lrhsi}$8), indicating the modulated fusion becomes more valuable as the reconstruction problem hardens.

Limitations and open questions

The paper does not include a dedicated limitations discussion, and several constraints can be identified from the experimental setup. The unmixing initialization depends on SVD with a fixed number of endmembers $\mathbf{E}_{lrhsi}$9; the paper does not analyze sensitivity to $\mathcal{X} \in \mathbb{R}^{h \times w \times B}$0 or to scenes violating the linear mixing assumption (e.g., nonlinear mixing or strong shadows). Evaluation on REAL is limited to 10 test pairs, so the reported margins, particularly the 0.37 dB gain at $\mathcal{X} \in \mathbb{R}^{h \times w \times B}$1, rest on a small sample without reported variance. The REAL dataset contains rigid, real-world misalignment from a fixed dual-camera rig; behavior under large parallax, occlusion, or non-rigid deformation between the HSI and RGB sensors is not characterized. Finally, ablations are conducted only at the $\mathcal{X} \in \mathbb{R}^{h \times w \times B}$2 scale for the component breakdown, leaving the interaction between unmixing-based decoupling and very large scale factors less examined.

Conclusion

The paper demonstrates that decoupling unregistered HSI SR into fixed-endmember, learned-abundance fusion is an effective alternative to explicit alignment followed by coupled spatial-spectral fusion. The combination of flow-guided sub-pixel deformable aggregation, spatial-channel abundance cross-attention, and gated encoder-decoder fusion delivers consistent state-of-the-art results on both simulated and real unregistered data, with substantially reduced parameter and FLOP budgets relative to the strongest prior method. The main open questions concern robustness of the fixed SVD-based unmixing initialization, statistical strength of the real-data evaluation, and generalization beyond the rigid misalignment regime.