PIF-Net: Prior-Guided MH Image Fusion

• PIF-Net is a fusion framework for multispectral and hyperspectral image fusion that integrates ill-posed residual priors, invertible spectral modeling, and efficient spatial calibration.
• It employs a three-component design combining a prior extraction module, an invertible Mamba branch, and a Fusion-Aware LoRA module to tackle spectral under-determination and spatial misalignment.
• Empirical evaluations on benchmarks like PaviaU and Chikusei demonstrate improvements in PSNR, SSIM, SAM, and ERGAS, highlighting enhanced spectral fidelity and spatial detail.

Searching arXiv for "PIF-Net" to ground the article in the primary paper and disambiguate other uses of the term. PIF-Net is a multispectral–hyperspectral image fusion framework introduced to address the ill-posedness of multispectral and hyperspectral image fusion (MHIF), with the stated goal of recovering a high-resolution hyperspectral image that jointly preserves spectral fidelity from a low-resolution hyperspectral image and spatial detail from a high-resolution multispectral image (Li et al., 1 Aug 2025). In the formulation given for this method, the target output is a high-resolution hyperspectral image $Z \in \mathbb{R}^{H\times W\times C}$ predicted from a low-resolution hyperspectral input $X \in \mathbb{R}^{h\times w\times C}$ and a high-resolution multispectral input $Y \in \mathbb{R}^{H\times W\times c}$, with $C \gg c$ and scale factor $s = H/h$ (Li et al., 1 Aug 2025). The framework combines three stated components: an ill-posed residual prior, an invertible Mamba-based spectral branch, and a Fusion-Aware Low-Rank Adaptation module for lightweight spatial fusion (Li et al., 1 Aug 2025).

1. Problem setting and ill-posedness

Multispectral and hyperspectral image fusion targets the recovery of a high-resolution hyperspectral image (HR-HSI) that jointly preserves spatial detail from a high-resolution multispectral image (HR-MSI) with $c$ bands and spectral fidelity from a low-resolution hyperspectral image (LR-HSI) with $C \gg c$ bands (Li et al., 1 Aug 2025). The paper defines the fusion objective as a mapping $M(\cdot;\theta)$ such that $\hat{Z} = M(X, Y \mid \theta) \approx Z$, where $Z$ denotes the desired HR-HSI (Li et al., 1 Aug 2025).

The sensing model described for training-pair synthesis uses a latent high-resolution scene together with sensor spectral response, blur, downsampling, and noise. In the equivalent matrix form given in the paper, MSI formation is written as $X \in \mathbb{R}^{h\times w\times C}$0, and HSI formation as $X \in \mathbb{R}^{h\times w\times C}$1, where $X \in \mathbb{R}^{h\times w\times C}$2 maps $X \in \mathbb{R}^{h\times w\times C}$3 bands to $X \in \mathbb{R}^{h\times w\times C}$4 bands, $X \in \mathbb{R}^{h\times w\times C}$5 is decimation, $X \in \mathbb{R}^{h\times w\times C}$6 and $X \in \mathbb{R}^{h\times w\times C}$7 are spatial PSFs, and $X \in \mathbb{R}^{h\times w\times C}$8 are noise (Li et al., 1 Aug 2025). If cross-sensor parallax or time-lag exists, an additional spatial transform $X \in \mathbb{R}^{h\times w\times C}$9 can be included before blur or downsampling (Li et al., 1 Aug 2025).

The framework is explicitly motivated by three sources of ill-posedness. The first is spectral under-determination, because $Y \in \mathbb{R}^{H\times W\times c}$0 makes $Y \in \mathbb{R}^{H\times W\times c}$1 rank-deficient and the inverse from $Y \in \mathbb{R}^{H\times W\times c}$2 bands to $Y \in \mathbb{R}^{H\times W\times c}$3 bands non-unique. The second is spatial under-determination, because downsampling removes high-frequency content and blur further reduces rank. The third is misalignment, because cross-modal spatial or spectral mismatch makes direct pixelwise consistency unreliable (Li et al., 1 Aug 2025). The paper therefore treats recovery of the latent HR-HSI as underdetermined and argues that regularization, priors, and cross-modal constraints are essential (Li et al., 1 Aug 2025).

In the reported experiments, synthetic LR-HSI is generated by Gaussian blur $Y \in \mathbb{R}^{H\times W\times c}$4 followed by decimation, and MSI is simulated under the stated spectral mixing protocol with band response $Y \in \mathbb{R}^{H\times W\times c}$5 (Li et al., 1 Aug 2025). This training setup places the method within the standard synthetic MHIF evaluation regime while emphasizing misalignment-aware prior design.

2. Core design: ill-posed prior, invertible Mamba, and FAM-LoRA

PIF-Net is organized around three interacting mechanisms that, according to the paper, address MHIF ill-posedness in complementary ways (Li et al., 1 Aug 2025). The first is the Ill-Posed Residual Prior Extraction Module (IPRPEM), the second is an invertible Mamba architecture for spectral–frequency modeling, and the third is the Fusion-Aware Low-Rank Adaptation module (FAM-LoRA) for lightweight spatial calibration (Li et al., 1 Aug 2025).

The ill-posed prior is designed to extract spatially invariant cues that persist under small spatial shifts and to amplify cross-modal residuals indicating where MSI spatial detail should guide HSI reconstruction (Li et al., 1 Aug 2025). Let $Y \in \mathbb{R}^{H\times W\times c}$6 and $Y \in \mathbb{R}^{H\times W\times c}$7 denote shallow feature tensors extracted from the upsampled HSI and HR-MSI after a shared Conv–ReLU–Conv head. The residue-channel modulation is defined as

$Y \in \mathbb{R}^{H\times W\times c}$8

and the prior is computed by

$Y \in \mathbb{R}^{H\times W\times c}$9

The resulting $C \gg c$0 is described as a robust, spatially invariant ill-posed residual prior injected into the spatial fusion branch (Li et al., 1 Aug 2025).

The invertible Mamba branch is introduced to balance global spectral modeling with computational efficiency while maintaining information consistency during feature transformation and fusion (Li et al., 1 Aug 2025). The framework operates in the frequency domain: upsampled HSI features are decomposed by a 2D Haar wavelet into low-frequency (LF) and high-frequency (HF) components, and an invertible affine coupling is applied through Segmented Spectral Mamba Modules built with 2D SSM kernels (SS2D) and light feed-forward layers (Li et al., 1 Aug 2025). The coupling equations are

$C \gg c$1

The paper states that this triangular-Jacobian design supports controllable log-volume and yields the log-determinant term used in invertibility regularization (Li et al., 1 Aug 2025).

FAM-LoRA provides the lightweight spatial branch adaptation. For a weight $C \gg c$2, the low-rank parameterization is

$C \gg c$3

with small rank $C \gg c$4, scaling $C \gg c$5, and learned factors $C \gg c$6 and $C \gg c$7 (Li et al., 1 Aug 2025). Adapters are inserted after channel transformation and around attention-like mixers, specifically a Large-Kernel Attention module and an SE gate, with a multi-head design that splits channels evenly into four heads (Li et al., 1 Aug 2025). The calibration mechanism uses the prior-guided auxiliary input $C \gg c$8 together with the main spatial feature $C \gg c$9, and one described instantiation applies SE-style reweighting:

$s = H/h$0

This is presented as a way to bias channel weighting toward misalignment-driven corrections while keeping the model lightweight (Li et al., 1 Aug 2025).

A plausible implication is that the framework’s overall architecture is not merely a fusion stack but an explicit decomposition of the ill-posed MHIF problem into prior extraction, reversible spectral transport, and parameter-efficient spatial adaptation. That interpretation follows the paper’s own separation of functions across $s = H/h$1, $s = H/h$2, and $s = H/h$3 (Li et al., 1 Aug 2025).

3. End-to-end pipeline and optimization

The pipeline begins with LR-HSI and HR-MSI inputs, where the LR-HSI is first upsampled by bicubic interpolation to the HR spatial resolution (Li et al., 1 Aug 2025). The aligned feature-space representation is then passed through Conv–ReLU–Conv and a linear projection to $s = H/h$4 channels, with $s = H/h$5 in the reported implementation (Li et al., 1 Aug 2025).

The spectral or frequency branch performs 2D Haar wavelet decomposition on the $s = H/h$6-channel features into LF and HF components (Li et al., 1 Aug 2025). HF is channel-compressed to $s = H/h$7 through a $s = H/h$8 convolution, LF is processed by $s = H/h$9 invertible Mamba blocks with $c$0, and inverse wavelet transform merges LF and HF to produce a spectral reference $c$1 followed by a Tail Conv–ReLU–Conv (Li et al., 1 Aug 2025). In parallel, the High-Frequency Semantic Perception Module fuses upsampled HF features with the ill-posed residual prior $c$2 to form a global spatial semantic reference (Li et al., 1 Aug 2025).

The spatial branch is a sequence of $c$3 FAM-LoRA blocks. Each block takes a main spatial feature $c$4 and a prior-guided auxiliary feature $c$5, applies channel transforms, LKA, SE with $c$6 injection, and multi-head LoRA adapters, then reconstructs the final fused output $c$7 through a tail module (Li et al., 1 Aug 2025). The paper identifies both $c$8 and $c$9 as guidance signals for consistency and calibration (Li et al., 1 Aug 2025).

The complete training objective combines reconstruction, invertibility regularization, and cross-representation consistency:

$C \gg c$0

with

$C \gg c$1

The paper reports $C \gg c$2 and $C \gg c$3 (Li et al., 1 Aug 2025). It further states that $C \gg c$4 regularizes the affine-coupling Jacobian volume and that $C \gg c$5 enforces angular agreement between branch outputs (Li et al., 1 Aug 2025).

Training and implementation are reported in PyTorch on a single NVIDIA A30, using AdamW with learning rate $C \gg c$6 halved every 200 epochs, 500 total epochs, and batch size 8 (Li et al., 1 Aug 2025). Training targets are $C \gg c$7 GT HR-HSI patches, with corresponding LR-HSI and HR-MSI inputs; for a $C \gg c$8 setting the LR-HSI input size is $C \gg c$9 (Li et al., 1 Aug 2025). The paper does not explicitly specify normalization or augmentation beyond the synthesis protocol, noting only that standard per-dataset normalization is implied by deep restoration practice (Li et al., 1 Aug 2025).

4. Empirical evaluation

The reported evaluation uses three benchmark datasets: Chikusei, Pavia University, and Houston (Li et al., 1 Aug 2025). Chikusei is described as an airborne HSI dataset with 128 bands and spatial size $M(\cdot;\theta)$0, with the top-left $M(\cdot;\theta)$1 region for training and non-overlapping $M(\cdot;\theta)$2 tiles for testing (Li et al., 1 Aug 2025). Pavia University has 103 bands and size $M(\cdot;\theta)$3, with the top $M(\cdot;\theta)$4 portion used for testing and the remainder for training (Li et al., 1 Aug 2025). Houston has 144 bands and size $M(\cdot;\theta)$5, with the left $M(\cdot;\theta)$6 region as test data and the rest for training (Li et al., 1 Aug 2025).

The evaluation metrics are PSNR, SSIM, SAM, and ERGAS (Li et al., 1 Aug 2025). The spectral angle mapper is defined as

$M(\cdot;\theta)$7

and ERGAS is reported with lower values indicating better performance (Li et al., 1 Aug 2025). Baselines include classical fusion methods Brovey and GSA, and deep MHIF models HSRnet, Fusformer, PSRT, U2Net, 3DT-Net, and SMGU-Net (Li et al., 1 Aug 2025).

The paper reports the following representative quantitative results (Li et al., 1 Aug 2025):

Setting	Reported result
PaviaU $M(\cdot;\theta)$8	PSNR 39.8246 dB, SSIM 0.9845, SAM 2.4018, ERGAS 1.8126
Chikusei $M(\cdot;\theta)$9	PSNR 50.0124 dB, SSIM 0.9979, SAM 2.2603, ERGAS 1.7328

For PaviaU $\hat{Z} = M(X, Y \mid \theta) \approx Z$0, the paper states that the method beats the second-best result by approximately $\hat{Z} = M(X, Y \mid \theta) \approx Z$1 dB PSNR (Li et al., 1 Aug 2025). It also reports consistent improvements across Houston and Chikusei at $\hat{Z} = M(X, Y \mid \theta) \approx Z$2 and $\hat{Z} = M(X, Y \mid \theta) \approx Z$3, with notable SAM and ERGAS reductions that are interpreted as superior spectral fidelity (Li et al., 1 Aug 2025). Qualitatively, the reported outputs exhibit sharper edges and textures, fewer artifacts, and lower SAM maps across scenes, with the authors attributing this to invertible spectral modeling and LoRA-calibrated spatial fusion (Li et al., 1 Aug 2025).

These results position PIF-Net as a method that attempts to improve both spectral fidelity and spatial restoration simultaneously rather than privileging one side of the trade-off. That interpretation is directly supported by the simultaneous improvement pattern across PSNR, SSIM, SAM, and ERGAS reported in the paper (Li et al., 1 Aug 2025).

5. Ablation findings and efficiency profile

The ablation study assigns a central role to the ill-posed residual prior. On PaviaU $\hat{Z} = M(X, Y \mid \theta) \approx Z$4, the scalar prior strength $\hat{Z} = M(X, Y \mid \theta) \approx Z$5 improves PSNR from 36.95 dB at $\hat{Z} = M(X, Y \mid \theta) \approx Z$6 to 39.82 dB at $\hat{Z} = M(X, Y \mid \theta) \approx Z$7, and the paper states that $\hat{Z} = M(X, Y \mid \theta) \approx Z$8 is best in that ablation (Li et al., 1 Aug 2025). This is presented as evidence that the prior helps resolve cross-modal ambiguities (Li et al., 1 Aug 2025).

Removing the invertible Mamba block reduces PaviaU $\hat{Z} = M(X, Y \mid \theta) \approx Z$9 performance from 39.82/0.9845 (PSNR/SSIM) to 37.18/0.9637, while removing FAM-LoRA reduces it to 36.92/0.9607 (Li et al., 1 Aug 2025). The paper interprets these drops as showing the importance of invertible global modeling and LF–HF bidirectional coupling, and of low-rank fusion-aware spatial adaptation, respectively (Li et al., 1 Aug 2025).

The loss ablation further decomposes the contribution of the optimization terms (Li et al., 1 Aug 2025):

Loss configuration	PSNR / SSIM on PaviaU $Z$0
$Z$1 alone	37.45 / 0.9187
$Z$2	38.68 / 0.9342
$Z$3	38.12 / 0.9278
$Z$4	39.82 / 0.9845

The paper states that $Z$5 stabilizes and preserves information flow, whereas $Z$6 enforces spectral-shape consistency between branches (Li et al., 1 Aug 2025). The joint objective achieves the best reported result in that analysis (Li et al., 1 Aug 2025).

On computational efficiency, the complete model is reported to have approximately 1.73 million parameters and to run at approximately 9.3 ms per image on an NVIDIA A30 for PaviaU $Z$7 (Li et al., 1 Aug 2025). The method is described as real-time capable, and LoRA is explicitly identified as a major contributor to parameter efficiency (Li et al., 1 Aug 2025). The paper also argues that Mamba’s linear-time global modeling scales better with resolution than quadratic self-attention, and that invertibility can reduce memory through reversible backpropagation while LoRA confines most adaptation to low-rank factors (Li et al., 1 Aug 2025).

6. Limitations, reproducibility, and terminology

The reported limitations concern alignment assumptions, synthetic supervision, underexplored LoRA hyperparameters, and robustness outside the tested regimes (Li et al., 1 Aug 2025). The method relies on $Z$8 and consistency losses for soft alignment, and the paper notes that severe geometric misregistration may require an explicit spatial transformer or deformable alignment (Li et al., 1 Aug 2025). It also notes that real sensors exhibit complex, band-dependent PSFs and nonideal responses, so learning $Z$9 and $X \in \mathbb{R}^{h\times w\times C}$00 from data or integrating physics-aware operators could improve transfer beyond the synthetic training protocol (Li et al., 1 Aug 2025). Rank $X \in \mathbb{R}^{h\times w\times C}$01 and head partition in LoRA are not exhaustively studied, and adaptive rank selection or sparsity-promoting variants such as DoRA are proposed as possible directions (Li et al., 1 Aug 2025). Performance beyond $X \in \mathbb{R}^{h\times w\times C}$02 scaling and under severe noise or blur is not reported (Li et al., 1 Aug 2025).

On robustness, the paper states that the design targets misalignment tolerance through $X \in \mathbb{R}^{h\times w\times C}$03 guidance and invertibility, but also notes that extensive stress tests with large shifts or heavy noise are not reported (Li et al., 1 Aug 2025). It nevertheless cites strong SAM and ERGAS gains and generalization across three datasets and multiple scales as evidence of improved spectral robustness (Li et al., 1 Aug 2025).

Reproducibility information is partial. Code and model availability are explicitly stated as not given in the paper (Li et al., 1 Aug 2025). The paper nevertheless specifies the method’s core components in implementable terms: shallow Conv–ReLU–Conv heads and tails, forward and inverse 2D Haar wavelets, VMamba (SS2D) blocks inside affine-coupling layers, IPRPEM as defined, FAM-LoRA with auxiliary $X \in \mathbb{R}^{h\times w\times C}$04 injection, and the stated AdamW schedule with $X \in \mathbb{R}^{h\times w\times C}$05, $X \in \mathbb{R}^{h\times w\times C}$06, $X \in \mathbb{R}^{h\times w\times C}$07, $X \in \mathbb{R}^{h\times w\times C}$08, batch size 8, and 500 epochs (Li et al., 1 Aug 2025). A plausible implication is that the architecture is reproducible at the level of components and hyperparameters but not yet at the level of a released reference implementation.

The term “PIF-Net” is used in multiple, unrelated senses in the broader literature represented in the supplied sources. In the photographic-style transfer paper “Personalized Image Filter: Mastering Your Photographic Style,” the authors state that “PIF-Net” does not appear in the paper and that the method is called PIF rather than PIF-Net (Zhu et al., 19 Oct 2025). In “Estimating Social Influence from Observational Data,” “PIF-Net” refers only to a variant of Poisson Influence Factorization that constructs per-person substitute confounders from the network alone, not to a neural architecture (Sridhar et al., 2022). In neuroimaging, “PIF-Net” denotes CNNs using patch individual filter layers in higher layers to exploit spatial homogeneity of normalized brain MRI (Eitel et al., 2020). Within current imaging literature, however, the capitalized term PIF-Net most directly denotes “PIF-Net: Ill-Posed Prior Guided Multispectral and Hyperspectral Image Fusion via Invertible Mamba and Fusion-Aware LoRA” (Li et al., 1 Aug 2025).