Physics-Informed Untrained Learning for RGB-Guided Superresolution Single-Pixel Hyperspectral Imaging

Abstract: Single-pixel imaging (SPI) offers a cost-effective route to hyperspectral acquisition but struggles to recover high-fidelity spatial and spectral details under extremely low sampling rates, a severely ill-posed inverse problem. While deep learning has shown potential, existing data-driven methods demand large-scale pretraining datasets that are often impractical in hyperspectral imaging. To overcome this limitation, we propose an end-to-end physics-informed framework that leverages untrained neural networks and RGB guidance for joint hyperspectral reconstruction and super-resolution without any external training data. The framework comprises three physically grounded stages: (1) a Regularized Least-Squares method with RGB-derived Grayscale Priors (LS-RGP) that initializes the solution by exploiting cross-modal structural correlations; (2) an Untrained Hyperspectral Recovery Network (UHRNet) that refines the reconstruction through measurement consistency and hybrid regularization; and (3) a Transformer-based Untrained Super-Resolution Network (USRNet) that upsamples the spatial resolution via cross-modal attention, transferring high-frequency details from the RGB guide. Extensive experiments on benchmark datasets demonstrate that our approach significantly surpasses state-of-the-art algorithms in both reconstruction accuracy and spectral fidelity. Moreover, a proof-of-concept experiment using a physical single-pixel imaging system validates the framework's practical applicability, successfully reconstructing a 144-band hyperspectral data cube at a mere 6.25% sampling rate. The proposed method thus provides a robust, data-efficient solution for computational hyperspectral imaging.

• The paper introduces a physics-informed, untrained network framework for SPHI that leverages RGB guidance to achieve robust hyperspectral reconstruction.
• The methodology combines analytical initialization with dual-stage neural architectures (UHRNet and USRNet) to enhance spatial and spectral resolution.
• Experiments demonstrate state-of-the-art performance under severe undersampling, achieving up to 42.38 dB PSNR and superior noise robustness.

Physics-Informed Untrained Learning for RGB-Guided Superresolution Single-Pixel Hyperspectral Imaging

Introduction

Single-pixel hyperspectral imaging (SPHI) systems enable spectral imaging with minimal hardware but face a severe ill-posed inverse problem at low sampling rates due to underdetermined measurements and the high dimensionality of data. Classic approaches, including compressed sensing and regularization-based optimization, suffer substantial degradation under heavy undersampling, producing poor reconstructions with noise, artifacts, and distorted spectral content. Data-driven deep learning solutions have demonstrated improved performance but at the cost of large-scale annotated datasets, which are rarely feasible in hyperspectral imaging contexts, especially for unaligned or diverse sensing platforms.

This paper introduces a rigorously physics-informed, untrained neural network framework for SPHI that employs RGB guidance to achieve robust, high-fidelity hyperspectral reconstruction and superresolution, all without reliance on external training data or paired RGB-HSI datasets. The proposed method integrates physical sensing models, RGB structural priors, untrained convolutional and transformer-based networks, and an advanced loss architecture, resulting in a scalable solution for extreme undersampling regimes.

Figure 1: The architecture integrates SPI physics, RGB guidance, and untrained neural networks in a three-stage pipeline (LS-RGP, UHRNet, USRNet) for RGB-guided hyperspectral image recovery and superresolution.

Methodology

Physics-Informed Problem Formulation and Initialization

In the SPHI system, the imaging task is the recovery of a hyperspectral image cube $X \in \mathbb{R}^{H \times W \times B}$ given a set of 1D measurements generated from structured patterns $P_m \in \mathbb{R}^{H \times W}$ at extremely low sampling rates ($M \ll HW$). The inverse problem, highly underdetermined, is initially addressed by regularized least-squares (LS) with an analytically derived grayscale prior obtained from the RGB image, yielding robust initialization and direct exploitation of cross-modal structure:

$$\hat{x}_b = (P^\top P + \lambda I)^{-1}(P^\top y_b + \lambda I_{\text{gray}})$$

This LS-RGP initialization preserves global spatial correlations, accelerating convergence and improving stability of the subsequent untrained neural network optimization stages.

Dual-Stage Untrained Neural Network Architecture

The method employs a sequential, untrained architecture:

• UHRNet: A U-Net backbone with SE attention, utilizing the RGB grayscale as explicit guidance, is optimized per-instance (no pretraining) with measurement-consistency, spectral Fourier, sharpness, spatial smoothness, and perceptual (VGG-feature) losses. This hybrid loss enforces physical measurement fidelity, spectral smoothness, and structural realism, efficiently mitigating noise and artifact amplification under sparse sampling conditions.
• USRNet: A transformer-based superresolution network, leveraging cross-modal attention, adapts the initial spectral cube to high spatial resolution via transfer of high-frequency content from the RGB guide. Transformer multi-head self-attention and cross-modal fusion enable the capture of both global contextual information and fine spatial-spectral details.

Both networks are untrained, with their parameters entirely optimized from scratch on each test instance, allowing integration of physical constraints and alleviation of dataset bias and domain generalization problems.

Experimental Results

Quantitative and Qualitative Performance

Experiments on the CAVE hyperspectral image benchmark demonstrate clear superiority over leading SPI inversion algorithms, both optimization- and deep learning-based, under extreme undersampling (6.25%):

Figure 2: Reconstruction quality for Bands 1, 16, and 21 across methods. The proposed method consistently achieves higher PSNR/SSIM and visual fidelity where alternatives fail.

The framework yields PSNR up to 42.38 dB and SSIM 0.98 at Band 16, with an overall average PSNR of 38.14 dB and SAM 0.13 rad, significantly outperforming strong baselines such as MST++ (25.44 dB, 0.86, 0.27 rad). Fine spatial and spectral detail is preserved, even under strong measurement noise.

Robustness and Adaptability

Figure 3: Systematic evaluation shows higher PSNR, SSIM, and lower SAM for the proposed method under diverse SNR conditions, confirming superior noise robustness.

Figure 4: Varying the number of measurement patterns, the method retains performance advantage, with PSNR >25 dB and SSIM ≈0.8 using only 4 patterns.

Notably, the framework's hybrid regularized objective consistently mitigates overfitting to noise—a dominant failure mode of neural and CS-based approaches at low SNR—thanks to the joint physical and perceptual losses and explicit RGB guidance.

Superresolution and Local Detail Enhancement

Figure 5: Superresolved outputs for selected bands show clear enhancements in detail and color fidelity over the baselines.

At the superresolution stage, the transformer-based USRNet combined with RGB guidance delivers highest reported scores (PSNR 35.01 dB, SSIM 0.9571, SAM 0.1326 rad). Ablation studies confirm that transformer-based cross-modal fusion and downsampling consistency loss are both critical for high-resolution fidelity.

Real-World Validation

To test hardware deployment, the authors implemented a physical SPHI system combining a DLP projector, fiber spectrometer, and RGB camera. At a sampling rate of only 6.25% (1024 patterns, 144 bands), the recovery framework reconstructed a $128 \times 128 \times 144$ cube with substantial spatial and spectral fidelity.

Figure 6: Schematic of the real-world SPHI setup with random patterned illumination, beam splitting, and dual-sensor acquisition paths.

Figure 7: Experimental system photograph detailing optics and detection hardware.

Spectra at selected ROIs closely match physical ground truth as quantified by Pearson correlation ($r=0.909$ after smoothing) and mean SAM ($22.0^\circ$), confirming accurate cross-domain reconstruction even under real-world sensing noise, parallax, and misalignment.

Figure 8: RGB image and corresponding ground-truth vs reconstructed spectra for five target locations show high spectral fidelity (correlation and SAM reported).

Theoretical and Practical Implications

This framework establishes the practical viability of untrained, physics-informed neural architectures for underdetermined inverse problems in hyperspectral imaging, removing the dependence on pretraining and large, annotated datasets. By tightly coupling physical modeling, cross-modal regularization, and differentiable optimization, the approach not only reaches SOTA results but also opens promising avenues for adaptive and generalizable computational imaging under severe data and measurement constraints. Its reliance on analytical initialization and modular losses supports robustness to noise, out-of-distribution scenes, and system variations.

The results suggest implications for a wide range of high-dimensional inverse imaging problems, particularly where annotation or collection of multimodal datasets is impractical. With this paradigm, model generalization is dictated more by priors and physical correspondences than by fixed learned weights, potentially accelerating domain translation, sensor adaptation, and real-time inference via ensuing developments in meta-learning or cross-modal registration.

Limitations and Future Work

The design currently incurs higher run-time due to iterative optimization and sensitivity to geometric misalignments between RGB and spectral sensors. Prospective improvements include integration of meta-learned initializations, further acceleration strategies, and robust cross-modal registration to handle parallax and sensor mismatch errors, facilitating real-time adaptive imaging in unconstrained environments.

Conclusion

This work systematically establishes a high-fidelity, physics-informed SPHI reconstruction framework that leverages untrained neural optimization and RGB structural priors. It demonstrably surpasses contemporary state-of-the-art methods in both spatial and spectral metrics at extreme undersampling, validated on simulation data and in a full hardware pipeline. Its general, data-independent methodology is especially important for advancing adaptive computational imaging in resource-limited, real-world scenarios, and will likely inform future developments in neural inverse imaging.