NIRSA-Net: Compact NIR Hyperspectral Imager

• NIRSA-Net is a learned reconstruction network that decodes a DOE-encoded 2D image into a 31-band near-infrared hyperspectral cube.
• It employs a U-shaped, transformer-inspired architecture with NIR-SA modules to capture long-range spectral–spatial correlations.
• The integrated optical–computational design yields improved PSNR (+1.47 dB) and SSIM (+0.006) over prior hyperspectral imaging methods.

Searching arXiv for the NIRSA-Net paper and closely related baseline papers to ground the article in current literature. NIRSA-Net is the learned reconstruction network in an integrated snapshot near-infrared hyperspectral imaging framework that combines a designed diffractive optical element (DOE) with a single near-infrared camera to recover a 31-band hyperspectral cube over 700–1000 nm in 10 nm steps from one encoded frame (Ma et al., 20 Aug 2025). In this framework, the DOE encodes the volumetric datacube via wavelength-dependent, rotationally symmetric point spread functions (PSFs), and NIRSA-Net decodes the resulting 2D measurement into a full hyperspectral cube. The reported system is positioned as a compact, low-cost, snapshot hyperspectral imager in the near-infrared, with results demonstrating near-infrared spectral imaging at 700–1000 nm with 10 nm resolution while achieving improvement of PSNR 1.47 dB and SSIM 0.006 over the best baseline (Ma et al., 20 Aug 2025).

1. Integrated optical–computational formulation

NIRSA-Net is not presented as an isolated reconstruction model; it is part of a tightly integrated optical–computational pipeline. The paper’s primary goal is to realize a compact, low-cost, snapshot hyperspectral imager in the near-infrared (700–1000 nm) with 10 nm spectral resolution (31 bands), by tightly integrating a passive diffractive optical element with a learned reconstruction network. A single near-infrared camera captures a 2D “encoded” image, and NIRSA-Net decodes this snapshot into a full hyperspectral cube (Ma et al., 20 Aug 2025).

The acquisition pipeline is specified in five steps. First, the scene is illuminated, and each point emits a spherical wave $U_1(x,y,\lambda)$. Second, the wavefront passes through a rotationally symmetric DOE with height map $h(x,y)$, imprinting a wavelength-dependent phase,

$$U_2 = A \cdot U_1 \cdot \exp\!\left[\frac{2\pi i (n_2-1) h}{\lambda}\right].$$

Third, propagation to the sensor focal plane yields $U_3$, and the PSF at each wavelength is

$P_{\mathrm{psf}}(x,y,\lambda)=|F\{U_3\}|^2.$

Fourth, the hyperspectral cube $I(x,y,\lambda)$ is convolved with these PSFs, weighted by the sensor response $R(\lambda)$, producing

$$\mathrm{encoded}(x,y)=\sum_{\lambda=\lambda_1}^{\lambda_K}\bigl[P_{\mathrm{psf}}(\cdot,\lambda)*I(\cdot,\lambda)\bigr]\cdot R(\lambda)+n.$$

Fifth, the single encoded frame is fed to NIRSA-Net, which predicts the 31-band cube.

The physical realization is also specified. The DOE sits 50 mm in front of a Basler a2A2560-70umSWIR camera; its 4.096 mm clear aperture and 16-step, 2.219 $\mu$m depth profile are designed via a 512$\times$1 trainable parameter vector. This suggests a deliberate co-design in which invertibility is shaped jointly by optics and learning rather than by post hoc reconstruction alone.

2. Optical encoding and DOE design

The DOE is described as passive and rotationally symmetric, and its central function is to create wavelength-dependent PSFs that encode spectral variation into a single 2D measurement. The paper contrasts this DOE-based encoding with generic masks and states that rotational symmetry yields smooth, wavelength-distinct PSFs, improving invertibility (Ma et al., 20 Aug 2025).

Within the framework, the DOE acts on the incoming wavefront before sensor acquisition, so the spectral inverse problem is determined in part by the optical transfer characteristics. The encoded measurement is therefore not a generic compressed representation; it is the result of convolution of each spectral slice with a wavelength-specific PSF, followed by weighting by the sensor response and corruption by additive noise. Because the PSFs are wavelength-distinct, the reconstruction problem becomes one of disentangling spectrally mixed spatial structure from a physically constrained encoding.

The significance of this design is twofold. First, snapshot acquisition avoids sequential band scanning, which is important for compact hyperspectral imaging. Second, the DOE makes the inverse problem structured rather than arbitrary. A plausible implication is that the network benefits from a forward model whose spectral signatures are smooth and physically organized, which may partly explain the reported gains over baselines.

3. Network architecture

NIRSA-Net is a U-shaped, transformer-inspired encoder–decoder that weaves together convolutional and self-attention blocks to recover high-fidelity spectra and spatial detail (Ma et al., 20 Aug 2025). Its input is an $h(x,y)$0 encoded image. The stem applies Conv1$h(x,y)$11 to produce $h(x,y)$2 with $h(x,y)$3, yielding $h(x,y)$4, followed by a Conv3$h(x,y)$53 embedding.

The encoder has three stages. Stage E1 applies 2$h(x,y)$6 NIR-Spectral Attention (NIR-SA) modules and produces $h(x,y)$7 with size $h(x,y)$8. A 4$h(x,y)$94 strided convolution downsamples to $$U_2 = A \cdot U_1 \cdot \exp\!\left[\frac{2\pi i (n_2-1) h}{\lambda}\right].$$0, after which Stage E2 applies 2$$U_2 = A \cdot U_1 \cdot \exp\!\left[\frac{2\pi i (n_2-1) h}{\lambda}\right].$$1 NIR-SA and produces $$U_2 = A \cdot U_1 \cdot \exp\!\left[\frac{2\pi i (n_2-1) h}{\lambda}\right].$$2. A second downsampling step leads to $$U_2 = A \cdot U_1 \cdot \exp\!\left[\frac{2\pi i (n_2-1) h}{\lambda}\right].$$3, and Stage E3 applies 2$$U_2 = A \cdot U_1 \cdot \exp\!\left[\frac{2\pi i (n_2-1) h}{\lambda}\right].$$4 NIR-SA at the deepest level.

The decoder mirrors this hierarchy. A 2$$U_2 = A \cdot U_1 \cdot \exp\!\left[\frac{2\pi i (n_2-1) h}{\lambda}\right].$$52 transposed convolution upsamples to $$U_2 = A \cdot U_1 \cdot \exp\!\left[\frac{2\pi i (n_2-1) h}{\lambda}\right].$$6, followed by 2$$U_2 = A \cdot U_1 \cdot \exp\!\left[\frac{2\pi i (n_2-1) h}{\lambda}\right].$$7 NIR-SA. Fusion1 uses NIR-Fusion$$U_2 = A \cdot U_1 \cdot \exp\!\left[\frac{2\pi i (n_2-1) h}{\lambda}\right].$$8 to integrate deep and shallow features via 3$$U_2 = A \cdot U_1 \cdot \exp\!\left[\frac{2\pi i (n_2-1) h}{\lambda}\right].$$93 convolutions. A second 2$U_3$02 transposed convolution upsamples to $U_3$1, followed by 2$U_3$2 NIR-SA, then Fusion2 with $U_3$3, and finally another 2$U_3$4 NIR-SA. Reconstruction uses a final Conv3$U_3$53 to produce $U_3$6, skip-adds $U_3$7, and applies Conv1$U_3$81 plus ReLU to obtain an $U_3$9 output.

The NIR-SA module is inspired by MST++ and is composed of LayerNorm, Multi-head Self-Attention with residual connection, a Conv1$P_{\mathrm{psf}}(x,y,\lambda)=|F\{U_3\}|^2.$01 plus GELU and Conv3$P_{\mathrm{psf}}(x,y,\lambda)=|F\{U_3\}|^2.$13 residual branch for expanded attention features, and then LayerNorm followed by a Feed-Forward Network consisting of two 1$P_{\mathrm{psf}}(x,y,\lambda)=|F\{U_3\}|^2.$21 convolutions and GELU with residual connection. The NIR-Fusion module is a three-stage 3$P_{\mathrm{psf}}(x,y,\lambda)=|F\{U_3\}|^2.$33-conv fusion: concat(shallow, deep) followed by Conv3$P_{\mathrm{psf}}(x,y,\lambda)=|F\{U_3\}|^2.$43 and addition with Conv3$P_{\mathrm{psf}}(x,y,\lambda)=|F\{U_3\}|^2.$53(deep), then a second deep branch processed by Conv3$P_{\mathrm{psf}}(x,y,\lambda)=|F\{U_3\}|^2.$63 and concatenated with the intermediate representation, and finally a Conv3$P_{\mathrm{psf}}(x,y,\lambda)=|F\{U_3\}|^2.$73 to produce fused features. The simplified block diagram is given as: $I(x,y,\lambda)$8

4. Training objective and experimental protocol

The reconstruction objective is purely an $P_{\mathrm{psf}}(x,y,\lambda)=|F\{U_3\}|^2.$8 loss between the predicted cube $P_{\mathrm{psf}}(x,y,\lambda)=|F\{U_3\}|^2.$9 and the ground truth $I(x,y,\lambda)$0:

$I(x,y,\lambda)$1

The total objective is $I(x,y,\lambda)$2, with no explicit regularizer (Ma et al., 20 Aug 2025).

Training is carried out for 150 epochs with initial learning rate $I(x,y,\lambda)$3, decayed by 0.9 every 30 epochs. The optimizer is not explicitly stated. The hyperspectral ground truth is captured by a commercial Specim IQ camera covering 700–1000 nm in 10 nm steps, yielding 31 bands. Scenes include fruits (tomatoes) and colored boxes for spectral validation.

The encoded inputs are generated by simulation. Each ground-truth cube is convolved with wavelength-dependent PSFs from the DOE design, multiplied by camera response $I(x,y,\lambda)$4, and additive noise $I(x,y,\lambda)$5 is injected to emulate sensor noise. No additional augmentation is detailed, and input and output are registered to $I(x,y,\lambda)$6. This protocol places the learning problem in a supervised setting where the forward model is explicitly tied to DOE optics.

5. Quantitative performance and comparative analysis

Performance is reported using PSNR and SSIM, with

$I(x,y,\lambda)$7

and SSIM defined as standard structural similarity (Ma et al., 20 Aug 2025).

On held-out test scenes, the reported quantitative comparison is as follows:

Method	PSNR	SSIM
Res-UNet	32.42 dB	0.968
HD-Net	31.99 dB	0.970
MST++	34.23 dB	0.971
NIRSA-Net + DOE	35.70 dB	0.977

The reported gain over the best baseline, MST++, is +1.47 dB PSNR and +0.006 SSIM. The paper further states that DOE+network outperforms both purely data-driven and purely coded-aperture methods in spectral accuracy, described as curve fitting for tomato vs. box, and in spatial smoothness, described as no luminance discontinuities in reconstructed bands.

The comparative interpretation offered by the paper has three parts. DOE-based encoding versus generic masks is said to improve invertibility because rotational symmetry yields smooth, wavelength-distinct PSFs. Transformer-inspired NIR-SA modules are said to capture long-range spectral–spatial correlations more effectively than pure CNNs. NIR-Fusion is said to bridge deep (coarse) and shallow (fine) features, sharpening spatial detail. Taken together, these points frame the reported performance not as an effect of network depth alone, but as the result of joint optical and architectural design.

6. Scope, limitations, and prospective extensions

The framework is presented with explicit limits. The current design assumes a fixed DOE and static scenes; dynamic adaptation through a reconfigurable DOE is identified as a possible extension. The system also has no explicit noise or motion compensation module, and future work might incorporate physics-informed priors or robust loss terms to handle low SNR or motion blur (Ma et al., 20 Aug 2025).

Additional open questions are identified at both the architectural and optical levels. Ablation on the number of NIR-SA stages, channel widths, or depth of DOE quantization remains to be explored. Extending the framework beyond 31 bands or to other spectral ranges, including mid-IR, may require customized DOE fabrication and sensor calibration.

These limitations also delimit common misconceptions. NIRSA-Net is not described as a universally adaptive hyperspectral reconstructor; it is tuned to a fixed optical encoder, a fixed spectral range of 700–1000 nm, and a 31-band setting. Likewise, the reported gains do not establish robustness to motion blur, severe low-SNR regimes, or arbitrary spectral ranges. A plausible implication is that the principal contribution lies in co-design methodology: the paper demonstrates that tightly co-designing a simple, rotationally symmetric DOE with a specialized spectral–spatial attention network enables compact, snapshot near-infrared hyperspectral imaging with superior fidelity under the stated acquisition model.