---
title: 'NIRSA-Net: Compact NIR Hyperspectral Imager'
url: https://www.emergentmind.com/topics/nirsa-net
type: topic
---

# NIRSA-Net: Compact NIR Hyperspectral Imager

Searching arXiv for the NIRSA-Net paper and closely related baseline papers to ground the article in current literature.
NIRSA-Net is the learned reconstruction network in an integrated snapshot near-infrared hyperspectral imaging framework that combines a designed diffractive optical element (DOE) with a single near-infrared camera to recover a 31-band hyperspectral cube over 700–1000 nm in 10 nm steps from one encoded frame [2508.14585]. In this framework, the DOE encodes the volumetric datacube via wavelength-dependent, rotationally symmetric point spread functions (PSFs), and NIRSA-Net decodes the resulting 2D measurement into a full hyperspectral cube. The reported system is positioned as a compact, low-cost, snapshot hyperspectral imager in the near-infrared, with results demonstrating near-infrared spectral imaging at 700–1000 nm with 10 nm resolution while achieving improvement of PSNR 1.47 dB and SSIM 0.006 over the best baseline [2508.14585].

## 1. Integrated optical–computational formulation

NIRSA-Net is not presented as an isolated reconstruction model; it is part of a tightly integrated optical–computational pipeline. The paper’s primary goal is to realize a compact, low-cost, snapshot hyperspectral imager in the near-infrared (700–1000 nm) with 10 nm spectral resolution (31 bands), by tightly integrating a passive diffractive optical element with a learned reconstruction network. A single near-infrared camera captures a 2D “encoded” image, and NIRSA-Net decodes this snapshot into a full hyperspectral cube [2508.14585].

The acquisition pipeline is specified in five steps. First, the scene is illuminated, and each point emits a spherical wave $U_1(x,y,\lambda)$. Second, the wavefront passes through a rotationally symmetric DOE with height map $h(x,y)$, imprinting a wavelength-dependent phase,
$$
U_2 = A \cdot U_1 \cdot \exp\!\left[\frac{2\pi i (n_2-1) h}{\lambda}\right].
$$
Third, propagation to the sensor focal plane yields $U_3$, and the PSF at each wavelength is
$$
P_{\mathrm{psf}}(x,y,\lambda)=|F\{U_3\}|^2.
$$
Fourth, the hyperspectral cube $I(x,y,\lambda)$ is convolved with these PSFs, weighted by the sensor response $R(\lambda)$, producing
$$
\mathrm{encoded}(x,y)=\sum_{\lambda=\lambda_1}^{\lambda_K}\bigl[P_{\mathrm{psf}}(\cdot,\lambda)*I(\cdot,\lambda)\bigr]\cdot R(\lambda)+n.
$$
Fifth, the single encoded frame is fed to NIRSA-Net, which predicts the 31-band cube.

The physical realization is also specified. The DOE sits 50 mm in front of a Basler a2A2560-70umSWIR camera; its 4.096 mm clear aperture and 16-step, 2.219 $\mu$m depth profile are designed via a 512$\times$1 trainable parameter vector. This suggests a deliberate co-design in which invertibility is shaped jointly by optics and learning rather than by post hoc reconstruction alone.

## 2. Optical encoding and DOE design

The DOE is described as passive and rotationally symmetric, and its central function is to create wavelength-dependent PSFs that encode spectral variation into a single 2D measurement. The paper contrasts this DOE-based encoding with generic masks and states that rotational symmetry yields smooth, wavelength-distinct PSFs, improving invertibility [2508.14585].

Within the framework, the DOE acts on the incoming wavefront before sensor acquisition, so the spectral inverse problem is determined in part by the optical transfer characteristics. The encoded measurement is therefore not a generic compressed representation; it is the result of convolution of each spectral slice with a wavelength-specific PSF, followed by weighting by the sensor response and corruption by additive noise. Because the PSFs are wavelength-distinct, the reconstruction problem becomes one of disentangling spectrally mixed spatial structure from a physically constrained encoding.

The significance of this design is twofold. First, snapshot acquisition avoids sequential band scanning, which is important for compact hyperspectral imaging. Second, the DOE makes the inverse problem structured rather than arbitrary. A plausible implication is that the network benefits from a forward model whose spectral signatures are smooth and physically organized, which may partly explain the reported gains over baselines.

## 3. Network architecture

NIRSA-Net is a U-shaped, transformer-inspired encoder–decoder that weaves together convolutional and self-attention blocks to recover high-fidelity spectra and spatial detail [2508.14585]. Its input is an $H\times W\times 1$ encoded image. The stem applies Conv1$\times$1 to produce $H\times W\times C$ with $C=32$, yielding $O_1$, followed by a Conv3$\times$3 embedding.

The encoder has three stages. Stage E1 applies 2$\times$ NIR-Spectral Attention (NIR-SA) modules and produces $O_2$ with size $H\times W\times C$. A 4$\times$4 strided convolution downsamples to $(H/2)\times(W/2)\times 2C$, after which Stage E2 applies 2$\times$ NIR-SA and produces $O_3$. A second downsampling step leads to $(H/4)\times(W/4)\times 4C$, and Stage E3 applies 2$\times$ NIR-SA at the deepest level.

The decoder mirrors this hierarchy. A 2$\times$2 transposed convolution upsamples to $(H/2)\times(W/2)\times 2C$, followed by 2$\times$ NIR-SA. Fusion1 uses NIR-Fusion$(O_3,\text{upsampled})$ to integrate deep and shallow features via 3$\times$3 convolutions. A second 2$\times$2 transposed convolution upsamples to $H\times W\times C$, followed by 2$\times$ NIR-SA, then Fusion2 with $O_2$, and finally another 2$\times$ NIR-SA. Reconstruction uses a final Conv3$\times$3 to produce $O_6$, skip-adds $O_1$, and applies Conv1$\times$1 plus ReLU to obtain an $H\times W\times 31$ output.

The NIR-SA module is inspired by MST++ and is composed of LayerNorm, Multi-head Self-Attention with residual connection, a Conv1$\times$1 plus GELU and Conv3$\times$3 residual branch for expanded attention features, and then LayerNorm followed by a Feed-Forward Network consisting of two 1$\times$1 convolutions and GELU with residual connection. The NIR-Fusion module is a three-stage 3$\times$3-conv fusion: concat(shallow, deep) followed by Conv3$\times$3 and addition with Conv3$\times$3(deep), then a second deep branch processed by Conv3$\times$3 and concatenated with the intermediate representation, and finally a Conv3$\times$3 to produce fused features. The simplified block diagram is given as:
```
Input → Conv1×1 → Conv3×3 → {↓ E1 → ↓ E2 → ↓ E3} → {↑ D2 → fuse1 → ↑ D1 → fuse2} → Recon
```

## 4. Training objective and experimental protocol

The reconstruction objective is purely an $L_1$ loss between the predicted cube $\hat{I}$ and the ground truth $I$:
$$
L_{\mathrm{rec}}=\frac{1}{NHWK}\sum_{n,x,y,\lambda}\left|\hat{I}_n(x,y,\lambda)-I_n(x,y,\lambda)\right|.
$$
The total objective is $L_{\mathrm{total}}=L_{\mathrm{rec}}$, with no explicit regularizer [2508.14585].

Training is carried out for 150 epochs with initial learning rate $5\times 10^{-4}$, decayed by 0.9 every 30 epochs. The optimizer is not explicitly stated. The hyperspectral ground truth is captured by a commercial Specim IQ camera covering 700–1000 nm in 10 nm steps, yielding 31 bands. Scenes include fruits (tomatoes) and colored boxes for spectral validation.

The encoded inputs are generated by simulation. Each ground-truth cube is convolved with wavelength-dependent PSFs from the DOE design, multiplied by camera response $R(\lambda)$, and additive noise $n$ is injected to emulate sensor noise. No additional augmentation is detailed, and input and output are registered to $H\times W$. This protocol places the learning problem in a supervised setting where the forward model is explicitly tied to DOE optics.

## 5. Quantitative performance and comparative analysis

Performance is reported using PSNR and SSIM, with
$$
\mathrm{PSNR}=10\cdot \log_{10}\!\left(\frac{\mathrm{MAX}^2}{\mathrm{MSE}}\right), \qquad
\mathrm{MSE}=\frac{1}{NHWK}\sum (\hat{I}-I)^2,
$$
and SSIM defined as standard structural similarity [2508.14585].

On held-out test scenes, the reported quantitative comparison is as follows:

| Method | PSNR | SSIM |
|---|---:|---:|
| Res-UNet | 32.42 dB | 0.968 |
| HD-Net | 31.99 dB | 0.970 |
| MST++ | 34.23 dB | 0.971 |
| NIRSA-Net + DOE | 35.70 dB | 0.977 |

The reported gain over the best baseline, MST++, is +1.47 dB PSNR and +0.006 SSIM. The paper further states that DOE+network outperforms both purely data-driven and purely coded-aperture methods in spectral accuracy, described as curve fitting for tomato vs. box, and in spatial smoothness, described as no luminance discontinuities in reconstructed bands.

The comparative interpretation offered by the paper has three parts. DOE-based encoding versus generic masks is said to improve invertibility because rotational symmetry yields smooth, wavelength-distinct PSFs. Transformer-inspired NIR-SA modules are said to capture long-range spectral–spatial correlations more effectively than pure CNNs. NIR-Fusion is said to bridge deep (coarse) and shallow (fine) features, sharpening spatial detail. Taken together, these points frame the reported performance not as an effect of network depth alone, but as the result of joint optical and architectural design.

## 6. Scope, limitations, and prospective extensions

The framework is presented with explicit limits. The current design assumes a fixed DOE and static scenes; dynamic adaptation through a reconfigurable DOE is identified as a possible extension. The system also has no explicit noise or motion compensation module, and future work might incorporate physics-informed priors or robust loss terms to handle low SNR or motion blur [2508.14585].

Additional open questions are identified at both the architectural and optical levels. Ablation on the number of NIR-SA stages, channel widths, or depth of DOE quantization remains to be explored. Extending the framework beyond 31 bands or to other spectral ranges, including mid-IR, may require customized DOE fabrication and sensor calibration.

These limitations also delimit common misconceptions. NIRSA-Net is not described as a universally adaptive hyperspectral reconstructor; it is tuned to a fixed optical encoder, a fixed spectral range of 700–1000 nm, and a 31-band setting. Likewise, the reported gains do not establish robustness to motion blur, severe low-SNR regimes, or arbitrary spectral ranges. A plausible implication is that the principal contribution lies in co-design methodology: the paper demonstrates that tightly co-designing a simple, rotationally symmetric DOE with a specialized spectral–spatial attention network enables compact, snapshot near-infrared hyperspectral imaging with superior fidelity under the stated acquisition model.

Source: https://www.emergentmind.com/topics/nirsa-net