BSNeRF: Broadband Spectral Neural Radiance Field

• BSNeRF is a computational model that reconstructs multispectral light fields by recovering spatial, angular, and wavelength-dependent radiance.
• It integrates broadband spectral decoupling with joint camera parameter optimization to enable snapshot acquisition with high spectral fidelity.
• Experimental results on a 3×3 kaleidoscopic system show improved color consistency and light throughput compared to conventional NeRF methods.

Broadband Spectral Neural Radiance Field (BSNeRF) is a computational model designed to recover and render high-dimensional multispectral light-field data—including spatial, angular, and spectral (wavelength-dependent) components—from snapshot acquisitions. Unlike conventional neural radiance field (NeRF) models, which primarily address view-dependent radiance in RGB space, BSNeRF explicitly incorporates a broadband spectral dimension, enabling representation and reconstruction of complex spectral multiplexed signals crucial to snapshot multispectral light-field imaging (SMLI) and modern plenoptic imaging systems.

1. Scope and Motivation

BSNeRF is motivated by the limitations of current SMLI systems, which seek to reconstruct data parameterized by $(x, y, z, \theta, \phi, \lambda)$ from a single sensor exposure. Existing approaches often compromise spectral fidelity or imaging speed, either by reducing light throughput or prolonging acquisition times due to the computational difficulty of model decoupling. BSNeRF is introduced specifically to address these limitations by enabling robust decoupling and high-fidelity reconstruction of multispectral light-field information. The model’s design is inherently self-supervised—jointly optimizing scene radiance parameters alongside camera intrinsics and extrinsics—and is constructed to preserve color consistency and spectral accuracy across all reconstructed views (Huang et al., 1 Sep 2025).

2. Mathematical Formulation and Spectral Decoupling

A defining feature of BSNeRF is its broadband spectral decoupling mechanism, which reconstructs the spectrum from multiplexed sensor measurements. Each pixel intensity is influenced by both the sensor’s spectral sensitivity and an external broadband filter, with the measurement model given by:

$$I_{d,k}(p) = \int_{\Omega} s(p, \lambda) f_{k}^{sensor}(\lambda) f_{d}^{filter}(\lambda) d\lambda$$

where $I_{d,k}(p)$ denotes the intensity at pixel $p$ for view/filter $d$ and color channel $k$, $s(p, \lambda)$ the spectral intensity, $f_{k}^{sensor}$ the sensor’s sensitivity, $f_{d}^{filter}$ the filter’s transmission, and $\Omega$ the spectral bandwidth (e.g., [430 nm, 670 nm]).

BSNeRF further extends the classical NeRF volume rendering by integrating over both ray depth $t$ and wavelength $\lambda$:

$$\hat{I}_{d,k}(p) = \mathcal{R}(p, \pi_d | \Theta) = \int_{\lambda_n}^{\lambda_f} f(\lambda) \left[ \int_{t_n}^{t_f} T(t) \sigma(r(t)) s(r(t), d, \lambda) dt \right] d\lambda$$

with $$f(\lambda) = f_{k}^{sensor}(\lambda) f_{d}^{filter}(\lambda)$$, $T(t) = \exp(-\int_{t_n}^{t} \sigma(r(m)) dm)$, and $s(r(t), d, \lambda)$ the spectral intensity at location $r(t)$ and view direction $d$. The formulation thus recovers multispectral radiance along rays, modulated by both the imaging system and external spectral filters (Huang et al., 1 Sep 2025).

3. Optimization and Loss Functions

BSNeRF is trained using a composite loss function that enforces both fidelity and color consistency:

• Fidelity Loss:

$$\mathcal{L}_{fidelity} = \sum_{d=1}^{D} \sum_{k=1}^{K} \| \hat{I}_{d,k} - I_{d,k} \|_2^2$$

This term penalizes discrepancies in spectral intensities between reconstruction $\hat{I}_{d,k}$ and ground truth $I_{d,k}$.

• Color Loss:

$$\mathcal{L}_{color} = \frac{1}{D} \sum_{d=1}^{D} \sum_{k=1}^{K} \left[ \| \hat{\mu}_k - \mu_k^{(d)} \|_2^2 + \| \hat{\sigma}_k - \sigma_k^{(d)} \|_2^2 \right]$$

where $\mu_k$, $\sigma_k$ are mean and standard deviation statistics for generated and measured images, respectively. This loss enforces consistent color statistics across all spectral channels and subviews.

Joint optimization is expressed as:

$$\Theta^*, \Pi^* = \arg\min_{\Theta, \Pi} [\alpha\, \mathcal{L}_{fidelity} + \beta\, \mathcal{L}_{color}]$$

where $\Theta$ are radiance field parameters, $\Pi$ camera intrinsics/extrinsics, and empirically $\alpha = \beta = 0.5$.

4. Model Architecture and Camera Parameter Estimation

Built on top of NeRF– architectures, BSNeRF includes explicit optimization for camera parameters. In addition to scene parameters, extrinsics (e.g., rotation matrix $R$ using Rodrigues' formula), intrinsics (including focal lengths), and translation vectors are optimized using separate Adam optimizers. The loss landscape is managed by tailored learning rate scheduling according to parameter category, ensuring stable convergence. The explicit saddlepoint optimization over both scene and camera parameters is critical for single-shot multispectral recovery—where calibration accuracy directly affects spectral decoupling (Huang et al., 1 Sep 2025).

5. Experimental Validation and Light-field Imaging Systems

Validation of BSNeRF is demonstrated on a 3×3 kaleidoscopic SMLI system comprising a light-field lens and commercial broadband filters, paired with a trichromatic camera capturing 27 spectral channels (from 9 filters × 3 sensor channels). Training over 10,000 epochs on an NVIDIA P100 GPU, the model demonstrates:

• Successful decoupling and reconstruction of multispectral light-fields.
• Enhanced consistency among different views and spectral channels as evidenced by qualitative reconstructions and quantitative performance metrics (measured via color and fidelity loss terms).
• Documented impact of color and fidelity loss on reconstruction accuracy and consistency in experiments.

This setup highlights BSNeRF’s capability for snapshot plenoptic imaging, avoiding sacrifices in light throughput often seen in methods requiring multiple exposures (Huang et al., 1 Sep 2025).

6. Applications and Further Implications

BSNeRF enables:

• Plenoptic Imaging: Simultaneous acquisition and reconstruction across spatial, angular, and spectral domains.
• Multispectral and Hyperspectral Reconstruction: High-fidelity recovery for remote sensing, biomedical imaging, and spectral scene analysis.
• Light-throughput Optimization: SMLI systems using BSNeRF avoid the throughput trade-offs of diffuser-based and narrow-band filter strategies.
• Self-supervised Calibration: Joint learning of camera and radiance parameters reduces dependence on external calibration.

The authors indicate a plausible future direction of extending BSNeRF to temporal dimensions ($t$), thus enabling dynamic multispectral scene reconstruction from video (Huang et al., 1 Sep 2025).

7. Relation to Broader BSNeRF and NeRF Literature

Prior works have proposed spectral NeRF variants, including Spec-NeRF (Li et al., 2023), SpectralNeRF (Li et al., 2023), and UnMix-NeRF (Perez et al., 27 Jun 2025), but BSNeRF distinguishes itself through explicit broadband spectral multiplexing, decoupling for snapshot acquisition, and joint camera-scene optimization. Its loss methodology and rendering process align with advances in physically based spectral rendering and plenoptic scene synthesis. The design principles outlined in (Huang et al., 1 Sep 2025) set a precedent for future high-dimensional neural field models addressing complex spectral, spatial, and angular coupling.

Summary Table: Core Components of BSNeRF

Component	Description	Reference
Spectral Decoupling	Integration over both depth and wavelength	(Huang et al., 1 Sep 2025)
Joint Camera Optimization	Intrinsics/extrinsics jointly learned	(Huang et al., 1 Sep 2025)
Composite Loss Functions	Fidelity and color loss for spectrum recovery	(Huang et al., 1 Sep 2025)
Plenoptic Imaging	3×3 SMLI system for multispectral views	(Huang et al., 1 Sep 2025)
Snapshot Acquisition	Single-shot, high-dimensional data	(Huang et al., 1 Sep 2025)

These elements collectively define the BSNeRF framework in the context of snapshot multispectral light-field imaging and the broader problem of high-dimensional scene reconstruction.