Multispectral-NeRF: Expanded Neural Radiance Fields

• Multispectral-NeRF is a framework that extends traditional NeRF by representing scene appearance with multiple spectral channels beyond standard RGB, such as NIR, thermal, and hyperspectral bands.
• It leverages shared geometry and multi-head architectures to fuse diverse sensor modalities, improving reconstruction accuracy and enabling task-specific applications.
• Empirical results show enhanced performance in satellite imaging, agriculture, and material segmentation through precise cross-modal calibration and sensor-aware rendering.

Multispectral-NeRF denotes a family of Neural Radiance Field formulations in which scene appearance is represented beyond three-channel RGB, either as a discrete multi-band vector, as a wavelength-dependent radiance function, or as a cross-spectral multimodal output tied to heterogeneous sensors. Across the literature, the shared principle is that geometry remains coupled to a richer appearance space: a single scene representation is optimized so that it can explain visible, near-infrared, thermal, panchromatic, multispectral, or hyperspectral observations through volumetric rendering or closely related rendering operators. Representative instantiations include cross-device multi-output fields, physically grounded spectral rendering, multisensor agricultural NeRFs, satellite NeRFs that fuse panchromatic and multispectral imagery, material-structured hyperspectral fields, and snapshot spectral light-field models (Poggi et al., 2022).

1. Conceptual scope and formal definitions

The canonical NeRF formulation maps a 3D point and viewing direction to density and color,

$$F_\theta : (\mathbf{x}, \mathbf{d}) \mapsto (\sigma(\mathbf{x}), \mathbf{c}(\mathbf{x}, \mathbf{d})),$$

with $\mathbf{c}\in\mathbb{R}^3$ in the RGB case. Multispectral-NeRF generalizes this color term in several distinct but compatible ways.

A first formulation is the discrete multi-channel extension, in which $\mathbf{c}\in\mathbb{R}^B$ for $B$ bands. This pattern appears in X-NeRF, which concatenates RGB, visible multispectral, and infrared channels under a shared density field; in the UAV-oriented "Multispectral-NeRF", which extends NeRFacto from 3 channels to 6 bands $(B,G,R,RE1,RE2,NIR)$; and in HS-3D-NeRF, which predicts a 204-dimensional hyperspectral vector over 397–1003 nm (Poggi et al., 2022, Zhang et al., 14 Sep 2025, Ku et al., 18 Feb 2026).

A second formulation treats radiance as a function of wavelength. BSNeRF defines

$$F_\Theta : (\mathbf{x}, \mathbf{d}) \mapsto (s, \sigma),$$

where $s=s(\mathbf{x},\mathbf{d},\lambda)$ is spectral intensity, and renders measurements by integrating jointly over depth and wavelength. Spec-NeRF similarly reconstructs a multispectral radiance field and explicitly models the camera spectral sensitivity functions (SSFs), while SpectralNeRF predicts a set of spectrum-map channels and then fuses them into white-light RGB through a learned spectral integrator (Huang et al., 1 Sep 2025, Li et al., 2023, Li et al., 2023).

A third formulation is cross-spectral or multimodal NeRF, where different sensors need not correspond to a single contiguous wavelength axis. AgriNeRF combines RGB, event, and thermal cameras; FusionRF fuses multispectral and panchromatic satellite imagery; MultimodalStudio supports RGB, monochrome, near-infrared, polarization, and multispectral channels in one radiance field (Chopra et al., 2024, Sprintson et al., 2024, Lincetto et al., 25 Mar 2025).

This usage makes the term multispectral broader than strict remote-sensing band stacks. In practice, the literature spans discrete spectral bands, continuous spectral rendering, and heterogeneous sensing modalities with different spectral sensitivities. This suggests that Multispectral-NeRF is best understood as a generalization of radiance-field appearance modeling rather than a single architecture class.

2. Rendering models and spectral observation operators

Most multispectral methods preserve NeRF’s volumetric rendering logic. For a ray $\mathbf{r}(t)=\mathbf{o}+t\mathbf{d}$, rendering takes the form

$$\hat{C}(\mathbf{r}) = \sum_i T_i \alpha_i \mathbf{c}_i,$$

with $\alpha_i = 1-\exp(-\sigma_i \Delta t_i)$ and $\mathbf{c}\in\mathbb{R}^3$0 the accumulated transmittance. In multi-band models, $\mathbf{c}\in\mathbb{R}^3$1 is simply a vector of spectral channels, and the same density field governs all bands. HS-3D-NeRF adopts exactly this pattern with $\mathbf{c}\in\mathbb{R}^3$2, while X-NeRF uses a shared density and a concatenated 14-channel radiance vector for RGB, visible multispectral, and IR outputs (Ku et al., 18 Feb 2026, Poggi et al., 2022).

A more physically explicit branch inserts sensor response functions into the rendering equation. BSNeRF renders each sensor channel $\mathbf{c}\in\mathbb{R}^3$3 in subview $\mathbf{c}\in\mathbb{R}^3$4 as

$\mathbf{c}\in\mathbb{R}^3$5

where $\mathbf{c}\in\mathbb{R}^3$6 is the known combined bandpass of sensor channel and filter. The model therefore treats each pixel as a broadband spectral mixture rather than a direct narrow-band measurement (Huang et al., 1 Sep 2025).

Spec-NeRF uses a closely related imaging model. The rendered spectral vector $\mathbf{c}\in\mathbb{R}^3$7 is converted to an observed measurement through the filter transmittance $\mathbf{c}\in\mathbb{R}^3$8 and learned SSFs $\mathbf{c}\in\mathbb{R}^3$9,

$\mathbf{c}\in\mathbb{R}^B$0

This makes the forward operator part of the optimization, rather than a preprocessing step (Li et al., 2023).

SpectralNeRF also starts from spectral rendering, but its visible-spectrum channels are ultimately fused into RGB through Spectrum Attention UNet (SAUNet). The method retains CIE-inspired spectral integration as a guiding principle, while replacing fixed analytical quadrature by a learned fusion module (Li et al., 2023).

Not all multispectral systems are fully volumetric. psPRF replaces 3D ray sampling with a planar MPI-style representation over discrete heights $\mathbf{c}\in\mathbb{R}^B$1, predicting RGB, PAN, and density per plane and integrating along the stack. The resulting model is a cross-spectral planar NeRF rather than a full volumetric field, but it still couples geometry and multi-band appearance in a NeRF-like rendering framework (Zhang et al., 2024).

3. Recurrent architectural patterns

The literature has converged on several recurring design patterns for Multispectral-NeRF systems.

System	Spectral scope	Distinctive mechanism
X-NeRF (Poggi et al., 2022)	RGB + 10-band visible MS + IR	Shared-density, multi-channel output with NXDC
Spec-NeRF (Li et al., 2023)	Multi-band spectra from filtered images	Joint radiance-field and SSF estimation
AgriNeRF (Chopra et al., 2024)	RGB + event + thermal	Shared TensorRF with RGB/thermal heads and cross-spectral regularization
FusionRF (Sprintson et al., 2024)	8-band MS + PAN satellite imagery	Cross-resolution kernel and multimodal appearance embeddings
UnMix-NeRF (Perez et al., 27 Jun 2025)	Hyperspectral radiance	Endmembers, abundances, diffuse/specular decomposition
Multispectral-NeRF (Zhang et al., 14 Sep 2025)	6-band UAV imagery	NeRFacto extension with widened MLPs and bit-depth-aware handling

One major pattern is the shared-geometry, multi-channel radiance field. X-NeRF, MultimodalStudio, FusionRF, and HS-3D-NeRF all use one density or SDF representation and let the appearance head output the channels required by each modality. This hard parameter sharing is intended to transfer structural information across spectra and sensors (Poggi et al., 2022, Lincetto et al., 25 Mar 2025, Sprintson et al., 2024, Ku et al., 18 Feb 2026).

A second pattern is the multi-head shared field. AgriNeRF uses TensorRF-style coarse and fine tensorial radiance volumes, then routes sampled features into modality-specific MLP heads. RGB and thermal are reconstructed explicitly, while event frames serve as a conditioning and regularization modality through

$\mathbf{c}\in\mathbb{R}^B$2

This design turns the latent field into a cross-spectral representation rather than a simple channel stack (Chopra et al., 2024).

A third pattern is physically structured spectral parameterization. UnMix-NeRF is explicit that hyperspectral modeling should not be reduced to independent channel regression. It embeds an ELMM-style spectral unmixing model inside the radiance field, with a learned global endmember matrix $\mathbf{c}\in\mathbb{R}^B$3, per-point abundances, scaling factors, and a separate specular branch. The diffuse component is

$\mathbf{c}\in\mathbb{R}^B$4

and the final spectral radiance is a dichromatic sum of diffuse and specular terms (Perez et al., 27 Jun 2025).

A fourth pattern is sensor-aware cross-resolution fusion. FusionRF learns directly from raw multispectral and panchromatic WorldView-3 imagery using a cross-resolution sparse kernel $\mathbf{c}\in\mathbb{R}^B$5 to model the spatial-resolution gap, while psPRF uses spectral-to-spatial convolution (SSConv) to convert LR-RGB spectral content into HR-compatible spatial detail. In both cases, the NeRF itself becomes the fusion engine, eliminating an external pansharpening stage (Sprintson et al., 2024, Zhang et al., 2024).

4. Calibration, acquisition, and sensor geometry

Calibration is not incidental in Multispectral-NeRF; it is often the central technical difficulty.

Poggi et al.’s X-NeRF addresses multi-device alignment by learning constant rigid transforms from an RGB reference camera to multispectral and IR cameras, and by replacing standard NDC with Normalized Cross-Device Coordinates (NXDC). NXDC forces all devices into a common normalized 3D coordinate system despite different focal lengths, fields of view, and resolutions, so that a single field can produce aligned cross-spectral outputs (Poggi et al., 2022).

AgriNeRF uses a rigidly co-mounted sensor tower with an RGB camera, a Prophesee Gen 4 event camera, and a FLIR Boson 320 thermal camera. Camera poses are estimated with COLMAP, and under low light COLMAP is run on enhanced thermal images rather than RGB. A multi-modal image alignment module then projects RGB, thermal, and event frames into a unified coordinate frame (Chopra et al., 2024).

Satellite-oriented methods substitute RPC geometry for pinhole cameras. FusionRF builds on satellite NeRFs with RPC-based ray sampling, and psPRF uses differentiable forward and inverse RPC projection in both rendering and reprojection losses. This is essential because WorldView-3 imagery violates the assumptions of standard NeRF camera models through large baselines, modality-dependent resolution, and sensor-specific bandpasses (Sprintson et al., 2024, Zhang et al., 2024).

Stationary-camera hyperspectral acquisition introduces a different geometric issue. HS-3D-NeRF captures 60 hyperspectral frames with a fixed SPECIM IQ push-broom camera while the object rotates in a Teflon chamber. Object motion is converted into equivalent camera motion through simulated pose transformations, allowing standard NeRF training on stationary-camera data. Pose estimation relies on ArUco markers and COLMAP, with reprojection error below 1 px (Ku et al., 18 Feb 2026).

Raw sensor handling is itself an acquisition problem. MultimodalStudio stores raw mosaicked, lens-distorted images for RGB, monochrome, NIR, polarization, and multispectral cameras, and its framework directly ray-samples through calibrated distortion models. For multispectral imaging, this avoids committing early to a fixed demosaicking pipeline, which the authors note is non-trivial and can introduce artifacts (Lincetto et al., 25 Mar 2025).

5. Empirical behavior and downstream uses

A consistent empirical theme is that richer spectral supervision improves not only appearance reconstruction, but also geometry, robustness, or downstream semantics.

In agriculture, AgriNeRF reports that RGB scene reconstruction improves in PSNR and SSIM by +2.06 dB and +8.3% respectively, and that cross-spectral scene reconstruction enhances downstream fruit detection by +43.0% in mAP50 and +61.1% increase in mAP50-95. The system is intended for challenging illumination, including low-light, extreme bright light, and varying lighting, and uses a learned cross-spectral representation for automated fruit detection (Chopra et al., 2024).

In satellite reconstruction, FusionRF reports an average of 17% improvement in depth reconstruction accuracy relative to a pansharpening-based baseline, while rendering sharp training and novel views from raw multispectral and panchromatic imagery. The practical significance is that image fusion occurs during radiance-field optimization rather than as a preprocessing step that may introduce biases and hallucinations due to domain gaps (Sprintson et al., 2024).

Material-aware hyperspectral modeling yields a different class of benefit. UnMix-NeRF reports unsupervised material segmentation with mIoU $\mathbf{c}\in\mathbb{R}^B$6 and F1 $\mathbf{c}\in\mathbb{R}^B$7 on extended NeSpoF, and in hyperspectral novel view synthesis it improves over HyperGS on Caladium from PSNR $\mathbf{c}\in\mathbb{R}^B$8 to $\mathbf{c}\in\mathbb{R}^B$9, SSIM $B$0 to $B$1, SAM $B$2 to $B$3, and RMSE $B$4 to $B$5. These numbers indicate that explicit material structure can improve both reconstruction fidelity and semantic interpretability (Perez et al., 27 Jun 2025).

In multimodal object-centric rendering, MultimodalStudio shows that joint RGB+MS training raises MS PSNR from $B$6 dB to $B$7 dB, and in five-modality training the MS channel reaches $B$8 dB, a $B$9 dB gain over MS-only training. The authors also report that even with very few MS views, the combined RGB+MS model yields MS renderings better than MS-only trained on the full MS set, which suggests that shared geometry and appearance priors can compensate for severe modality imbalance (Lincetto et al., 25 Mar 2025).

For controlled hyperspectral phenotyping, HS-3D-NeRF reports a maize F-score of $(B,G,R,RE1,RE2,NIR)$0 at a 2 mm threshold and an ICP RMS error of $(B,G,R,RE1,RE2,NIR)$1 m. With loss weights $(B,G,R,RE1,RE2,NIR)$2, it achieves on maize SAM $(B,G,R,RE1,RE2,NIR)$3, RMSE $(B,G,R,RE1,RE2,NIR)$4, SSIM $(B,G,R,RE1,RE2,NIR)$5, and PSNR $(B,G,R,RE1,RE2,NIR)$6 dB across 204 bands (Ku et al., 18 Feb 2026).

The UAV-oriented "Multispectral-NeRF" reports PSNR $(B,G,R,RE1,RE2,NIR)$7 dB, SSIM $(B,G,R,RE1,RE2,NIR)$8, and LPIPS $(B,G,R,RE1,RE2,NIR)$9, outperforming NeRF, Instant-NGP, Mega-NeRF, and NeRFacto on the reported real large-scale scene. The ablation in the same paper shows that a naïve six-channel extension is insufficient: without parameter optimization, performance drops to PSNR $$F_\Theta : (\mathbf{x}, \mathbf{d}) \mapsto (s, \sigma),$$0, SSIM $$F_\Theta : (\mathbf{x}, \mathbf{d}) \mapsto (s, \sigma),$$1, and LPIPS $$F_\Theta : (\mathbf{x}, \mathbf{d}) \mapsto (s, \sigma),$$2 (Zhang et al., 14 Sep 2025).

These results collectively indicate that Multispectral-NeRF is not confined to photorealistic RGB view synthesis. The same underlying idea supports fruit detection, material segmentation, pansharpening-free satellite reconstruction, hyperspectral phenotyping, and multisensor transfer.

6. Misconceptions, limitations, and open directions

A recurrent misconception is that Multispectral-NeRF is merely RGB-NeRF with extra output channels. Several systems explicitly contradict that view. Spec-NeRF introduces a spectral imaging forward model and jointly learns camera SSFs; SpectralNeRF inserts learned spectral integration grounded in color science; UnMix-NeRF constrains radiance through endmembers and abundances rather than unconstrained per-band regression (Li et al., 2023, Li et al., 2023, Perez et al., 27 Jun 2025).

A second misconception is that multispectral fusion can be safely delegated to preprocessing. The satellite literature is explicit that pansharpening may introduce biases and hallucinations due to domain gaps, which is why FusionRF and psPRF internalize fusion inside the radiance-field optimization (Sprintson et al., 2024, Zhang et al., 2024).

The main technical limitations are also consistent across papers. Precise cross-modal calibration and synchronization remain difficult; AgriNeRF identifies sensor alignment and calibration as critical, and cites X-NeRF as struggling with cross-modal alignment. Many systems are still scene-specific, static-scene models. Spectral range is often fixed by the sensor and architecture, and scaling from six or eight bands to hundreds of channels raises memory and optimization costs. Controlled illumination remains important in several settings, particularly Spec-NeRF, SpectralNeRF, and HS-3D-NeRF (Chopra et al., 2024, Poggi et al., 2022, Li et al., 2023, Ku et al., 18 Feb 2026).

Some limitations are domain-specific. AgriNeRF notes that complete darkness with no heat sources remains difficult because RGB and event cameras fail completely and thermal can resolve only coarse hot objects. MultimodalStudio reports lower quality on reflective and transparent materials. The UAV Multispectral-NeRF emphasizes substantial CPU and GPU memory requirements for high-resolution six-band imagery. BSNeRF highlights the ill-posedness of broadband spectral decoupling from multiplexed snapshot measurements (Chopra et al., 2024, Lincetto et al., 25 Mar 2025, Zhang et al., 14 Sep 2025, Huang et al., 1 Sep 2025).

Open directions are already visible in the literature. BSNeRF proposes extending from 6D multispectral light fields to full 7D plenoptic imaging $$F_\Theta : (\mathbf{x}, \mathbf{d}) \mapsto (s, \sigma),$$3. UnMix-NeRF points toward more advanced reflectance models, outdoor illumination handling, temporal scenes, and spectral priors from libraries. FusionRF suggests extending raw-fusion radiance fields to hyperspectral data, LiDAR, SAR, and time series. The UAV Multispectral-NeRF explicitly raises hyperspectral UAV reconstruction and dynamic multispectral scenes as future targets (Huang et al., 1 Sep 2025, Perez et al., 27 Jun 2025, Sprintson et al., 2024, Zhang et al., 14 Sep 2025).

Taken together, the field suggests a broad research program: shared geometry, sensor-aware forward models, and structured spectral priors are becoming the defining ingredients of Multispectral-NeRF. The central unresolved question is not whether NeRF can emit more than three channels, but how spectral information should be parameterized, calibrated, and regularized so that richer radiance models improve both reconstruction and task-relevant interpretation.