NIRSplat: Multimodal 3D Agri Reconstruction

• NIRSplat is a multimodal 3D reconstruction framework that integrates aligned RGB, near-infrared, and textual metadata to address challenges like uneven illumination, occlusions, and sparse viewpoints in agricultural imaging.
• It employs geometry-aware positional encoding and deformable cross-attention to fuse spectral cues, semantic context, and spatial information, ensuring robust plant structure recovery.
• Empirical evaluations on the NIRPlant dataset demonstrate significant improvements in SSIM, PSNR, and LPIPS over RGB-only methods, especially under difficult lighting and occlusion scenarios.

NIRSplat is a multimodal extension of 3D Gaussian Splatting (3DGS) for agricultural 3D reconstruction that fuses aligned RGB imagery, single-channel near-infrared (NIR) imagery, and textual metadata. It was introduced together with the NIRPlant dataset to address plant-scene failure modes that are prominent under uneven illumination, frequent occlusions, limited field of view, sparse viewpoints, and the modality restrictions of RGB-only splatting systems. Its central claim is that plant reconstruction benefits from combining spectral cues that are invisible in RGB with botanic and environmental context, then grounding that fusion with 3D point-based positional encoding and deformable cross-attention before Gaussian parameter prediction and splat rendering (Chang et al., 20 Aug 2025).

1. Concept and problem setting

NIRSplat is defined around a specific reconstruction regime rather than a generic NIR rendering problem. The target domain is agricultural 3D reconstruction, where direct sunlight, low-light conditions, sunset lighting, artificial lighting, dense foliage, and sparse viewpoints jointly degrade standard RGB-based 3DGS pipelines. In this setting, RGB observations alone are described as insufficient because visible appearance is highly sensitive to lighting and does not directly encode plant-specific physiological information. NIR imagery is introduced as a complementary modality because it carries cues related to chlorophyll absorption, water content, surface reflectance, and plant health and vigor. Textual metadata is then used as a higher-level semantic prior describing indices, structure, phenology, lighting, occlusion, and weather.

The associated NIRPlant dataset contains RGB, depth, NIR, LiDAR, and text or metadata, with 34 scenes and 360 views per scene. The acquisition platform uses a ZED 2i camera, a Nikon D3400 HD RGB camera, an Alvium 1800 U-501 NIR sensor, and a Neuvition Titan S2-70C LiDAR sensor. Captures span indoor and outdoor conditions under artificial light, strong sunlight, low light, and normal daylight, and the paper also mentions sunset and noon as collection times. Ground-truth models were constructed using landmark Structure-from-Motion, and backgrounds were removed to prioritize object reconstruction quality. Although the dataset is multimodal, the architecture itself is consistently described as tri-modal: RGB, NIR, and text; depth and LiDAR are available in the dataset, but their explicit use inside the model is not specified (Chang et al., 20 Aug 2025).

A recurring misconception is that NIRSplat is simply a near-infrared version of standard 3DGS. The method is more specific than that: it is a semantics-aware, agriculture-oriented multimodal splatting framework whose NIR branch is coupled to metadata and to geometry-aware positional anchoring. Another common misconception is that it uses all dataset modalities directly during reconstruction. The paper does not specify direct architectural use of depth or LiDAR; instead, the geometric prior actually used by the model comes from MASt3R-predicted dense 3D point maps.

2. Inputs, metadata, and geometric priors

The model input is formalized as

$$\mathcal{I}_{rgb}^{N} \in \mathbb{R}^{N \times H \times W \times 3}, \quad \mathcal{I}_{nir}^{N} \in \mathbb{R}^{N \times H \times W \times 1}, \quad \mathcal{T} \in \mathbb{R}^{L \times C}.$$

These correspond to aligned RGB frames, aligned single-channel NIR frames, and contextual text prompts. The paper distinguishes carefully between raw numeric side information and the textual representation actually consumed by the network. Metadata is not fed as a simple tabular vector. Instead, it is rewritten as botanic-aware prompts that incorporate image descriptions, NIR-derived vegetation indices, structural traits such as leaf shape and stem thickness, phenological stage such as sprouting or flowering, and environmental variables including temperature, dew, humidity, precipitation, precipitation probability, cloud cover, solar radiation, UV index, wind gust, wind speed, and visibility.

The metadata references NDVI, NDWI, and chlorophyll index, but the paper does not provide explicit mathematical formulas for these indices in the material provided. Consequently, they are best understood here as semantically meaningful prompt components rather than as defined analytic operators inside the splatting objective. This suggests that the method treats plant physiology primarily as contextual supervision for fusion rather than as a spectrally constrained physical model.

The geometric prior is supplied by MASt3R rather than by a depth-estimation network or by LiDAR. MASt3R predicts a dense 3D point map

$P = \{\mathbf{p}_i\}_{i=1}^{N},$

and NIRSplat initializes Gaussian centers as

$\mu_i = \mathbf{p}_i.$

A coarse camera extrinsic matrix is also estimated:

$T = [R \mid \mathbf{t}] \in SE(3).$

Given intrinsics $K$, each point is projected into the image plane as

$$\tilde{\mathbf{u}}_i = K \cdot (R \cdot \mathbf{p}_i + \mathbf{t}) \in \mathbb{R}^{3},$$

$$\mathbf{u}_i = \left[ \frac{\tilde{u}_i^x}{\tilde{u}_i^z}, \frac{\tilde{u}_i^y}{\tilde{u}_i^z} \right] \in \mathbb{R}^{2}.$$

These projected anchors are then encoded via

$$PE_i = \Phi\!\left(\left[\sin(\lambda^\top \mathbf{u}_i)\oplus \cos(\lambda^\top \mathbf{u}_i)\right]\right),$$

where $\lambda$ is a learnable frequency scale, $\oplus$ denotes concatenation, and $P = \{\mathbf{p}_i\}_{i=1}^{N},$0 is an MLP. The role of this 3D point-based positional encoding is to create a shared geometry-aware reference frame across RGB, NIR, and text-attended features, thereby reducing spectral discrepancy, semantic mismatch, and the absence of spatial grounding for metadata (Chang et al., 20 Aug 2025).

3. Multimodal fusion and Gaussian prediction

The architecture first extracts modality-specific tokens $P = \{\mathbf{p}_i\}_{i=1}^{N},$1, $P = \{\mathbf{p}_i\}_{i=1}^{N},$2, and $P = \{\mathbf{p}_i\}_{i=1}^{N},$3, then performs staged deformable cross-attention. The first fusion stage couples RGB and NIR:

$P = \{\mathbf{p}_i\}_{i=1}^{N},$4

This attention is stacked for $P = \{\mathbf{p}_i\}_{i=1}^{N},$5 layers to obtain the fused RGB-NIR representation $P = \{\mathbf{p}_i\}_{i=1}^{N},$6. The paper’s interpretation is that RGB should selectively attend to useful NIR signals, especially in overexposed, low-contrast, occluded, or texture-poor regions where visible imagery is unreliable.

The second stage fuses the resulting visual representation with text embeddings produced by a pre-trained vision-LLM:

$P = \{\mathbf{p}_i\}_{i=1}^{N},$7

followed by

$P = \{\mathbf{p}_i\}_{i=1}^{N},$8

This yields a representation that combines spectral, semantic, and geometric information. The paper explicitly argues that naive fusion is inadequate because RGB and NIR differ in spectral statistics and value distribution, while image and text embeddings are semantically and spatially misaligned. The cross-attention mechanism is therefore not auxiliary; it is presented as the main alignment mechanism that makes multimodal splatting effective.

After fusion, a lightweight decoder predicts Gaussian parameters:

$P = \{\mathbf{p}_i\}_{i=1}^{N},$9

where $\mu_i = \mathbf{p}_i.$0 is the 3D Gaussian mean, $\mu_i = \mathbf{p}_i.$1 is opacity, $\mu_i = \mathbf{p}_i.$2 is anisotropic covariance or a low-rank approximation, and $\mu_i = \mathbf{p}_i.$3 is RGB appearance feature. These Gaussians are then rendered by a 3DGS renderer with Adaptive Density Control. In contrast to classical 3DGS, where Gaussian attributes are directly optimized from RGB photometric supervision, NIRSplat uses multimodal fusion to predict and semantically ground the Gaussian field before rendering (Chang et al., 20 Aug 2025).

4. Rendering objective and optimization

The supplementary material recalls the standard 3DGS primitive in the form

$\mu_i = \mathbf{p}_i.$4

with notation described in the paper as somewhat inconsistent but intended to represent a Gaussian centered at $\mu_i = \mathbf{p}_i.$5 with covariance $\mu_i = \mathbf{p}_i.$6 and opacity $\mu_i = \mathbf{p}_i.$7. Rendering is optimized jointly over the Gaussian field $\mu_i = \mathbf{p}_i.$8 and pose $\mu_i = \mathbf{p}_i.$9 using a photometric objective:

$T = [R \mid \mathbf{t}] \in SE(3).$0

Here $T = [R \mid \mathbf{t}] \in SE(3).$1 indexes views and $T = [R \mid \mathbf{t}] \in SE(3).$2 indexes pixels, with $T = [R \mid \mathbf{t}] \in SE(3).$3 the observed image pixel and $T = [R \mid \mathbf{t}] \in SE(3).$4 the rasterized pixel.

The paper also specifies two auxiliary geometric objectives. The first is a point-map regression term against pseudo ground truth from MASt3R:

$T = [R \mid \mathbf{t}] \in SE(3).$5

where $T = [R \mid \mathbf{t}] \in SE(3).$6 denotes predicted point maps or Gaussian means, $T = [R \mid \mathbf{t}] \in SE(3).$7 denotes pseudo ground truth from MASt3R, and $T = [R \mid \mathbf{t}] \in SE(3).$8 is a normalization factor. The second is a confidence-weighted term:

$T = [R \mid \mathbf{t}] \in SE(3).$9

where $K$0 is a pixel-aligned confidence score. The first part emphasizes accurate predictions in confident regions, and the second regularizes against overconfidence. The paper does not provide a single explicit total loss formula with weighting coefficients, so the exact global combination of photometric, regression, and confidence terms is unspecified.

Initialization and supervision strategy are central to the method’s behavior. Gaussian centers and coarse poses are initialized by MASt3R; supervision is drawn from observed RGB images, MASt3R pseudo-ground-truth point maps, and confidence weighting. Sparse-view settings of 3, 6, and 12 input views are emphasized, with 200 and 1,000 iterations for training and 500 iterations for test-view optimization in the supplementary implementation details. All methods are trained and evaluated on two NVIDIA RTX 6000 ADA GPUs (Chang et al., 20 Aug 2025).

5. Empirical results, modality ablations, and observed behavior

The main experiments are conducted on NIRPlant under sparse-view evaluation with 3, 6, and 12 training views. The baselines reported are 3DGS, CoR-GS, SplatFields, and InstantSplat. Evaluation uses SSIM, PSNR, and LPIPS. The strongest reported NIRSplat numbers are SSIM $K$1, PSNR $K$2, and LPIPS $K$3 for 3, 6, and 12 views, respectively (Chang et al., 20 Aug 2025).

Method	3-view	6-view	12-view
NIRSplat SSIM / PSNR / LPIPS	0.8268 / 20.7182 / 0.2070	0.8311 / 21.0169 / 0.2071	0.8421 / 21.0814 / 0.2080
InstantSplat SSIM / PSNR / LPIPS	0.7984 / 18.3849 / 0.2797	0.8126 / 18.9233 / 0.2689	0.8134 / 19.0333 / 0.2438
3DGS SSIM / PSNR / LPIPS	0.5074 / 14.1552 / 0.4586	0.5590 / 15.5586 / 0.4469	0.6531 / 17.4352 / 0.4033

The paper summarizes these results as large gains over 3DGS and consistent gains over InstantSplat, including the claim that NIRSplat with 3 views can outperform prior models using 12 views. Qualitative examples focus on Lemon under strong sunlight, Kale under artificial light, Art2 with occlusion, and Cabbage as a small object. Standard methods are described as producing blurred textures, semantic loss, structural collapse, duplicated geometry under occlusion, and washed-out appearance under overexposure, whereas NIRSplat improves saturated-region recovery, fine detail in small objects, structural consistency under occlusion, and scene coherence in low-texture and low-visibility regions.

The ablations identify the relative contribution of each modality. Adding text alone to the RGB baseline yields modest changes, while adding NIR alone gives a much larger improvement. The combined NIR + text configuration is best overall. At 3 views, the supplementary summary reports gains over the RGB-only baseline of $K$4 SSIM, $K$5 PSNR, and $K$6 LPIPS for the combined configuration. This suggests that the dominant improvement comes from NIR, with metadata acting as a secondary but measurable refinement.

The positional encoding ablation is especially strong. Without 3D point-based positional encoding, the model reports SSIM $K$7, PSNR $K$8, and LPIPS $K$9; with positional encoding, it reports SSIM $$\tilde{\mathbf{u}}_i = K \cdot (R \cdot \mathbf{p}_i + \mathbf{t}) \in \mathbb{R}^{3},$$0, PSNR $$\tilde{\mathbf{u}}_i = K \cdot (R \cdot \mathbf{p}_i + \mathbf{t}) \in \mathbb{R}^{3},$$1, and LPIPS $$\tilde{\mathbf{u}}_i = K \cdot (R \cdot \mathbf{p}_i + \mathbf{t}) \in \mathbb{R}^{3},$$2. The metadata ablation at 6 views is smaller but still favorable: without metadata, SSIM $$\tilde{\mathbf{u}}_i = K \cdot (R \cdot \mathbf{p}_i + \mathbf{t}) \in \mathbb{R}^{3},$$3, PSNR $$\tilde{\mathbf{u}}_i = K \cdot (R \cdot \mathbf{p}_i + \mathbf{t}) \in \mathbb{R}^{3},$$4, LPIPS $$\tilde{\mathbf{u}}_i = K \cdot (R \cdot \mathbf{p}_i + \mathbf{t}) \in \mathbb{R}^{3},$$5; with metadata, SSIM $$\tilde{\mathbf{u}}_i = K \cdot (R \cdot \mathbf{p}_i + \mathbf{t}) \in \mathbb{R}^{3},$$6, PSNR $$\tilde{\mathbf{u}}_i = K \cdot (R \cdot \mathbf{p}_i + \mathbf{t}) \in \mathbb{R}^{3},$$7, LPIPS $$\tilde{\mathbf{u}}_i = K \cdot (R \cdot \mathbf{p}_i + \mathbf{t}) \in \mathbb{R}^{3},$$8. The paper also reports that botanic-aware prompts outperform plain prompts, and that among tested vision-language encoders BLIP-2 gives the best overall tradeoff at 6 views, with SSIM $$\tilde{\mathbf{u}}_i = K \cdot (R \cdot \mathbf{p}_i + \mathbf{t}) \in \mathbb{R}^{3},$$9, PSNR $$\mathbf{u}_i = \left[ \frac{\tilde{u}_i^x}{\tilde{u}_i^z}, \frac{\tilde{u}_i^y}{\tilde{u}_i^z} \right] \in \mathbb{R}^{2}.$$0, and LPIPS $$\mathbf{u}_i = \left[ \frac{\tilde{u}_i^x}{\tilde{u}_i^z}, \frac{\tilde{u}_i^y}{\tilde{u}_i^z} \right] \in \mathbb{R}^{2}.$$1 (Chang et al., 20 Aug 2025).

A notable negative result concerns naive multimodal fusion. Directly using $$\mathbf{u}_i = \left[ \frac{\tilde{u}_i^x}{\tilde{u}_i^z}, \frac{\tilde{u}_i^y}{\tilde{u}_i^z} \right] \in \mathbb{R}^{2}.$$2, simple concatenation $$\mathbf{u}_i = \left[ \frac{\tilde{u}_i^x}{\tilde{u}_i^z}, \frac{\tilde{u}_i^y}{\tilde{u}_i^z} \right] \in \mathbb{R}^{2}.$$3, elementwise summation $$\mathbf{u}_i = \left[ \frac{\tilde{u}_i^x}{\tilde{u}_i^z}, \frac{\tilde{u}_i^y}{\tilde{u}_i^z} \right] \in \mathbb{R}^{2}.$$4, or naive text addition either degrades performance or yields only trivial and inconsistent gains. The paper attributes this to spectral discrepancy, value-distribution mismatch, and semantic or spatial misalignment. In the context of NIRSplat, this makes cross-attention and shared positional anchoring structural necessities rather than optional embellishments.

6. Position within the literature, scope, and limitations

NIRSplat occupies a distinct position within Gaussian-splatting research. It is more specialized than generic view-dependent hybrid splatting methods such as Viewing Direction Gaussian Splatting, which combines explicit Gaussian geometry with a NeRF-style neural appearance module conditioned on viewing direction; that line of work is relevant as a precursor for neural appearance attached to Gaussians, but it does not address multimodal NIR-text fusion or agricultural sensing priors (Malarz et al., 2023). It is also different from raw-domain radiometric adaptations such as HDRSplat, which adapts 3DGS to 14-bit linear raw images for low-light and high-dynamic-range reconstruction; both methods depart from standard RGB pipelines, but HDRSplat is organized around radiometric fidelity in visible raw imaging rather than NIR-informed agricultural multimodality (Singh et al., 2024). Relative to spectral rendering methods such as Hyperspectral Gaussian Splatting, NIRSplat is narrower spectrally but richer semantically: HS-GS extends 3DGS to $$\mathbf{u}_i = \left[ \frac{\tilde{u}_i^x}{\tilde{u}_i^z}, \frac{\tilde{u}_i^y}{\tilde{u}_i^z} \right] \in \mathbb{R}^{2}.$$5-channel hyperspectral rendering with wavelength-conditioned spherical harmonics, spectral losses, and diffusion refinement, whereas NIRSplat uses single-channel NIR plus text and geometry-aware cross-attention to stabilize plant reconstruction under sparse views and difficult lighting (Narayanan et al., 28 May 2025).

The name can also invite confusion with near-infrared spectroscopy. In the literature provided, broadband NIRS refers to spectroscopic measurement systems such as microCYRIL, a portable fibreless wearable broadband near-infrared spectroscopy device for tracking HbO$$\mathbf{u}_i = \left[ \frac{\tilde{u}_i^x}{\tilde{u}_i^z}, \frac{\tilde{u}_i^y}{\tilde{u}_i^z} \right] \in \mathbb{R}^{2}.$$6, HHb, and oxCCO, while operator-adaptive calibration refers to chemometric model design for NIRS regression and preprocessing selection. These are distinct from NIRSplat in both sensing modality and objective: NIRSplat is a 3D reconstruction framework built on image-based Gaussian splatting, not a spectroscopy device or a spectral-regression calibration model (Talati et al., 2024, Beurier et al., 13 May 2026).

The main stated limitation of NIRSplat is computational overhead. Additional modalities increase cost, and transformer-based fusion raises model complexity, reducing real-time rendering efficiency relative to simpler systems. The paper also leaves several implementation boundaries explicit. Depth and LiDAR are part of the dataset, but their direct use by the model is not specified. Precise calibration and registration procedures are only partially described. The method assumes aligned RGB and NIR frames and relies on MASt3R for dense point maps and coarse extrinsics. A plausible implication is that the method’s gains depend not only on access to NIR but also on successful cross-modal alignment and on the availability of sufficiently informative metadata. Within those constraints, NIRSplat is best understood as a multimodal, semantics-aware agricultural reconstruction system that extends 3DGS by making NIR cues and contextual priors first-class components of Gaussian prediction and sparse-view optimization (Chang et al., 20 Aug 2025).