---
title: Active RGB-NIR Imaging
url: https://www.emergentmind.com/topics/active-rgb-nir-imaging
type: topic
---

# Active RGB-NIR Imaging

Active RGB-NIR imaging denotes imaging systems that combine visible RGB sensing with near-infrared sensing under controlled acquisition, often with active NIR illumination or actively controlled scene illumination, in order to exploit the complementarity between visible-spectrum color information and NIR-derived structure, shading, reflectance, or spectral cues. The term covers several distinct but related regimes: pixel-aligned RGB-NIR cameras with 850 nm illuminators for robot vision, gated RGB-NIR stereo with 808 nm flood illumination, single-sensor VIS-NIR-mix capture without an IR-cut filter, actively illuminated RGB plus NIR flash inverse-rendering platforms, and jointly acquired RGB with NIR hyperspectral imaging for scientific phenotyping [2411.18025] [2405.12759] [2005.04580] [2605.30250] [2504.16658]. Across these regimes, the central premise is stable: RGB retains visible appearance and chromatic semantics, while NIR can provide higher SNR in darkness, more stable active illumination, or wavelength-specific material response that is either unavailable or difficult to isolate in passive RGB alone.

## 1. Spectral scope and physical rationale

The NIR component in active RGB-NIR imaging is not tied to a single band or sensor model. Reported systems span approximately \(700\text{–}1000\,\mathrm{nm}\) in dual-CCD road-scene colorization, 850 nm in robot vision with Advanced Illumination AL295-150850IC, approximately 808 nm flood illumination in cross-spectral gated stereo, approximately \(780\text{–}2500\,\mathrm{nm}\) in NIR-to-RGB spectral domain translation, and \(900\text{–}1700\,\mathrm{nm}\) in Specim FX17 line-scan hyperspectral imaging [1604.02245] [2411.18025] [2405.12759] [2312.16040] [2504.16658]. This range diversity reflects different objectives: low-light structure sensing, depth from active gating, spectral translation, and chemometric analysis are not constrained to the same optical band.

The underlying motivation is likewise task-dependent. In low-light fusion and denoising, NIR images under invisible near-infrared flash or built-in NIR illumination retain edges and shading when RGB is dominated by Poisson-Gaussian noise or high-ISO degradation [2303.06834] [2404.08514]. In inverse rendering, the key advantage is that the NIR flash lies outside the RGB sensor’s spectral passband and is imperceptible to humans, so the NIR channel can be dominated by a controlled point-light term while RGB remains passive and records ambient visible appearance [2605.30250]. In 24-hour colorful imaging with a single silicon sensor, removing the IR-cut filter exposes the Bayer channels to both VIS and NIR, creating mixed measurements that can be computationally separated and then re-integrated using NIR-guided enhancement [2005.04580]. In agricultural phenotyping, concurrent RGB and NIR-HSI acquisition under the same illumination enables morphology, reflectance, and pseudo-absorbance to be analyzed jointly over time [2504.16658].

Several papers make explicit that the complementarity is conditional rather than absolute. NIR can be structurally informative but spectrally inconsistent with visible RGB, because materials may show different contrast, shadows, or reflectance in the two domains [2303.06834] [2405.19914] [2404.08514]. This directly challenges any assumption that active RGB-NIR imaging is merely “RGB plus a cleaner grayscale channel.” The literature instead treats cross-spectral inconsistency as a first-order modeling problem.

## 2. Acquisition architectures, synchronization, and calibration

Representative active RGB-NIR systems differ sharply in acquisition geometry. Pixel-aligned designs use prism-based or dichroic beam-splitter cameras so that RGB and NIR share an optical path and are co-axial at the sensor level. The robot-vision system with two JAI FS‑1600D‑10GE RGB-NIR cameras forms a stereo pair, each camera being paired with an NIR illuminator, and supplements the imaging stack with an Ouster OS1 LiDAR and PTP synchronization [2411.18025]. The inverse-rendering platform similarly uses a prism-based RGB-NIR camera, JAI FS-1600, synchronized with an Advanced Illumination AL295 NIR flash and mounted on a robotic arm atop a wheeled mobile base, thereby enabling dense multi-view RGB-NIR capture with flash-on/flash-off subtraction [2605.30250].

Other systems are explicitly non-prismatic but still actively controlled. The barley germination dataset uses a Basler Ace 2 Pro RGB line-scan camera and a Specim FX17 line-scan hyperspectral camera, both operating simultaneously while a conveyor belt moves a Petri dish plate carrying chessboards and PTFE white references under six 20 W halogen bulbs [2504.16658]. The hardware is active in the sense that both illumination and geometry are controlled, and RGB and NIR-HSI are acquired simultaneously under the same illumination. In contrast, Real-NAID uses a Huawei X2381‑VG surveillance camera with a built-in NIR illuminator; captures are sequential and restricted to static scenes so that RGB and NIR remain aligned without explicit geometric calibration [2404.08514]. The single-sensor VIS-NIR-MIX system uses a Bitran CS-63C camera without an IR-cut filter and a motorized filter wheel for dataset construction, with a xenon lamp and an 880 nm band-pass filter used to emulate active NIR LED illumination at night [2005.04580].

Radiometric standardization is central to these systems. In the barley setup, white and dark correction are applied row-wise and per-channel via
$$
R(i,j,c)=\frac{I_{\text{raw}}(i,j,c)-D(i,c)}{W(i,c)-D(i,c)},
$$
with \(D=0\) for RGB and a shutter-closed dark reference for NIR-HSI; PTFE foil provides the white reference, and size correction is then performed by bilinear interpolation using chessboard geometry [2504.16658]. In robot vision, the per-channel image-formation model explicitly includes active illumination only in NIR,
$$
I_i^c(p^c)=\eta_1+g_i\left[\eta_2+t_i\left(R_i^c(p^c)\left(E_i^c(p^c)+L_i^c(p^c)\right)\right)\right],
$$
with \(L=0\) for RGB and \(L_{\text{NIR}}\ge 0\) from the 850 nm LED [2411.18025]. In inverse rendering, the NIR flash-only radiance is isolated by
$$
I^{\text{NIR}}=I^{\text{NIR-on}}-I^{\text{NIR-off}},
$$
which removes ambient NIR and yields a point-light shading term suitable for BRDF and geometry estimation [2605.30250].

Geometric calibration ranges from identity alignment in prism cameras to explicit affine or projective registration in heterogeneous systems. The barley dataset estimates RGB-to-RGB alignment across sessions with ArUco-based RANSAC affine transformation and RGB-to-HSI alignment with chessboard centers, enabling the same physical kernel to be cropped in both modalities at the same time point [2504.16658]. Cross-modal registration more generally is benchmarked by RGB-NIR-IRegis, where aligned RGB-NIR pairs are available through a monocular RGB-NIR camera and cross-view homographies are annotated for unaligned pairs, with evaluation based on corner reprojection error and AUC@3px, AUC@5px, and AUC@10px [2405.19914].

## 3. Core computational paradigms

A dominant paradigm is RGB-NIR fusion under explicit inconsistency modeling. “DarkVisionNet: Low-Light Imaging via RGB-NIR Fusion with Deep Inconsistency Prior” formulates fusion around deep structures extracted from RGB and NIR feature spaces and a Deep Inconsistency Prior,
$$
\mathcal{F}(\text{edge}^C,\text{edge}^N)=\lambda (1-\text{edge}^C)(1-\text{edge}^N)+\text{edge}^C\text{edge}^N,
$$
with \(\lambda=0.5\) in experiments, so that inconsistent NIR structures are suppressed during fusion [2303.06834]. “NIR-Assisted Image Denoising: A Selective Fusion Approach and A Real-World Benchmark Dataset” instead introduces a Selective Fusion Module that factorizes the NIR/RGB fusion weights into global and local components, \(W=W^g\odot W^l\), and performs complementary softmax gating for both modalities before reconstruction [2404.08514]. Both approaches directly reject the naïve assumption that NIR structure should always be injected into RGB.

A second paradigm is spectral-domain translation or colorization. “Multi-scale Progressive Feature Embedding for Accurate NIR-to-RGB Spectral Domain Translation” decomposes the problem into a NIR\(\rightarrow\)grayscale domain-translation stage and a grayscale\(\rightarrow\)RGB colorization stage, with multi-scale SCCM supervision and feature-level adversarial alignment, and reports that MPFNet outperforms state-of-the-art counterparts by \(2.55\,\mathrm{dB}\) PSNR on the VCIP2020 dataset [2312.16040]. The earlier “Infrared Colorization Using Deep Convolutional Neural Networks” likewise treats NIR-to-RGB estimation as low-frequency RGB regression plus deterministic high-frequency NIR detail reinjection, using a multi-scale CNN with a low-frequency bypass and joint bilateral filtering on the output [1604.02245]. These methods are not fusion in the strict sense; they generate RGB-like imagery from NIR inputs and therefore inherit the one-to-many ambiguity of cross-spectral color assignment.

A third paradigm centers on geometric inference. In robot vision, pixel-aligned RGB-NIR stereo supports both image-level fusion compatible with RGB-pretrained models and feature-level fusion integrated into RAFT-Stereo, where alternating Fusion–NIR cost volumes produced the best depth RMSE, \(6.747\,\mathrm{m}\), relative to RGB-only, NIR-only, or simpler fusion baselines [2411.18025]. In cross-spectral gated stereo, active NIR time-gated slices, passive high-resolution RCCB stereo, physics-based gated reconstruction, and LiDAR supervision are fused in a CREStereo-style architecture with pose refinement and attention-based cross-modal feature fusion; the method improves MAE by about \(39\%\) over the next best method in the \(100\text{–}220\,\mathrm{m}\) range [2405.12759]. In inverse rendering, a three-stage pipeline initializes geometry from ambient RGB using 2D Gaussian splatting, refines geometry and NIR BRDF under flash-only NIR, and then solves for RGB diffuse albedo and the RGB environment map with cross-spectral sharing of roughness and metallic parameters [2605.30250].

Cross-modality registration has become an independent front-end problem rather than a pre-processing detail. “Towards RGB-NIR Cross-modality Image Registration and Beyond” argues that inconsistent local features have a toxic impact on registration quality and proposes SGFormer, which injects high-level semantic guidance into a LoFTR-style matcher via a Semantic Injection Module and a Semantic Triplet Loss [2405.19914]. This suggests that active RGB-NIR imaging increasingly depends on semantics-aware correspondence, not only on photometric or descriptor-level similarity.

## 4. Datasets, benchmarks, and evaluation regimes

The dataset ecosystem is heterogeneous because active RGB-NIR imaging spans scientific imaging, surveillance, robot vision, inverse rendering, and low-light restoration. The barley germination dataset provides RGB images, NIR-HSI images, segmentation masks, full-dish imagery, grid coordinates, and mean pseudo-absorbance spectra for 2242 individual barley kernels, each imaged pre-moisture and then every 24 hours for five consecutive days, for a total of six sessions per kernel [2504.16658]. It is notable for simultaneous RGB and NIR-HSI acquisition, explicit segmentation via Otsu’s method, and time-series labeling of germination day.

For low-light fusion, Dark Vision Dataset is described as the first public RGBNIR fusion benchmark and contains 5k aligned RGB-NIR reference pairs cropped to \(256\times256\) for training, 1k reference pairs for testing, and 10 additional real noisy pairs at \(1920\times1080\) [2303.06834]. Real-NAID complements this with real noisy RGB plus clean RGB and clean NIR guidance: 100 static scenes, each with three noisy RGB images, one clean RGB image, and one clean NIR image, for 300 RGB-NIR noisy/clean instances split into 70 training scenes and 30 test scenes [2404.08514]. These two datasets occupy different positions in the literature: DVD emphasizes aligned RGB-NIR fusion under synthetic and real low-light noise, whereas Real-NAID emphasizes real surveillance-style capture with an active built-in NIR illuminator.

For robot vision, the pixel-aligned RGB-NIR stereo dataset includes 39 training videos with approximately 73,000 frames and 4 test videos with approximately 7,000 frames, for a total of approximately 80,000 frames across 43 scenes, with synchronized LiDAR and per-sensor exposure times [2411.18025]. RGB-NIR-IRegis addresses registration rather than reconstruction, providing 25 scene sequences and 260 unaligned image pairs with viewpoint variation, plus aligned pairs from a monocular RGB-NIR camera, thereby enabling fair evaluation of cross-modality registration under both aligned and cross-view settings [2405.19914]. Cross-spectral gated stereo extends the evaluation regime to automotive-scale depth, using test sets with 2463 frames to \(160\,\mathrm{m}\) and 655 additional frames with accumulated LiDAR maps to \(220\,\mathrm{m}\) [2405.12759].

Inverse rendering introduces yet another benchmark structure. The active RGB-NIR inverse-rendering dataset is described as the first multi-view RGB-NIR inverse-rendering dataset captured across multiple ambient illumination conditions; each object–environment pair contains over 100 synchronized RGB, ambient NIR, and flash-on NIR frames, plus masks and camera poses [2605.30250]. By contrast, the VIS-NIR-MIX dataset for 24-hour colorful imaging contains 102 scenes and 714 images, organized as seven captures per scene across day and night conditions, with filter-wheel-based alignment and long-exposure VIS references at night [2005.04580]. The coexistence of such different benchmarks is itself informative: active RGB-NIR imaging has no single canonical protocol because the sensing objectives vary from reflectance inversion to robot depth to spectral chemometrics.

## 5. Major application domains

Low-light restoration is the most established application area. DarkVisionNet reports DVD performance of \(31.50\,\mathrm{dB}\) and SSIM \(0.9551\) at \(\sigma=2\), \(29.62\,\mathrm{dB}\) and SSIM \(0.9400\) at \(\sigma=4\), and best PSNR/SSIM at heavier noise levels \(\sigma=6\) and \(\sigma=8\), outperforming fusion and denoising baselines especially under heavy noise [2303.06834]. Real-NAID shows that SFM-equipped denoisers improve most strongly at higher noise; for example, NIR‑Restormer improves from \(23.82/0.7297/0.387\) to \(24.76/0.7626/0.315\) in PSNR/SSIM/LPIPS at the high-noise setting [2404.08514]. The integrated single-sensor 24-hour imaging pipeline goes further by addressing both daytime NIR contamination and nighttime chrominance absence, reaching daytime PSNR \(22.25\), SSIM \(0.871\), Colorfulness \(73.24\), and nighttime PSNR \(17.17\), SSIM \(0.835\), Colorfulness \(89.74\) on its test set [2005.04580].

Depth estimation and robot perception constitute a second major domain. In pixel-aligned robot stereo, learned HSV-brightness fusion improves YOLOv8 object detection without retraining from mAP \(0.756\) on RGB to \(0.809\) on fused images, while feature fusion improves stereo depth RMSE from \(8.943\,\mathrm{m}\) for RGB-only RAFT-Stereo to \(6.747\,\mathrm{m}\) [2411.18025]. Cross-spectral gated stereo extends active RGB-NIR to long-range automotive depth, reporting night RMSE \(6.23\,\mathrm{m}\), MAE \(2.03\,\mathrm{m}\), ARD \(0.04\), and \(\delta_1=96.69\%\) to \(160\,\mathrm{m}\), as well as \(100\text{–}220\,\mathrm{m}\) MAE values of \(11.15\,\mathrm{m}\) at night and \(10.07\,\mathrm{m}\) by day [2405.12759]. The paper explicitly emphasizes applications such as autonomous driving and “lost cargo” detection.

Inverse rendering and reflectance recovery form a third domain in which active RGB-NIR is used not to improve photographic appearance but to stabilize physical decomposition under uncontrolled ambient light. The active RGB-NIR inverse-rendering method reports RGB diffuse albedo \(27.89/0.9675/0.0680\) in PSNR/SSIM/LPIPS, roughness RMSE \(0.0713\), normal MAE \(7.73^\circ\), and relighting PSNR \(31.01\,\mathrm{dB}\), while requiring \(1.2\) hours per scene [2605.30250]. Here the principal benefit is not merely low-light robustness but ambient robustness: NIR flash provides stable point-light shading that does not contaminate RGB acquisition.

Further domains include NIR-to-RGB visualization and application-specific classification. MPFNet reaches PSNR \(22.14\), SSIM \(0.63\), AE \(3.68\), and LPIPS \(0.253\) on the VCIP test set, improving PSNR by \(2.55\,\mathrm{dB}\) over ATCycleGAN [2312.16040]. The older deep colorization model, trained on 38,495 RGB-NIR road-scene pairs from a multi-CCD camera, achieves RMSE \(0.130\pm0.043\) and S-CIELAB \(8.88\pm3.15\) for its best topology [1604.02245]. In night-time fire detection, grayscale surrogates used as NIR-like data yield mAP@50 \(0.908\), mAP\(_{50:95}\) \(0.622\), and F1 \(0.86\), and a two-stage pipeline using YOLOv11n and EfficientNetV2-B0 is designed to reduce false positives from bright artificial lights [2512.23594]. In plant science, the barley dataset explicitly supports classification of germinated versus non-germinated kernels, multimodal time-series analysis, and exploratory chemometric analysis via cleaned pseudo-absorbance spectra [2504.16658].

## 6. Limitations, misconceptions, and research directions

A recurrent misconception is that pixel alignment or simultaneous acquisition eliminates the substantive difficulty of RGB-NIR fusion. The literature shows the opposite. Cross-spectral inconsistency arises because NIR can contain shadows absent in RGB, RGB can contain color patterns absent in NIR, and local gradient statistics can differ enough to destabilize feature matching and fusion [2303.06834] [2405.19914] [2404.08514]. Pixel alignment removes a geometric nuisance variable, but it does not resolve modality-dependent reflectance or semantics.

A second misconception is that NIR-to-RGB translation recovers ground-truth visible color. “Multi-scale Progressive Feature Embedding for Accurate NIR-to-RGB Spectral Domain Translation” states that the mapping is intrinsically ambiguous because NIR reflectance is non-overlapping with the visible spectrum and therefore induces a one-to-many color mapping [2312.16040]. The earlier road-scene colorization work likewise notes failures when visible signals are absent from NIR, such as LED traffic lights [1604.02245]. These results imply that colorized NIR is best interpreted as a learned RGB surrogate, not as a spectrally faithful reconstruction.

Active NIR itself is also not uniformly robust. Outdoor sunlight contains strong ambient NIR, which reduces the headroom of flash-after-subtraction in inverse rendering and degrades gated SNR in automotive depth estimation [2605.30250] [2405.12759]. Strong solar IR, bright artificial lights, glare, fog, rain, and material-specific NIR reflectance all remain failure modes in the surveyed systems [2512.23594] [2405.12759]. A plausible implication is that “active” should not be conflated with “ambient-invariant”; rather, it improves controllability within a bounded radiometric regime.

Dataset-specific caveats are equally important. The barley germination dataset is explicitly biased because glue residue from the 3D-printed grid strongly inhibited germination, so transfer to standard malting conditions requires caution [2504.16658]. Real-NAID is restricted to static scenes with sequential capture [2404.08514]. The VIS-NIR-MIX dataset uses sequential filter-wheel acquisition and leaves dynamic scenes for future work [2005.04580]. The 2016 road-scene colorization dataset is summer daylight only and therefore does not cover night or active illumination despite its relevance to active systems [1604.02245]. These constraints underscore that benchmark diversity in active RGB-NIR imaging still lags the diversity of intended deployments.

Current research directions in the cited works are relatively consistent. They include physics-informed constraints and richer spectral priors for NIR-to-RGB translation [2312.16040], improved alignment and temporally consistent fusion [2303.06834], adaptive NIR illumination control and learned geometric warping for denoising [2404.08514], tighter LiDAR integration and generative RGB-NIR modeling for robot vision [2411.18025], coded or structured NIR illumination and joint RGB-NIR BRDF learning for inverse rendering [2605.30250], and broader integration of patch-based detection and temporal flame dynamics for surveillance [2512.23594]. Taken together, these directions suggest that the field is moving from simple two-image fusion toward tightly coupled spectral, geometric, temporal, and physical inference.

Source: https://www.emergentmind.com/topics/active-rgb-nir-imaging