NightVision: Advances in Low-Light Imaging

• NightVision is a research area focused on converting low-light or infrared inputs into visually coherent representations that reliably preserve object details.
• Various methodologies—including thermal-to-visible synthesis, biologically inspired preprocessing, and sensor fusion—enable robust detection and tracking in challenging environments.
• Advanced techniques such as event-based sensing, single-pixel imaging, and nonlinear upconversion tackle issues like low signal, glow suppression, and distribution shifts.

NightVision denotes a heterogeneous research area concerned with making scenes observed in darkness or severe low illumination interpretable to humans and usable by downstream vision systems. In the cited literature, the term spans thermal infrared to visible-like image synthesis, low-light enhancement with explicit suppression of glow and light effects, infrared/visible fusion for surveillance, event-based night-time sensing, night-time satellite IR-to-visible generation, and physical infrared-to-visible conversion or projection systems (Almasri et al., 2020, Stracke et al., 29 Sep 2025, Yasin et al., 2020, Harder et al., 2020). Across these settings, the central technical tension is recurrent: night-vision systems must improve visibility without deleting objects, hallucinating structure, or collapsing under distribution shift.

1. Problem formulations and sensing regimes

A foundational formulation treats night vision as the task of converting a non-visible or weakly visible scene into a visible or visible-like representation. In perceptual thermal colorization, the input is a thermal long-wave infrared image $x \in \mathbb{R}^{H \times W}$ and the target is a visible RGB image $\hat Y \in \mathbb{R}^{H \times W \times 3}$, but the objective is explicitly not exact physical RGB reconstruction; it is a robust, visually coherent representation that preserves all objects present in thermal (Almasri et al., 2020). This distinction matters because thermal IR encodes emitted radiation proportional to object temperature and emissivity, whereas visible RGB encodes reflected light and is strongly dependent on illumination and material reflectance. The mapping is therefore ill-posed: one thermal pattern can correspond to many possible visible colors, and some objects appear in one spectrum but not the other (Almasri et al., 2020).

Other NightVision formulations retain visible sensing but redesign the input pipeline. A biologically inspired line of work models early human visual processing by converting RGB into grayscale or opponent-color channels and applying Difference-of-Gaussians-like contrast extraction before a standard segmentation network, with no architectural change to the downstream model (Stracke et al., 29 Sep 2025). A further class of methods works with paired or aligned visible and infrared streams and fuses them so that thermally salient targets, especially pedestrians, are emphasized inside a visible contextual frame (Malviya et al., 2010).

NightVision also includes sensors that depart from conventional frame-based imaging. Event-based cameras operate asynchronously, trigger events when a sufficient change in log intensity occurs, and provide about $120\,\mathrm{dB}$ dynamic range together with microsecond temporal resolution, which makes them attractive for obstacle detection in low-light conditions where conventional cameras suffer from low signal-to-noise ratio, motion blur, and limited dynamic range (Yasin et al., 2020). At the other extreme, night-time satellite NightVision uses thermal IR observations from GOES-16 ABI to synthesize visible-like RGB composites so that visible-trained Earth-observation pipelines can operate continuously across the diurnal cycle (Harder et al., 2020).

2. Thermal-to-visible synthesis and perceptual rendering

One influential formulation of perceptual night vision decouples low-frequency visible appearance from high-frequency thermal structure. The proposed pipeline maps a thermal image $x$ to a visible RGB prediction $G(x)$, extracts low-frequency components with a Gaussian blur, and constructs the final image by pan-sharpening: $\hat{Y}_{pan} = G(x)^{LF} + \lambda x^{HF}.$ With a $25\times25$ Gaussian kernel and $\sigma=12$, the method explicitly uses thermal high-frequency residuals to preserve edges, shapes, and small objects while asking the network to predict only coarse visible color and luminance; empirically, $\lambda=3$ is chosen as a trade-off between sharpness and clipping artifacts (Almasri et al., 2020). Berg et al.’s TIR2Lab backbone is reused in lightweight encoder-decoder form, with a variant replacing BatchNorm by InstanceNorm following Ulyanov et al.; training uses an $L_1$ content loss and a heavily weighted low-frequency MSE term,

$\hat Y \in \mathbb{R}^{H \times W \times 3}$0

so that large regions such as sky, vegetation, roads, and buildings are encouraged to match visible low-frequency structure while thermal high-frequency injection guarantees structural faithfulness (Almasri et al., 2020). A common misconception is that night vision of this kind seeks exact color recovery; the paper instead states that the model generates color values consistent with ground truth when object appearance is stable and averaged grey values in ambiguous cases, which is presented as a conservative behavior rather than a failure (Almasri et al., 2020).

This robustness-oriented position is sharpened by comparison with adversarial baselines. TICPan does not achieve the best PSNR or SSIM on ULB17-VT.v2 or KAIST-MS, but it preserves people, cars, road markings, and buildings more reliably than methods that produce sharper or more vivid outputs at the cost of hallucinating or deleting objects (Almasri et al., 2020). This suggests a substantive controversy within NightVision research: pixel-wise realism and operator safety are not always aligned.

A different solution to thermal night vision is IR2VI, which formulates night IR to day visible translation as an unsupervised image-to-image translation problem between night-time IR and day-time visible domains. IR2VI uses two generators, global and ROI discriminators, a structure connection module, and ROI focal loss so that scene geometry is preserved and small targets such as soldiers and vehicles receive explicit local supervision (Liu et al., 2018). In downstream evaluation, a Faster R-CNN detector trained only on day-time visible images attains AP values of $\hat Y \in \mathbb{R}^{H \times W \times 3}$1 for CycleGAN, $\hat Y \in \mathbb{R}^{H \times W \times 3}$2 for UNIT, $\hat Y \in \mathbb{R}^{H \times W \times 3}$3 for StarGAN, and $\hat Y \in \mathbb{R}^{H \times W \times 3}$4 for IR2VI on translated night IR, which the paper interprets as evidence that structure connection prevents incorrect mapping while ROI-focused losses recover fine detail (Liu et al., 2018).

NightVision has also been extended to remote sensing. In the satellite setting, GOES-16 ABI IR composites derived from bands 8–16 are translated into visible RGB composites using U-Net, cGAN, and U-Net++ architectures on $\hat Y \in \mathbb{R}^{H \times W \times 3}$5 imagery. U-Net++ without deep supervision achieves SSIM $\hat Y \in \mathbb{R}^{H \times W \times 3}$6 and RMSE $\hat Y \in \mathbb{R}^{H \times W \times 3}$7 on an independent daylight test set, while the cGAN yields lower quantitative fidelity but sometimes more uniformly colored night-time outputs (Harder et al., 2020). Here again, the same broad issue appears: day-trained IR-to-visible mappings can produce visually convincing outputs at night, but their night-time behavior is qualitatively assessed because direct visible ground truth is unavailable (Harder et al., 2020).

3. Enhancement, preprocessing, and suppression of light effects

A major branch of NightVision research keeps the visible sensor but modifies the image formation or preprocessing chain so that downstream tasks become more robust under night-time and adverse conditions. A biologically inspired approach applies linear color reparameterization—identity RGB, grayscale, or opponent-color channels—followed by a fixed, non-trainable center–surround filter implemented with recursive $\hat Y \in \mathbb{R}^{H \times W \times 3}$8 box blurs: $\hat Y \in \mathbb{R}^{H \times W \times 3}$9 Inserted before DeepLabV3+, SegFormer, or InternImage, this preprocessing materially improves robustness on Dark Zurich and ACDC Night while largely preserving Cityscapes performance. For DeepLabV3+ with ResNet-50, baseline Dark Zurich performance is $120\,\mathrm{dB}$0 mIoU; equal-weights grayscale at $120\,\mathrm{dB}$1 raises it to $120\,\mathrm{dB}$2, and grayscale plus DoG at depth $120\,\mathrm{dB}$3 yields $120\,\mathrm{dB}$4. On ACDC Night, the same baseline rises from $120\,\mathrm{dB}$5 to $120\,\mathrm{dB}$6 with equal grayscale at $120\,\mathrm{dB}$7 and $120\,\mathrm{dB}$8 with grayscale plus DoG at depth $120\,\mathrm{dB}$9 (Stracke et al., 29 Sep 2025). The paper’s interpretation is that when low luminance is the dominant problem, luminance-focused preprocessing is more important than preserving color.

Where night scenes contain strong artificial lights, the dominant degradation may be glow rather than darkness alone. One physics-guided model derives a Nighttime Imaging Model with Near-field Light Sources,

$x$0

and casts glow suppression as blind deconvolution in which the light-source map $x$1, the atmospheric point spread function, and the direct transmission are jointly estimated by a zero-shot Light-aware Blind Deconvolution Network, followed by a Retinex-based Enhancement Module (Wu et al., 2023). This line of work explicitly distinguishes glow from generic haze and argues that standard low-light enhancement enlarges and diffuses halos instead of removing them (Wu et al., 2023). A related unsupervised decomposition model uses

$x$2

to separate reflectance, shading, and light-effects layers, then uses a light-effects suppression network guided by the estimated $x$3 to suppress glare while enhancing dark regions (Jin et al., 2022). On LOL-test, that model reports MSE $x$4, PSNR $x$5, and SSIM $x$6; on LOL-Real it reports PSNR $x$7 and SSIM $x$8, while user study scores for realism, light-effects suppression, and visibility are all أعلى among the compared methods (Jin et al., 2022). These results reinforce a second recurrent theme in NightVision: enhancement without explicit handling of light effects often worsens exactly the regions that matter most.

NightVision pipelines aimed at automatic recognition in NIR imagery show the same dependence on preprocessing. In a controlled NIR surveillance setup using a Raspberry Pi IR camera and active NIR LEDs, Histogram Equalization and Adaptive Histogram Equalization improve classification over raw IR, but CLAHE is the strongest preprocessing: object-classification accuracy rises from $x$9 on original low-light IR to $G(x)$0 with HE, $G(x)$1 with AHE, and $G(x)$2 with CLAHE (Bhandari et al., 2020). This suggests that even where end-to-end CNNs are available, carefully chosen spatial-domain enhancement remains consequential.

4. Fusion, detection, tracking, and downstream perception

NightVision systems are often judged less by image fidelity than by whether they improve detection, tracking, or human situational awareness. A classical surveillance formulation fuses a thermal IR stream and a visible RGB stream from a pre-aligned static camera pair. Background estimation and object extraction are performed in the IR domain, moving pedestrians are segmented with thresholding plus area and height/width-ratio constraints, and the corresponding RGB pixels are brightened or boxed so that thermal targets are embedded into a familiar visible scene (Malviya et al., 2010). The method is qualitative, but it demonstrates an enduring design principle: IR supplies target saliency, while visible imagery supplies spatial context.

A more task-specific thermal surveillance pipeline uses FLIR imagery, HOG features, and adaptive background subtraction for human detection. For a static camera, HOG alone yields precision $G(x)$3, recall $G(x)$4, and execution time $G(x)$5 s, while HOG plus background subtraction yields precision $G(x)$6, recall $G(x)$7, and execution time $G(x)$8 s. For a dynamic camera, adaptive background subtraction improves precision from $G(x)$9 to $\hat{Y}_{pan} = G(x)^{LF} + \lambda x^{HF}.$0 and reduces execution time from $\hat{Y}_{pan} = G(x)^{LF} + \lambda x^{HF}.$1 s to $\hat{Y}_{pan} = G(x)^{LF} + \lambda x^{HF}.$2 s, again at $\hat{Y}_{pan} = G(x)^{LF} + \lambda x^{HF}.$3 recall (Khandhediya et al., 2017). In this setting, NightVision is not colorization or enhancement but stable separation of warm moving humans from a changing thermal background.

Event-based NightVision replaces image enhancement by sensing that is intrinsically more robust to low light. A DVS-based collision-avoidance pipeline suppresses background activity noise with a local spatio-temporal filter, slices events adaptively by event count, extracts objects with a randomized Hough transform, detects features using LC-Harris, estimates depth by triangulation, and applies an Asynchronous Adaptive Collision Avoidance algorithm (Yasin et al., 2020). The cited work is qualitative, but it reports that a running person is clearly delineated in the event stream at night, whereas the frame-based intensity image remains very dark and yields poor object extraction (Yasin et al., 2020).

Infrared NightVision is also central to UAV surveillance. TF-Net, an improved YOLOv5s for infrared UAV detection, is evaluated on approximately 3891 IR images and reports $\hat{Y}_{pan} = G(x)^{LF} + \lambda x^{HF}.$4 precision, $\hat{Y}_{pan} = G(x)^{LF} + \lambda x^{HF}.$5 recall, $\hat{Y}_{pan} = G(x)^{LF} + \lambda x^{HF}.$6 mAP@0.5, and $\hat{Y}_{pan} = G(x)^{LF} + \lambda x^{HF}.$7 IoU, with a $\hat{Y}_{pan} = G(x)^{LF} + \lambda x^{HF}.$8 MB model running at $\hat{Y}_{pan} = G(x)^{LF} + \lambda x^{HF}.$9 FPS on a Tesla T4 GPU (Misbah et al., 2022). The network is explicitly tuned for tiny thermal features in complex night-time backgrounds, and the reported comparisons show it outperforming YOLOv5n, s, m, and l in precision, mAP, and IoU (Misbah et al., 2022).

NightVision conditions also appear in pose estimation. A multi-animal pose-estimation and tracking pipeline for cattle, trained on balanced daylight and nightvision data, reports overall keypoint recovery $25\times25$0 by day and $25\times25$1 at night for direct pose estimation at $25\times25$2 px width, with KeySORT improving these to $25\times25$3 and $25\times25$4, respectively (Perneel et al., 13 Mar 2025). The paper characterizes the degradation from daylight to nightvision as limited and attributes much of the robustness to keypoint-centric tracking rather than appearance-based identity cues (Perneel et al., 13 Mar 2025).

5. Correlation imaging, single-pixel sensing, and nonlinear infrared-to-visible conversion

A physically distinct family of NightVision systems avoids conventional IR focal-plane arrays and instead reconstructs or directly projects visible representations from infrared light by exploiting structured illumination, single-pixel detection, or nonlinear optics. One correlation-imaging system modulates paired IR and visible beams with the same SLM pattern, measures a scalar IR signal with a photomultiplier tube, records the corresponding visible pattern with a CCD, and reconstructs the object by intensity cross-correlation. Using $25\times25$5 IR paired with $25\times25$6 visible and $25\times25$7 IR paired with $25\times25$8 visible, the method forms a multi-channel color night-vision image without pseudo-color post-processing; the main demonstration uses $25\times25$9 measurements, and the reported Color Colorfulness Index is higher than both conventional visible-light images and pseudo-color night-vision images (Duan et al., 2021).

A related single-pixel formulation dispenses with any infrared focal-plane array by reflecting two-wavelength IR light from a target onto an amplitude-only SLM, measuring the modulated light with a PMT, reconstructing two grayscale IR images by second-order intensity correlation, and then applying a neural-network color mapping to obtain a visible color result (Duan et al., 2020). This work positions the method as a shift from optical-electric detection to light-field control and reports high-quality color night-vision images from $\sigma=12$0 and $\sigma=12$1 channels, albeit with the familiar trade-off of many measurements per scene (Duan et al., 2020).

SIVIS extends single-pixel NightVision toward photon-starved operation. It combines $\sigma=12$2 IR and $\sigma=12$3 visible beams on a DMD, uses a single-pixel IR detector for feedback, and iteratively evolves the binary pattern $\sigma=12$4 to maximize the cost function

$\sigma=12$5

Because the visible beam is co-modulated by the same DMD pattern, the optimized infrared geometry is projected directly in visible light (Wang et al., 29 Jun 2026). The reported detection limit is $\sigma=12$6 photons per pixel per frame, corresponding to approximately $\sigma=12$7, and demonstrations include anti-counterfeiting features and vascular-like structures beneath tissue phantoms (Wang et al., 29 Jun 2026). A plausible implication is that NightVision can be cast not only as imaging but as active, adaptive probing.

Nonlinear upconversion offers another route. A nonlocal lithium-niobate metasurface performs sum-frequency generation with $\sigma=12$8, converting $\sigma=12$9 SWIR plus an $\lambda=3$0 pump into visible output near $\lambda=3$1: $\lambda=3$2 The reported SFG enhancement is $\lambda=3$3 relative to a bare LiNbO$\lambda=3$4 film, with normalized conversion efficiency $\lambda=3$5, and the authors further show direct upconverted imaging and edge-detection image processing in different diffraction orders (Molina et al., 2024). In this branch of NightVision, the objective is not visible-like hallucination but physical transfer of infrared information into a visible band that can be recorded by standard silicon cameras.

6. Benchmarks, evaluation tensions, and open directions

NightVision research is fragmented across task-specific datasets such as ULB17-VT.v2 and KAIST-MS for thermal-to-visible synthesis, SENSIAC for unsupervised IR-to-visible translation, Cityscapes together with Dark Zurich and ACDC for night-time segmentation robustness, GOES-16 ABI composites for satellite IR-to-visible generation, OTCBVS and ETHZ for thermal surveillance, and UAV_IR for infrared UAV detection. LENVIZ was introduced to address a different gap: large-scale low-exposure visible night vision. It contains $\lambda=3$6 frames from $\lambda=3$7 scenes, uses three mobile CMOS sensors, reaches up to $\lambda=3$8 resolution, and provides up to $\lambda=3$9 short or mid exposures plus $L_1$0 long-exposure image per scene together with expert human-generated ground truth (Aithal et al., 25 Mar 2025). The benchmark spans indoor and outdoor scenes, with and without human subjects, and stores lux metadata derived from

$L_1$1

which makes it suitable for both enhancement and exposure-aware analysis (Aithal et al., 25 Mar 2025).

Evaluation remains contentious. In perceptual thermal colorization, TICPan is quantitatively below TIR2Lab and TIC-CGAN on PSNR and SSIM, yet is argued to be more reliable because thermal high-frequency injection preserves all thermally visible objects and avoids the missing pedestrians and deformed cars observed in adversarial baselines (Almasri et al., 2020). In low-light and glow-suppression work, user studies, CEIQ, edge visibility, and local saturation measures are introduced precisely because global fidelity metrics do not capture whether headlights, signboards, or street lamps remain operationally interpretable (Wu et al., 2023, Jin et al., 2022). This suggests that a general NightVision benchmark requires both perceptual and task-level criteria.

The cited literature also converges on several unresolved directions. Thermal-to-visible systems identify clipping during high-frequency injection, uncertainty for dynamic object color, and limited cross-sensor generalization as open issues (Almasri et al., 2020). Biologically inspired preprocessing reports sensitivity to the blur-depth parameter $L_1$2 and proposes learnable or adaptive preprocessing, architecture-specific tuning, and ISP-level deployment as future work (Stracke et al., 29 Sep 2025). Single-pixel and adaptive-projection systems point to spatial resolution, optimization speed, and motion robustness as the main blockers to practical deployment (Wang et al., 29 Jun 2026). Nonlinear metasurface upconversion identifies bandwidth, pump requirements, and fabrication tolerance as the main constraints on compact SWIR night-vision hardware (Molina et al., 2024). Across these disparate subfields, the common research trajectory is clear: NightVision is moving from isolated enhancement or display modules toward integrated systems that jointly optimize sensing, structure preservation, uncertainty management, and downstream task robustness.