Multi-Sensor Nighttime Dataset

• Multi-sensor nighttime dataset is a benchmark comprising aligned measurements from diverse modalities acquired under low illumination for robust perception tasks.
• It employs advanced calibration, synchronization, and preprocessing techniques such as coaxial imaging and hardware triggers to ensure pixel-level alignment.
• The dataset supports applications in autonomous driving, robotics, surveillance, and remote sensing while addressing challenges like noise and dynamic range disparities.

A multi-sensor nighttime dataset is a benchmark corpus composed of aligned measurements from two or more imaging or ranging devices, specifically acquired under low-illumination or night conditions. These datasets are designed to address key challenges in perception, recognition, enhancement, and localization for vision algorithms exposed to reduced ambient lighting, strong dynamic range disparities, and noise artifacts. They typically support tasks such as low-light enhancement, depth estimation, segmentation, multimodal fusion, and detection in automotive, robotics, surveillance, remote sensing, and mobile contexts.

1. Sensor Modalities and Acquisition Architectures

Nighttime multi-sensor datasets employ an array of sensor combinations to achieve complementary coverage where each modality compensates for specific limitations in nocturnal imaging.

• Event Cameras: Devices such as the Prophesee EVK4 (1280×720 px, ≃120 dB dynamic range, ≈1 μs latency) are central to datasets like RLED, capturing per-pixel brightness changes asynchronously and excelling in motion deblurring and dynamic range (Liu et al., 2024). Event modalities achieve robust temporal resolution under low-light yet impose unique calibration challenges due to their sparse, non-frame-based data.
• Conventional Frame Cameras: RGB sensors (e.g., FLIR BFS-U3-32S4C, 2048×1536 px) provide standard image frames but struggle with long exposure-induced blur at night. Their integration via perfect coaxial beam-splitting and external triggering synchronizes exposures and fields-of-view.
• Multi-exposure CMOS Sensors: LENVIZ utilizes three mobile CMOS sensors (front: 8 MP, rear: 50 MP; pixel pitch range 0.64–1.12 μm), capturing exposure-bracketed stacks and long-exposure references to maximize dynamic range and provide noise/SNR calibration (Aithal et al., 25 Mar 2025).
• Infrared and Thermal Cameras: Datasets such as MS² and InfraParis integrate LWIR imagers (FLIR A65C, Optris PI 450i) for temperature-contrast-based detection, with spatial resolutions from 382×288 px to 640×512 px and spectral bands (8–14 μm). Thermal modalities exhibit robustness to visible-light deprivation but often trade-off spatial fidelity (Shin et al., 28 Mar 2025, Franchi et al., 2023).
• Stereo, NIR, and LiDAR: Stereo RGB/NIR pairs, multi-spectral cameras (VIS-SWIR), and two synchronized Velodyne LiDAR units expand geometric and spectral coverage, as in MS² and MineInsight (Malizia et al., 5 Jun 2025).
• Calibration and Synchronization: Most datasets employ hardware triggers, master clocks (PTP), and multi-level calibration (intrinsics, extrinsics, rectification) to ensure pixel-level alignment across modalities, minimizing parallax and registration errors.

2. Dataset Construction and Annotation Protocols

• Acquisition Pipelines: Data are captured via either co-axial systems (shared lens and beam splitter), rigid multi-head mounts (e.g., aluminum plate in InfraParis), or arm/base dual-platform UGVs (MineInsight). Exposure times, ISO, and sensor gain are systematically varied to span 0.01–1000 lux.
• Temporal Synchronization: Synchronization circuits distribute trigger pulses to all sensors for matched timestamping, ensuring event and frame alignment without post-hoc drift correction (RLED).
• Geometric Calibration: Intrinsics are typically estimated via checkerboard or AprilTag boards; extrinsics use stereo calibration, SLAM alignment, and ICP for depth consistency.
• Annotation: Semantic segmentation datasets (e.g., DSEC Night-Semantic) rely on manual polygon masks across 18–20 urban classes, while detection datasets integrate bounding boxes, world coordinates, and AprilTag IDs. Ground truth for depth is produced via LiDAR reprojection and multi-scan ICP refinement (MS²).
• Quality Control: Photographer-edited ground truths (LENVIZ), perceptual metrics (LPIPS for enhancement quality), annotation consistency audits (>95%), and physical model-based SNR/noise checks are standard.

3. Data Organization, Modalities, and Metadata

Multi-sensor nighttime datasets are typically organized in hierarchical folder structures, with modality-specific subfolders for events, images, depth, thermal, and calibration. Data formats vary: PNG (images, semantic masks), .aedat4 (event streams), GeoTIFF (remote sensing), ROS 2 bags (robotics), JSON/CSV (annotations, manifests).

Dataset	Modalities	Resolution/Count
RLED	Event + RGB	64,200 aligned pairs
LENVIZ	3x CMOS RGB	234,688 frames, 24k scenes
DSEC Night	Event + RGB	1,692 pairs, 150 labeled
InfraParis	RGB, Depth, LWIR	7,301 annotated, 20 classes
MS²	RGB, NIR, Thermal, LiDAR	162k samples
MineInsight	RGB, VIS-SWIR, LWIR, LiDAR	38k RGB, 53k SWIR, 108k LWIR

Each dataset includes a manifest (CVS/JSON) indexing file names, timestamps, calibration metadata, and often task-specific split lists (train/val/test).

4. Quantitative Characteristics and Evaluation Metrics

• Illumination and Event Statistics: RLED spans 0.5–1000 lux (mean ≈75, median ≈20), with bright regions yielding up to 10× event density versus dark. Datasets model photocurrent $I_\mathrm{ph} = RAE$, artificial-light fall-off $I(x) = \Phi/(4\pi d^2)$, and event-rate $\lambda(t) = E(t)/\Delta t$ (Liu et al., 2024).
• Noise Modeling: LENVIZ applies per-pixel photon-shot and read-noise models $y = x + n_s + n_r$, with $n_s \sim \mathrm{Poisson}(\phi x)$ and $n_r \sim \mathcal{N}(0,\sigma_r^2)$. SNR is computed by $SNR(x) = \phi x / \sqrt{\phi x + \sigma_r^2}$.
• Performance Metrics: Enhancement (LENVIZ): PSNR, SSIM, LPIPS. Reconstruction (RLED): MSE, SSIM, LPIPS. Depth (InfraParis, MS²): AbsRel, RMSE, SqrRel, RMSElog, $\delta < 1.25^n$. Segmentation: mIoU. Detection: AP metrics.
• Results: E.g., RLED: NER-Net achieves MSE 0.011, SSIM 0.717, LPIPS 0.309, outperforming prior event-based methods. LENVIZ-enhanced models (LLFormer, ExpoMamba, MEFNet) exhibit improvements in PSNR (21.13 dB), SSIM (0.665), LPIPS (0.358). InfraParis depth estimation on thermal yields AbsRel=0.152, RMSE=2.64 m, while RGB reaches AbsRel=0.203, RMSE=3.64 m (Liu et al., 2024, Aithal et al., 25 Mar 2025, Franchi et al., 2023, Shin et al., 28 Mar 2025).
• Cross-dataset Generalization: Training on real nighttime data (e.g., RLED) improves downstream performance (LOE, SSIM) on other nighttime benchmarks (DSEC-night, MVSEC-night, VECtor-hdr).

5. Benchmark Tasks, Applications, and Limitations

Key tasks addressed by multi-sensor nighttime datasets:

• Low-light Enhancement: Multi-exposure image fusion, exposure balancing, denoising, detail sharpening (LENVIZ).
• Semantic Segmentation and Detection: Cityscapes-compliant class schema (InfraParis, DSEC Night-Semantic), event-modality fusion, domain adaptation from day to night (CMDA).
• Depth Estimation: Monocular/stereo RGB, NIR, and thermal; LiDAR-projected ground truth (MS², InfraParis).
• 3D Localization: SLAM-derived target positions, 2D bounding box projection, robust pose estimation (MineInsight).
• Remote Sensing Super-resolution: Multi-modal NTL fusion, calibration-aware alignment, auxiliary-embedded refinement (DeepLightMD, DeepLightSR) (Zhang et al., 2024).

Applications reported include autonomous driving, robotics in demining and off-road environments, mobile video enhancement, face preservation/fairness studies, and SDG-oriented remote sensing. Limitations comprise sensor gaps (no SAR/thermal in DeepLightMD), temporal bias, overpass mismatch, domain shift in thermal/NIR, stereo matching errors in rain/glare, and annotation cost.

6. Methodological Innovations and Preprocessing Pipelines

• Pixel-level Alignment: Coaxial imaging (beam splitters), hardware synchronization, checkerboard calibration ensure high registration accuracy (sub-pixel RMS).
• Event Preprocessing: Accumulation over temporal windows (DSEC: 50 ms), polarity-weighted voxel grids, noise filtering, log-difference and clipping normalization, style-invariant edge extraction (CMDA).
• Photographer-driven Ground Truth: Manual retouching (exposure/contrast, skin tone anchoring, color fidelity) dominates in LENVIZ; avoids automated ISP artifacts.
• Calibration-Aware Alignment: DeepLightSR employs STN-driven global warps, deformable convolutions, and CMFM fusion blocks, with multi-scale SR losses and auxiliary supervision (Zhang et al., 2024).

7. Current Impact and Research Directions

Multi-sensor nighttime datasets have become benchmarks for evaluating algorithmic robustness to low-light scenarios and supporting development of cross-modal fusion, enhancement, and perception systems:

• Event-based vision: Extends capability to dynamic, high-motion scenes where conventional frames fail.
• Thermal and SWIR imaging: Augments detection and depth inference especially under lighting deprivation.
• Domain adaptation: Enables unsupervised/semi-supervised learning from day-to-night and modality-to-modality splits.
• Open benchmarks: Publicly available corpora (e.g., RLED, LENVIZ, MS², MineInsight, DeepLightMD) with code/tools have accelerated reproducible research.
• Future expansions: Incorporation of SAR, temporally consistent super-resolution, adversarial tails modeling, and joint SR/enhancement networks are foregrounded.

In summary, the rigorous design of multi-sensor nighttime datasets provides essential infrastructure for validating advanced perception and enhancement algorithms, facilitating progress in night-robust computer vision, autonomous navigation, and remote sensing (Liu et al., 2024, Aithal et al., 25 Mar 2025, Xia et al., 2023, Malizia et al., 5 Jun 2025, Franchi et al., 2023, Zhang et al., 2024, Shin et al., 28 Mar 2025).