KITTI-Weather Benchmark for Autonomous Driving

• KITTI-Weather Benchmark is a comprehensive family of datasets that augments original KITTI with weather-induced corruptions for robust evaluation.
• It employs physics-based simulation and controlled data capture to generate realistic adverse weather conditions across modalities and perception tasks.
• The benchmark protocols incorporate detailed evaluation metrics and baseline comparisons to assess performance variances in depth completion, object detection, tracking, and more.

The KITTI-Weather Benchmark encompasses a family of weather-augmented benchmarks derived from the original KITTI suite, targeting the rigorous assessment of autonomous driving perception algorithms in realistic adverse-weather conditions. These benchmarks simulate or record modalities—RGB, LiDAR, stereo—under fog, rain, and snow with physically motivated models and/or chamber experiments, facilitating standardized evaluations for depth completion, object detection, tracking, LiDAR denoising, and place recognition. Multiple research groups have independently contributed KITTI-Weather derivatives, each tailored to a specific perception subtask and offering distinct forms of weather corruption, annotation, and evaluation methodology.

1. Dataset Variants, Construction, and Physical Modeling

A central feature of KITTI-Weather benchmarks is the application of parametric weather corruption models to original KITTI data, or the use of controlled acquisition to record authentic adverse-weather data at scale.

• Adverse-Weather Augmentation Pipelines
  • RGB/2D augmentations: Rain augmentation is implemented via the Halder et al. (2019) rendering pipeline, injecting five rain-rate levels (17–200 mm/h). Fog simulation is achieved by modulating contrast and applying atmospheric attenuation to yield multiple visibility levels (e.g., 30–750 m). These augmentations operate on the full 7 481 frames of KITTI-Object Detection without changing annotations (Mirza et al., 2021).
  • LiDAR/3D augmentations: Physics-based simulation for rain and snow leverages Mie scattering theory, with each droplet/snowflake represented as a sphere, and includes backscatter, attenuation, and point drop-out. The attenuation coefficient $\beta$ is parameterized by meteorological optical range for fog (e.g., $\beta = 3/\mathrm{MOR}$ with MOR in [10, 100] m), and by rain/snow rate for precipitation (Zhao et al., 13 Jan 2025, Kuang et al., 16 Mar 2025, Zhao et al., 2024).
  • High-fidelity ground-truth acquisition: In addition to simulation, “Pixel-Accurate Depth Evaluation in Realistic Driving Scenarios” (Gruber et al., 2019) constructed a 1 600-frame dataset with per-pixel ground-truth depth maps at 25" angular resolution using terrestrial Leica P30 scan registration and chamber-based fog/rain application.
• Dataset Scale and Structure
  • AURORA-KITTI: 82,177 RGB–LiDAR pairs with clean metric depth ground truth; eight compound conditions (clear, fog, rain, snow × day/night), three severity levels (physical parameters provided), lens occlusion variations, paired clean references for a subset, and frame-level textual weather metadata (Wang et al., 16 Mar 2026).
  • Weather-KITTI—Denoising/LPR: 130,656–156,951 LiDAR-only point clouds, each with three severity levels for fog, rain, and snow, retaining all KITTI/ SemanticKITTI per-point semantic labels, with weather noise points explicitly labeled (class 110/111/112) (Zhao et al., 2024, Kuang et al., 16 Mar 2025).
  • KITTI-A/KITTI-Weather (Tracking): 96,360/91,320 car/pedestrian frames, each weather-augmented at five levels per type, using original KITTI Tracking 3D bounding box annotations (Zhao et al., 13 Jan 2025).

2. Benchmark Protocols and Task Formulations

Each KITTI-Weather variant instantiates a protocol specific to a perception task, with detailed definitions of both data splits and evaluation.

• Depth Completion and Denoising (DCD, AURORA-KITTI)
  • Task: Given weather-corrupted RGB and sparse LiDAR depth, reconstruct the clean, dense depth $D^*$ from $(I_w, S_w)$.
  • Supervision: Combination of per-pixel $\ell_1$ reconstruction, scale-and-shift-invariant multi-scale distillation from clean priors, and residual-gradient regularization to suppress weather artefacts (Wang et al., 16 Mar 2026).
• Dense Depth Evaluation (Pixel-Accurate KITTI-Weather)
  • Task: Estimate depth maps under pixel-accurate ground truth; inputs are stereo images, sparse Velodyne, and (optionally) synthetic weathered versions.
  • Metrics: RMSE, MAE, threshold accuracies ($\delta$), log-RMSE, scale invariance, ARD, SRD, SIlog, SSIM, PSNR, relative PSNR, all computed across raw and binned depths to correct for range bias (Gruber et al., 2019).
• Object Detection (KITTI-Weather)
  • Task: 2D/3D bounding box prediction under rain and fog simulations.
  • Modalities: Camera-only (YOLOv3), LiDAR-only (PointPillars), fusion (AVOD).
  • Evaluation: Average precision under IoU thresholds, precision–recall curves, and comparisons in clear, fog, rain (across five rain/four fog levels). Sensor ablation studies probe the impact of single/multimodal failures (Mirza et al., 2021).
• LiDAR Tracking (KITTI-A/KITTI-Weather)
  • Task: 3D single-object tracking with appearance- and motion-based methods.
  • Metrics: Success (3D IoU over thresholds), precision (center error below thresholds), degradation rate ($DR$) against clean baseline, performance range and standard deviation across weather levels (Zhao et al., 13 Jan 2025).
• LiDAR Place Recognition/Loop Closure (WeatherKITTI)
  • Task: Retrieve matching clean frames for corrupted queries; evaluation uses Recall@K, F1 at top-1, AUC of Recall@N, and the aggregate mean stability rate ($\mathrm{mSR}_{\ell}$) as robustness indicator (Kuang et al., 16 Mar 2025).
• Point Cloud Denoising (WeatherKITTI, TripleMixer)
  • Task: Per-point binary classification (noise, non-noise), using explicit labels for weather-induced points.
  • Metrics: Precision, recall, F1, and mIoU, with evaluation against ground-truth weather-labels per scan (Zhao et al., 2024).

3. Evaluation Metrics and Baseline Results

The table below summarizes representative metrics and core baselines evaluated under KITTI-Weather conditions, consistent with the original dataset documentation.

Task / Dataset	Metric(s)	Weather Types / Levels	Representative Baselines	Key Performance Ranges
Depth Completion	RMSE, MAE, iRMSE, iMAE	{Clear, Fog, Rain, Snow} × 3	DDCD, BP-Net, UniK3D, Marigold-DC	DDCD RMSE: 1799.6 mm; Next best: 2773–5047 mm (Wang et al., 16 Mar 2026)
Dense Depth Estimation	RMSE, MAE, $\delta$, SIlog, binned	Clear/Night/Fog(17×)/Rain(2×)	PSMNet, SGM, Monodepth, Sparse2Dense	Deep stereo: MAE < 1m, robust to rain/fog; lidar/mono collapse in fog (Gruber et al., 2019)
Object Detection	mAP, AP (IoU≥0.5)	Rain(5), Fog(4)	YOLOv3, PointPillars, AVOD	YOLOv3: mAP drops from 76.8%→33.2% in dense fog; PP: −6pp under heavy rain (Mirza et al., 2021)
3D Tracking	Success, Precision, Degradation Rate	Rain(5), Fog(5), Snow(5)	BAT, STNet, CXTrack, MBPTrack, MMTrack	MBPTrack: Car success falls by 34–44% (rain/snow), 17–24% (fog); MMTrack most stable (Zhao et al., 13 Jan 2025)
LPR	R@K, F1, AUC, mSR	Fog, Rain, Snow (3 levels)	Scan Context, CVTNet, LPSNet	SC mSR: 0.310→0.949 after ResLPRNet (Kuang et al., 16 Mar 2025)
Denoising	mIoU, Precision/Recall/F1	Snow, Fog, Rain (3 levels)	TripleMixer, Cylinder3D, 3D-OutDet	TripleMixer: mIoU 96.31%; Cylinder3D 94.28% (Zhao et al., 2024)

Comprehensive metric suites are uniformly adopted, including error-based, accuracy, and robustness measures, often both aggregated and binned by depth or weather severity.

4. Physical Models, Simulation Fidelity, and Annotation

KITTI-Weather datasets prioritize physically plausible simulative corruption, ensuring that algorithms are evaluated under realistic sensor degradation scenarios.

• Fog Simulation: Based on exponential attenuation $I(d) = I_0 \exp(-\beta d)$; $\beta = 3/\mathrm{MOR}$0 tied to meteorological optical range ($\beta = 3/\mathrm{MOR}$1) (Zhao et al., 13 Jan 2025, Zhao et al., 2024).
• Rain/Snow Simulation: Incorporates particle-scattering, occlusion, random drop-out, and intensity adjustments; parameters derived from measured precipitation rates and physical optics (Zhao et al., 2024).
• Authentic Ground Truth: In (Gruber et al., 2019), “true” depth is reconstructed in controlled environments using a Leica P30, achieving ground-truth density > 10× that of Velodyne (3 mm spacing), and covering all image pixels.
• Weather Noise Labeling: For denoising tasks, per-point weather-induced noise is flagged automatically by tracking points with significant range/intensity alterations or introduced by the simulator (Zhao et al., 2024).

Paired clean-reference data is systematically included in several variants (e.g., AURORA-KITTI, WeatherKITTI LPR) to support teacher/student distillation or restoration loss computation.

5. Baselines, Robustness Analysis, and Practical Recommendations

Performance under adverse weather reveals substantial differences across sensor modalities and algorithm classes.

• Sensor Modalities:
  • Stereo vision is most robust to fog and rain, outperforming monocular and LiDAR-based depth estimation (MAE increase for deep stereo is ~0.45 m under heavy fog vs. 2–5 m for lidar completion) (Gruber et al., 2019).
  • Image-only detection collapses in dense fog (YOLOv3 mAP drops by >50%), but is only modestly affected by rain (Mirza et al., 2021).
  • LiDAR-based approaches are resilient to low illumination and moderate fog, but degrade under heavy precipitation, due to spurious direct returns on droplets.
  • Fusion pipelines typically default to LiDAR for proposal generation, yielding failure when LiDAR returns collapse, unless sensor reliability is explicitly modeled.
• Algorithmic Attributes:
  • Motion-based trackers (MMTrack) exhibit higher robustness to template/target corruption than purely appearance-based methods in tracking (Zhao et al., 13 Jan 2025).
  • Plug-and-play restoration networks (ResLPRNet) or denoisers (TripleMixer) can restore performance of downstream LPR/segmentation models to near-clean-weather levels, as evidenced by mSR or mIoU gains (Kuang et al., 16 Mar 2025, Zhao et al., 2024).
  • Multi-severity curricula and physically consistent synthetic corruptions yield larger performance gains (up to 57% RMSE drop in depth completion) than architectural modifications alone (Wang et al., 16 Mar 2026).
• Practical Guidelines:
  • Always train or fine-tune on physically-based synthetic weather-corrupted data for robust deployment (Wang et al., 16 Mar 2026, Mirza et al., 2021).
  • Employ paired clean references for distillation, leveraging scale-and-shift normalization in teacher-student architectures.
  • Augment with lens occlusion (adherent raindrop/snowflake masks), as weather artefacts can localize on the sensor (Wang et al., 16 Mar 2026).
  • For efficiency, balance accuracy and runtime; e.g., Marigold-DC is slow but accurate, DDCD achieves real-time performance (Wang et al., 16 Mar 2026).

6. Impact, Limitations, and Future Research Directions

KITTI-Weather benchmarks have prompted systematic stress-testing of perception algorithms and have led to several methodological advances:

• Empirical Insights: High performance in clear-sky benchmarks does not generalize to adverse weather; explicit weather robustness must be designed and evaluated (Mirza et al., 2021).
• Algorithmic Innovation: Domain-randomization, context modeling, and plug-and-play denoising have demonstrated transfer of robustness from synthetic to real-weather data (Zhao et al., 13 Jan 2025, Kuang et al., 16 Mar 2025, Zhao et al., 2024).
• Dataset Limitations: Synthetic augmentation is effective but cannot substitute for real-world measurement in extreme weather (e.g., freezing rain, dust storms). Representative real-data is still lacking for several environmental scenarios (Mirza et al., 2021).
• Recommended Extensions: Integration of additional weather phenomena (e.g., dust, hail), multi-sensor fusion (thermal/LiDAR), online adaptation to small real-weather datasets, and per-modality reliability modeling are suggested research frontiers (Wang et al., 16 Mar 2026, Mirza et al., 2021).
• Release and Reproducibility: All major KITTI-Weather datasets and codebases are released under permissive research licenses, with standardized scripts for protocol enforcement and metric calculation (Gruber et al., 2019, Wang et al., 16 Mar 2026, Zhao et al., 2024).

The proliferation of KITTI-Weather benchmarks aligns the field toward principled evaluation and robust real-world performance for safety-critical driving systems in challenging environmental conditions.