---
title: KITTI-Weather Benchmark for Autonomous Driving
url: https://www.emergentmind.com/topics/kitti-weather-benchmark
type: topic
---

# KITTI-Weather Benchmark for Autonomous Driving

The KITTI-Weather Benchmark encompasses a family of weather-augmented benchmarks derived from the original KITTI suite, targeting the rigorous assessment of autonomous driving perception algorithms in realistic adverse-weather conditions. These benchmarks simulate or record modalities—RGB, LiDAR, stereo—under fog, rain, and snow with physically motivated models and/or chamber experiments, facilitating standardized evaluations for depth completion, object detection, tracking, LiDAR denoising, and place recognition. Multiple research groups have independently contributed KITTI-Weather derivatives, each tailored to a specific perception subtask and offering distinct forms of weather corruption, annotation, and evaluation methodology.

## 1. Dataset Variants, Construction, and Physical Modeling

A central feature of KITTI-Weather benchmarks is the application of parametric weather corruption models to original KITTI data, or the use of controlled acquisition to record authentic adverse-weather data at scale.

- **Adverse-Weather Augmentation Pipelines**
  - *RGB/2D augmentations*: Rain augmentation is implemented via the Halder et al. (2019) rendering pipeline, injecting five rain-rate levels (17–200 mm/h). Fog simulation is achieved by modulating contrast and applying atmospheric attenuation to yield multiple visibility levels (e.g., 30–750 m). These augmentations operate on the full 7 481 frames of KITTI-Object Detection without changing annotations [2106.08795].
  - *LiDAR/3D augmentations*: Physics-based simulation for rain and snow leverages Mie scattering theory, with each droplet/snowflake represented as a sphere, and includes backscatter, attenuation, and point drop-out. The attenuation coefficient $\beta$ is parameterized by meteorological optical range for fog (e.g., $\beta = 3/\mathrm{MOR}$ with MOR in [10, 100] m), and by rain/snow rate for precipitation [2501.07133, 2503.12350, 2408.13802].
  - *High-fidelity ground-truth acquisition*: In addition to simulation, “Pixel-Accurate Depth Evaluation in Realistic Driving Scenarios” [1906.08953] constructed a 1 600-frame dataset with per-pixel ground-truth depth maps at 25" angular resolution using terrestrial Leica P30 scan registration and chamber-based fog/rain application.

- **Dataset Scale and Structure**
  - *AURORA-KITTI*: 82,177 RGB–LiDAR pairs with clean metric depth ground truth; eight compound conditions (clear, fog, rain, snow × day/night), three severity levels (physical parameters provided), lens occlusion variations, paired clean references for a subset, and frame-level textual weather metadata [2603.14701].
  - *Weather-KITTI—Denoising/LPR*: 130,656–156,951 LiDAR-only point clouds, each with three severity levels for fog, rain, and snow, retaining all KITTI/ SemanticKITTI per-point semantic labels, with weather noise points explicitly labeled (class 110/111/112) [2408.13802, 2503.12350].
  - *KITTI-A/KITTI-Weather (Tracking)*: 96,360/91,320 car/pedestrian frames, each weather-augmented at five levels per type, using original KITTI Tracking 3D bounding box annotations [2501.07133].

## 2. Benchmark Protocols and Task Formulations

Each KITTI-Weather variant instantiates a protocol specific to a perception task, with detailed definitions of both data splits and evaluation.

- **Depth Completion and Denoising (DCD, AURORA-KITTI)**
  - Task: Given weather-corrupted RGB and sparse LiDAR depth, reconstruct the clean, dense depth $D^*$ from $(I_w, S_w)$.
  - Supervision: Combination of per-pixel $\ell_1$ reconstruction, scale-and-shift-invariant multi-scale distillation from clean priors, and residual-gradient regularization to suppress weather artefacts [2603.14701].

- **Dense Depth Evaluation (Pixel-Accurate KITTI-Weather)**
  - Task: Estimate depth maps under pixel-accurate ground truth; inputs are stereo images, sparse Velodyne, and (optionally) synthetic weathered versions.
  - Metrics: RMSE, MAE, threshold accuracies ($\delta$), log-RMSE, scale invariance, ARD, SRD, SIlog, SSIM, PSNR, relative PSNR, all computed across raw and binned depths to correct for range bias [1906.08953].

- **Object Detection (KITTI-Weather)**
  - Task: 2D/3D bounding box prediction under rain and fog simulations.
  - Modalities: Camera-only (YOLOv3), LiDAR-only (PointPillars), fusion (AVOD).
  - Evaluation: Average precision under IoU thresholds, precision–recall curves, and comparisons in clear, fog, rain (across five rain/four fog levels). Sensor ablation studies probe the impact of single/multimodal failures [2106.08795].

- **LiDAR Tracking (KITTI-A/KITTI-Weather)**
  - Task: 3D single-object tracking with appearance- and motion-based methods.
  - Metrics: Success (3D IoU over thresholds), precision (center error below thresholds), degradation rate ($DR$) against clean baseline, performance range and standard deviation across weather levels [2501.07133].

- **LiDAR Place Recognition/Loop Closure (WeatherKITTI)**
  - Task: Retrieve matching clean frames for corrupted queries; evaluation uses Recall@K, F1 at top-1, AUC of Recall@N, and the aggregate mean stability rate ($\mathrm{mSR}_{\ell}$) as robustness indicator [2503.12350].

- **Point Cloud Denoising (WeatherKITTI, TripleMixer)**
  - Task: Per-point binary classification (noise, non-noise), using explicit labels for weather-induced points.
  - Metrics: Precision, recall, F1, and mIoU, with evaluation against ground-truth weather-labels per scan [2408.13802].

## 3. Evaluation Metrics and Baseline Results

The table below summarizes representative metrics and core baselines evaluated under KITTI-Weather conditions, consistent with the original dataset documentation.

| Task / Dataset        | Metric(s)             | Weather Types / Levels    | Representative Baselines            | Key Performance Ranges                 |
|---------------------- |----------------------|--------------------------|-------------------------------------|----------------------------------------|
| Depth Completion      | RMSE, MAE, iRMSE, iMAE| {Clear, Fog, Rain, Snow} × 3| DDCD, BP-Net, UniK3D, Marigold-DC   | DDCD RMSE: 1799.6 mm; Next best: 2773–5047 mm [2603.14701]  |
| Dense Depth Estimation| RMSE, MAE, $\delta$, SIlog, binned | Clear/Night/Fog(17×)/Rain(2×) | PSMNet, SGM, Monodepth, Sparse2Dense| Deep stereo: MAE < 1m, robust to rain/fog; lidar/mono collapse in fog [1906.08953]      |
| Object Detection      | mAP, AP (IoU≥0.5)    | Rain(5), Fog(4)           | YOLOv3, PointPillars, AVOD          | YOLOv3: mAP drops from 76.8%→33.2% in dense fog; PP: −6pp under heavy rain [2106.08795] |
| 3D Tracking           | Success, Precision, Degradation Rate | Rain(5), Fog(5), Snow(5)| BAT, STNet, CXTrack, MBPTrack, MMTrack| MBPTrack: Car success falls by 34–44% (rain/snow), 17–24% (fog); MMTrack most stable [2501.07133]|
| LPR                   | R@K, F1, AUC, mSR    | Fog, Rain, Snow (3 levels)| Scan Context, CVTNet, LPSNet        | SC mSR: 0.310→0.949 after ResLPRNet [2503.12350]               |
| Denoising             | mIoU, Precision/Recall/F1 | Snow, Fog, Rain (3 levels)| TripleMixer, Cylinder3D, 3D-OutDet  | TripleMixer: mIoU 96.31%; Cylinder3D 94.28% [2408.13802]        |

Comprehensive metric suites are uniformly adopted, including error-based, accuracy, and robustness measures, often both aggregated and binned by depth or weather severity.

## 4. Physical Models, Simulation Fidelity, and Annotation

KITTI-Weather datasets prioritize physically plausible simulative corruption, ensuring that algorithms are evaluated under realistic sensor degradation scenarios.

- **Fog Simulation**: Based on exponential attenuation $I(d) = I_0 \exp(-\beta d)$; $\beta$ tied to meteorological optical range ($\mathrm{MOR}$) [2501.07133, 2408.13802].
- **Rain/Snow Simulation**: Incorporates particle-scattering, occlusion, random drop-out, and intensity adjustments; parameters derived from measured precipitation rates and physical optics [2408.13802].
- **Authentic Ground Truth**: In [1906.08953], “true” depth is reconstructed in controlled environments using a Leica P30, achieving ground-truth density > 10× that of Velodyne (3 mm spacing), and covering all image pixels.
- **Weather Noise Labeling**: For denoising tasks, per-point weather-induced noise is flagged automatically by tracking points with significant range/intensity alterations or introduced by the simulator [2408.13802].

Paired clean-reference data is systematically included in several variants (e.g., AURORA-KITTI, WeatherKITTI LPR) to support teacher/student distillation or restoration loss computation.

## 5. Baselines, Robustness Analysis, and Practical Recommendations

Performance under adverse weather reveals substantial differences across sensor modalities and algorithm classes.

- **Sensor Modalities**:
  - Stereo vision is most robust to fog and rain, outperforming monocular and LiDAR-based depth estimation (MAE increase for deep stereo is ~0.45 m under heavy fog vs. 2–5 m for lidar completion) [1906.08953].
  - Image-only detection collapses in dense fog (YOLOv3 mAP drops by >50%), but is only modestly affected by rain [2106.08795].
  - LiDAR-based approaches are resilient to low illumination and moderate fog, but degrade under heavy precipitation, due to spurious direct returns on droplets.
  - Fusion pipelines typically default to LiDAR for proposal generation, yielding failure when LiDAR returns collapse, unless sensor reliability is explicitly modeled.

- **Algorithmic Attributes**:
  - Motion-based trackers (MMTrack) exhibit higher robustness to template/target corruption than purely appearance-based methods in tracking [2501.07133].
  - Plug-and-play restoration networks (ResLPRNet) or denoisers (TripleMixer) can restore performance of downstream LPR/segmentation models to near-clean-weather levels, as evidenced by mSR or mIoU gains [2503.12350, 2408.13802].
  - Multi-severity curricula and physically consistent synthetic corruptions yield larger performance gains (up to 57% RMSE drop in depth completion) than architectural modifications alone [2603.14701].

- **Practical Guidelines**:
  - Always train or fine-tune on physically-based synthetic weather-corrupted data for robust deployment [2603.14701, 2106.08795].
  - Employ paired clean references for distillation, leveraging scale-and-shift normalization in teacher-student architectures.
  - Augment with lens occlusion (adherent raindrop/snowflake masks), as weather artefacts can localize on the sensor [2603.14701].
  - For efficiency, balance accuracy and runtime; e.g., Marigold-DC is slow but accurate, DDCD achieves real-time performance [2603.14701].

## 6. Impact, Limitations, and Future Research Directions

KITTI-Weather benchmarks have prompted systematic stress-testing of perception algorithms and have led to several methodological advances:

- **Empirical Insights**: High performance in clear-sky benchmarks does not generalize to adverse weather; explicit weather robustness must be designed and evaluated [2106.08795].
- **Algorithmic Innovation**: Domain-randomization, context modeling, and plug-and-play denoising have demonstrated transfer of robustness from synthetic to real-weather data [2501.07133, 2503.12350, 2408.13802].
- **Dataset Limitations**: Synthetic augmentation is effective but cannot substitute for real-world measurement in extreme weather (e.g., freezing rain, dust storms). Representative real-data is still lacking for several environmental scenarios [2106.08795].
- **Recommended Extensions**: Integration of additional weather phenomena (e.g., dust, hail), multi-sensor fusion (thermal/LiDAR), online adaptation to small real-weather datasets, and per-modality reliability modeling are suggested research frontiers [2603.14701, 2106.08795].
- **Release and Reproducibility**: All major KITTI-Weather datasets and codebases are released under permissive research licenses, with standardized scripts for protocol enforcement and metric calculation [1906.08953, 2603.14701, 2408.13802].

The proliferation of KITTI-Weather benchmarks aligns the field toward principled evaluation and robust real-world performance for safety-critical driving systems in challenging environmental conditions.

Source: https://www.emergentmind.com/topics/kitti-weather-benchmark