Fog-Augmented Lost & Found Dataset

• The paper introduces a fog-augmented benchmark that extends the Lost & Found dataset using physics-based simulation to improve detection under low-visibility conditions.
• It employs realistic RGB and LiDAR fog simulation methods—using models like Koschmieder and convolutional sensor modeling—to generate adverse-weather imagery.
• Advanced domain adaptation techniques such as CMAda, FIFO, and TDo-Dif are applied to enhance scene understanding, tracking, and detection performance in foggy environments.

A Fog-Augmented Lost & Found Dataset denotes a specialized benchmark for evaluating scene understanding, object detection, and tracking under foggy conditions, built by extending the established Lost & Found dataset (originally focused on small road hazards in clear weather) with synthetic or real-world fog. This involves the systematic addition of physically accurate fog effects to both RGB and LiDAR imagery as well as the application of advanced adaptation techniques to enable robust perception by autonomous vehicles or surveillance systems in adverse atmospheric scenarios.

1. Foundations: The Lost & Found Dataset and Fog Augmentation Rationale

The original Lost & Found dataset (Pinggera et al., 2016) comprises thousands of stereo image frames recorded under real-world driving conditions and annotated at pixel-level for small road hazards such as lost cargo, debris, and other small obstacles. Its images present low-contrast, visually ambiguous scenes and are tailored for testing small-object detection pipelines.

Fog augmentation targets several shortcomings:

• Most existing benchmarks lack adverse-weather diversity, especially fog, even though fog degrades contrast, color, and depth cues, severely hampering detection, semantic segmentation, and tracking (1901.01415, Hahner et al., 2019, Kirillova et al., 12 Apr 2024).
• Robust real-world deployment requires models that generalize beyond clear-weather data to adverse conditions with reduced visibility.

A Fog-Augmented Lost & Found Dataset is constructed by systematically simulating realistic fog on the base images, or by capturing new annotated hazards in natural fog, and by integrating domain adaptation, curriculum learning, or fog-invariant representation methods to bridge the domain gap between clear-weather and fog.

2. Synthetic Fog Simulation: Physics-Based Models for RGB and LiDAR Data

Fog simulation is predicated on physically accurate optical models.

2.1 RGB Image Fog Simulation

The Koschmieder model is typically used (1901.01415, Hahner et al., 2019, Kirillova et al., 12 Apr 2024):

$$I(x) = R(x) \cdot t(x) + L_{\infty} \cdot (1 - t(x)), \quad t(x) = \exp(-\beta D(x))$$

where $I(x)$ is the foggy image, $R(x)$ the clear-image radiance, $L_{\infty}$ atmospheric light (estimated via dark channel prior if sky visibility is limited), $t(x)$ the transmission map, $\beta$ is the attenuation coefficient encoding fog density and $D(x)$ is the (monocular or stereo) estimate of scene depth.

Depth estimation can use state-of-the-art monocular networks such as MiDaS (Kirillova et al., 12 Apr 2024), with scale and shift adjustments as needed.

For heterogeneous fog effects, turbulence textures (multi-level Perlin noise) modulate $t(x)$:

$t(x) = \exp(-\beta \cdot \tau(x) \cdot D(x)),$

where $\tau(x)$ is the noise texture.

2.2 LiDAR Point Cloud Fog Simulation

Physically valid fog simulation for LiDAR employs a linear convolutional model of received power (Hahner et al., 2021):

$$P_R(R) = C_A \int_0^{2R/c} P_T(t) \cdot H(R - c t/2) \, dt$$

with $P_T$ the transmitted pulse, $H$ the environment impulse response, $C_A$ a sensor constant, and $c$ the speed of light.

Fog introduces attenuation and soft distributed returns:

• "Hard" returns are attenuated as $$P_{R,\text{fog}}^{\text{hard}}(R) = e^{-2\alpha R_0} P_{R,\text{clear}}(R)$$, with $\alpha$ the fog extinction coefficient.
• "Soft" returns from fog scatter require numerical integration.

This simulates pointwise modifications of intensity and (where the soft component dominates) spatial position, yielding foggy point clouds consistent with real degraded LiDAR data.

3. Domain Adaptation and Curriculum Learning for Scene Understanding

Fog greatly increases domain shift, necessitating specialized adaptation techniques:

• Curriculum Model Adaptation (CMAda): Segmentation models are gradually adapted from clear-weather through synthetic light fog to real, dense fog, optimizing loss functions that balance synthetic precision and real-scene fidelity (1901.01415). Synthetic fog is generated using dual-reference cross-bilateral filtering leveraging semantic and color edge cues, while a fog density estimator (typically a regression CNN) orders real scenes for curriculum progression. Fog densification is applied to real data lacking ground-truth depth.
• Fog-Invariant Feature Learning (FIFO): Treats fog condition as a stylization effect in feature space. Introduces learnable "fog-pass filter" modules that distill fog-related style from Gram matrices of feature maps, allowing auxiliary losses that match fog factors across domains during segmentation training (Lee et al., 2022). Prediction consistency and style matching losses minimize the feature gap across clear, synthetic, and real fog domains.
• Self Spatial-Temporal Label Diffusion (TDo-Dif): For unsupervised adaptation, confident pseudo labels in foggy images are diffused spatially (within superpixels) and temporally (via optical flow between sequential frames), with additional local spatial similarity and temporal contrastive losses during retraining (Liao et al., 2022).
• Domain Adaptation for Detection: Alignment is enforced not only on global features but also on physical cues (depth, transmission maps). Encoder-decoder architectures reconstruct clear backgrounds to penalize spurious object features, and consistency losses constrain the relationship between depth estimation and transmission maps per the physical fog model (Yang et al., 2022).

4. Dataset Construction and Evaluation: RGB, LiDAR, and Multi-Modal Extensions

RGB Datasets:

• Foggy Zurich (1901.01415, Liao et al., 2022): ~3800 real images, 40 densely annotated under dense fog. Semantic masks follow Cityscapes schema.
• Foggy Synscapes (Hahner et al., 2019): 25,000 unique, high-resolution, photo-realistic synthetic foggy road scenes, generated from gapless depth using physically based rendering.

LiDAR Datasets:

• Seeing Through Fog (STF) (Hahner et al., 2021): Used to evaluate fog simulation on point clouds. Baselines established for 3D detection under simulated fog.

Multi-Object Tracking (MOT):

• MOTChallenge Augmented (Kirillova et al., 12 Apr 2024): MOT17 tracked videos are fog-augmented using volumetric simulation, with fog intensity levels varied. Both outdoor (fog) and indoor (smoke) impairments are rendered. Monocular depth via MiDaS and atmospheric light estimation via dark channel prior are key components.

Evaluation across these datasets reveals:

• Substantial, quantifiable degradation of SOTA detection and tracking under increasing fog density (e.g., for MOT, HOTA scores drop up to 61% at severe fog).
• CMAda, FIFO, and TDo-Dif yield improvements in mIoU (e.g., CMAda3+ reaches 46.8% on Foggy Zurich-test, TDo-Dif 51.9% on Foggy Zurich, FIFO improves segmentation on both foggy and clear scenes).
• LiDAR fog simulation improves 3D AP scores and reduces false positives.

5. Integration Strategies and Practical Impact

A Fog-Augmented Lost & Found Dataset synthesizes multiple research thrusts:

• Benchmarking Fog-Robust Perception: Offers a testbed for evaluating algorithms under challenging, low-contrast conditions. Encourages robust domain adaptation, fog-invariant feature learning, and curriculum learning.
• Training Data Diversity: By mixing high-quality synthetic fog scenes (Foggy Synscapes) with real fog data, models benefit from both accurate physics and realistic appearance cues, which is critical for real-world generalization (Hahner et al., 2019).
• Extensibility to Modalities: Incorporates LiDAR (point cloud) fog simulation and blends multi-modal approaches (RGB, LiDAR, possibly radar) for comprehensive adverse-weather benchmarks (Hahner et al., 2021).
• Object Detection Accuracy: Techniques that preserve depth, background, and enforce transmission–depth consistency lead to improved detection performance on fog-augmented benchmarks (e.g., 47.6 mAP (Yang et al., 2022) on Foggy Cityscapes).
• Tracking and Re-Identification: Multi-object tracking systems suffer in persistent fog, motivating new research into weather-invariant feature extraction and association strategies (Kirillova et al., 12 Apr 2024).

6. Challenges, Controversies, and Prospects

• Synthetic vs. Real Fog: The absolute realism of synthetic fog effects hinges on accurate depth estimation and faithful atmospheric modeling. Artifact-free simulation requires gapless depth; turbulent/heterogeneous fog raises complexity (Perlin noise, dark channel prior for night scenes).
• Annotation Bottlenecks: Real foggy data remains costly to annotate; semi-supervised and self-training strategies (pseudo labels, diffusion) attempt to overcome sparsity.
• Sensor Fidelity: LiDAR fog simulation depends on accurate modeling of hardware (pulse width, dynamic gain), and homogeneous attenuation assumptions may not fully capture real-world variability (Hahner et al., 2021).
• Performance Gaps: Despite adaptation, MOT and detection performance under severe fog remains below clear weather, especially for small or distant objects.

Future directions include curriculum adaptation of LiDAR data, meta-learning for dynamic weather conditions, expansion to other adverse scenarios (rain, snow), sensor fusion, and targeted domain adaptation for small object recovery in cluttered fog.

7. Summary Table: Fog-Augmented Dataset Ingredients

Component	Key Methodology	Primary Benefit
RGB Fog Simulation	Physically-based optical model	Realistic fog rendering
LiDAR Fog Simulation	Sensor impulse model + attenuation	Congruent foggy point clouds
Domain Adaptation	Curriculum, fog-invariant, label diffusion	Cross-domain generalization
Dataset Composition	Synthetic + real fog, annotated hazards	Benchmark for fog robustness
Multi-Modal Fusion	RGB + LiDAR + mixed data	Adverse-weather perception

A Fog-Augmented Lost & Found Dataset provides an essential resource for evaluating and improving perception algorithms in adverse, real-world conditions, especially focusing on small object detection, tracking, and semantic understanding under fog. Its construction leverages principled simulation, adaptation, and multi-source data integration, setting a rigorous standard for future robust autonomous vision benchmarks.